dtSearch system and hardware requirements

dtSearch Support

Support index

Search

dtSearch Home

Downloads

Quick Start

dtSearch system and hardware requirements

Last Reviewed: December 20, 2004

Article: DTS0155

Applies to: dtSearch 6.04 and later

For information about dtSearch Web requirements, please see these articles:

dtSearch Web 6 system requirements and performance?

Operating systems supported

Windows 95, Windows 98, Windows ME, Windows NT 4 or later, Windows 2000, Windows XP (through SP2), and Windows 2003 Server.

Web Browser

dtSearch Desktop 6 also requires that Internet Explorer 5 or later be installed. Internet Explorer does not have to be your default browser (you can still use Netscape, Opera, or any other browser). dtSearch just needs certain components that are only available if Internet Explorer 5 or 6 is installed on the computer.

Not supported: Windows 3.x (16-bit), Windows NT 3.51 or earlier

Hardware Requirements

dtSearch Desktop dialog boxes are designed for systems with a screen resolution of 800x600 or higher.

Installation takes between 4 and 30 megabytes of hard disk space, depending on the options selected. (There is an optional thesaurus that takes about 20 Mb installed. The thesaurus is used for synonym searching.)

Additionally, you should have enough free space on the hard disk to build indexes of your documents. A dtSearch index takes about 1/3 the size of the original documents. Please see the article on Index Size for more information.

Recommended memory size is 256 Mb or more. dtSearch itself requires about 10-20 Mb of memory to search. Indexing can be done with as little as 30 Mb but more memory will result in faster performance.

Performance Recommendations

1. Do not put dtSearch indexes on a compressed drive or folder. For example, do not use NTFS compression or encryption on dtSearch indexes. dtSearch indexes are internally compressed, so you will not reduce their size much, and operating system file compression can slow index access considerably. (dtSearch performance is not affected much if the documents are stored in a compressed drive or folder. It is only the indexes that should not be compressed.)

2. For best performance, use a defragmentation utility such as Diskeeper to keep your hard disk defragmented, and keep the hard disk at least 25% empty so the defragmentation utility can run efficiently.

3. Index updates will be faster and more reliable if dtSearch Desktop runs on the machine where the index is stored. It matters much less where the documents are located, because dtSearch just needs to read each document once to index it.

Copyright © 1995-2005 dtSearch Corp.


How to change the size of the "title" field

dtSearch Home

Downloads

Quick Start

How to change the size of the "title" field

Last Reviewed: November 29, 2000

Article: DTS0144

Applies to: dtSearch 5, dtSearch 6

When dtSearch indexes a document, it collects the first 80 characters of text in the document and uses this text as the "title" property of the document. The title is displayed in search results and is also available to through the developer APIs.

The amount of text stored in the title is determined by the titleSize option, which can be increased from the default of 80 up to a maximum of 2048 characters or reduced to make the size of an index smaller.

Changes to the titleSize option only affect documents indexed after the change was made, so to change the title size of an existing index, the index would have to be completely rebuilt after changing the titleSize option.

How to change the titleSize option setting:

dtSearch Desktop 6 A registry entry can be used to change this in dtSearch. The key is:

HKEY_CURRENT_USER\Software\dtSearch Corp.\dtSearch\Settings\TitleSize

dtSearch Engine (Visual Basic) Use the Options.TitleSize property

dtSearch Engine (C/C++) Use the titleSize member of dtsOptions

dtSearch Engine (Java) Use the titleSize member of the Options object

HTML and PDF files have their own "Title" property that dtSearch can use instead of the first 80 characters in the file. For more information on HTML and PDF titles, see Article DTS0141 "How to display PDF and HTML titles in search results".


How to make blocks of text non-searchable

dtSearch Home

Downloads

Quick Start

How to make blocks of text non-searchable

Last Reviewed: March 27, 2003

Article: DTS0185

Applies to: dtSearch 6.1 or later

Excluding sections of web pages or XML files

To tell dtSearch not to index parts of an HTML or XML file, add comments around the text to be excluded, like this:

<!--BeginNoIndex-->

... nothing here will be searchable...

<!--EndNoIndex-->

The BeginNoIndex and EndNoIndex tags must look exactly as they do in this example. dtSearch will skip everything between the two markers when it indexes web pages.

Other file types

dtSearch does not have an option to prevent indexing blocks of text in other file formats. However, there are options to exclude document properties or summary information fields. For more information on these options, see:

How to exclude document properties from indexing


How to prevent filenames and document properties from appearing in documents

dtSearch Home

Downloads

Quick Start

How to prevent filenames and document properties from appearing in documents

Last Reviewed: September 5, 2004

Article: DTS0173

Applies to: dtSearch 6

Some document formats contain fields or properties in addition to the document text. For example, Word documents contain a Document Summary area with information such as the Author and Subject of a document. dtSearch automatically recognizes these fields and indexes them so you can search for text across the whole document or just in a particular field. These fields are added to the end of the document text during indexing and are displayed at the end of the file when it is opened after a search.

dtSearch also adds the document filename as a field named "Filename".

dtSearch currently indexes document properties in the following file types:

Microsoft Word, Excel, PowerPoint -- Document Summary Information

WordPerfect -- Document Summary Information

PDF -- Document Properties, Summary

HTML -- META tags

dtSearch Desktop

To prevent document properties from being indexed (so only the main text of the document will be searchable), start dtSearch Desktop and click Options > Preferences > Indexing Options, and un-check the Index document properties box.

To prevent filenames (including the path) from being indexed as part of the document text, start dtSearch Desktop and click Options > Preferences > Indexing Options, and un-check the Index filenames as text box.

dtSearch Engine

Programmers using the dtSearch Engine developer API can change this option as follows:

(Visual Basic, .NET) Use the Options.FieldFlags property

(C/C++) Use the fieldFlags member of dtsOptions

(Java) Use the fieldFlags member of the Options object

dtSearch Desktop 6 A registry entry can be used to change this in dtSearch Desktop. The key is:

HKEY_CURRENT_USER\Software\dtSearch Corp.\dtSearch\Settings\FieldFlags

Use dtsoFfSkipDocumentProperties to suppress document properties, and use dtsoFfSkipFilenameField to suppress the filename field. See the dtSearch Text Retrieval Engine Programmer's Reference (dtengine.chm) for complete documentation for FieldFlags.

The dtSearch Engine does not permanently store the FieldFlags value anywhere, so if you set Options.FieldFlags before indexing documents, you will need to set it to the same value before invoking FileConverter to highlight hits.


How to get field data in search results

dtSearch Home

Downloads

Quick Start

How to get field data in search results

Last Reviewed: Novemebr 9, 2004

Article: DTS0109

Applies to: dtSearch 5.x, 6

Some document formats contain fields (or "properties") in addition to the document text. For example, Word documents contain a Document Summary area with information such as the Author and Subject of a document. dtSearch automatically recognizes these fields and indexes them so that you can search for text across the whole document or just in a particular field. See the "Field searching" topic in the dtSearch help file for more information on field searches.

In some cases, it is useful not only to search for a particular field but to see the field as part of the search results list. dtSearch can do this if a field is identified as one that should be stored in the index along with other document information (such as the filename, date, etc.) and returned in search results. These fields are called "stored fields." When a field is set up as a stored field, the value of the field will appear in search results along with other document properties, and the field can also be used to sort search results.

Sources of field data

Field data can come from any of the following sources: document properties, HTML META tags, fields in databases and XML files, and text fields defned with the dtSearch "Define Text Fields" feature.

Storing Fields

dtSearch Desktop

1. In dtSearch Desktop, click Options > Preferences > Text Fields.

2. Create a field with the same name as the property that you want to store. For example, to store a "Subject" property, create a text field item named "Subject". You do not need to put anything in the spaces for Beginning of field, End of field, etc.

3. Check the box labelled Display field in search results

Fields relating to a document are stored when the document is indexed, so you will need to rebuild any indexes containing the field data for this change to take effect. When you rebuild your indexes, check the Clear index before adding documents option in the Update Index dialog box so that all of your documents will be reindexed.

You can also specify a list of fields to be stored when you create an index. To do this, click Index > Create Index (Advanced) in dtSearch Desktop, and list the fields to store under Fields to display in search results

dtSearch Engine (C/C++) When you index documents, you can specify in the dtsIndexJob that the text in certain fields should be collected, stored in the index, and returned in searches. To specify the fields to be stored, set the storedFields member to point to a null-delimited list of field names, terminated with a double-null. The field names in the list can contain wildcards (* and ?). A set containing a single entry "*" would match all fields, causing the text of every field to be stored in the index.

dtSearch Engine (Visual Basic) When you index documents, you can specify in the IndexJob that the text in certain fields should be collected, stored in the index, and returned in searches. To specify the fields to be stored, set the storedFields property to a list of field names, separated by spaces. The field names in the list can contain wildcards (* and ?). A single entry "*" would match all fields, causing the text of every field to be stored in the index.

Limits on Stored Fields

By default, the maximum size of a stored field is 128 characters. This can be changed with the maxStoredFieldSize option setting, and can be increased to up to 8192 characters. The total size of all all stored fields associated with a single document, including field names, is limited to 27k.

How to change the limit on the size of stored fields:

dtSearch Desktop 6 A registry entry can be used to change this in dtSearch. The key is:

HKEY_CURRENT_USER\Software\dtSearch Corp.\dtSearch\Settings\MaxStoredFieldSize

dtSearch Engine (Visual Basic) Use the Options.MaxStoredFieldSize property

dtSearch Engine (C/C++) Use the maxStoredFieldSize member of dtsOptions

These limits do not affect field searching. All fields in a document are searchable, regardless of how long they are or how many fields are in the document. For example, in a document with 100 fields, each 64k in length, all of the data in all of the fields would be searchable. The limits described here apply only to fields that are copied into the index and returned in search results.

The limit on stored field size applies when a document is indexed, so after changing the value you will need to rebuild your indexes for the change to take effect.

Obtaining Stored Fields

dtSearch After a search, dtSearch displays any stored field data returned as additional columns in search results. For example, if you create a "Subject" stored field, the search results list will contain a "Subject" column after the other columns.

dtSearch Web In dtSearch Web 6, the Form Builder dialog box has a Search Results tab that lets you specify the items to include in search results. To add a custom field, click the Add button and add an item with the name of the field you want. Under Content, use %% around the name of a field to get the value of that field. For example, to add the "Subject" field to search results, you would create a new item in Search Results named "Subject", and under Content you would enter

%%Subject%%

See also: "How to add custom fields to dtSearch Web search results"

Active Server Pages and Visual Basic In an ASP or Visual Basic application, you can use the DocDetailItem method of a SearchResults object to obtain custom fields. For example, to obtain the text of a "Subject" field, you would use: results.DocDetailItem("Subject").

dtSearch Engine (C/C++) During a search, fields that match the storedFields list will be returned through the search status callback function in the dtsSearchResultsItem structure. The userFields member of the dtsSearchResultsItem will point to a null-delimited list of field names and field values.

Troubleshooting Stored Fields

1. Make sure that the file that contains the field data was indexed after the Text Fields definition was added. Field data is extracted from documents when the documents are indexed.

2. Check that you are spelling the stored field name exactly as dtSearch is spelling it in the index. For example, punctuation or spaces in field names are generally removed when dtSearch creates a stored field. To see all of the stored fields for a document, search for the document in dtSearch and press Ctrl-Y in search results after opening the document.

Additional Information

For more information about stored fields, see the following topics:

"How to add custom fields to dtSearch Web search results"

"Define Text Fields" in the dtSearch manual

"Database and Field Searching" in the dtSearch Engine manual

"How to add fields to documents"

"How to index databases with the dtSearch Engine"


How to index a web site with the dtSearch Spider

dtSearch Home

Downloads

Quick Start

How to index a web site with the dtSearch Spider

Last Reviewed: November 26, 2002

Article: DTS0102

Applies to: dtSearch 6

dtSearch Spider

dtSearch Desktop and Network contain a built-in Web Spider for indexing and searching publicly-accessible Web sites. The dtSearch Spider automatically recognizes and supports HTML, PDF, XML, as well as other online text documents, like word processor files and spreadsheets. dtSearch Desktop and Network will display Web pages and documents that the Spider finds with highlighted hits as well as (for HTML and PDF) links and images intact.

Indexing and Searching with the Spider

To index a Web site, select "Add web" in the update index dialog box. Enter the name of the Web site, for example, www.frb.gov. Then select the crawl depth. The crawl depth is the number of levels into the web site dtSearch will reach when looking for pages. You could spider www.frb.gov to a crawl depth of 1 to reach only pages on the site linked directly to the home page. Or you could enter a crawl depth of 4 to reach four levels deep into the site.

The spider will not follow links from one web site that lead to a different web site. For example, if you ask the spider to index www.frb.gov, it will not follow a link on www.frb.gov to the www.fdic.gov web site. However, you can add any number of web sites to the list of sites to spider, so you could ask the spider to index www.frb.gov and www.fdic.gov.

After a search, dtSearch Spider will display retrieved HTML or PDF files with hit highlighting, and all links and images intact. The result looks and acts just like the original Web page, but with highlighted hits and additional navigation options ("next hit," "previous document," "next documents," etc.). dtSearch uses built-in HTML file converters to convert other text formats, such as word processor and preadsheet, to HTML for display with highlighted hits.

Indexing and search features such as automatic indexing with the Windows Task Scheduler, file filters, etc., are fully compatible with the use of the dtSearch Spider for scheduling automatic indexing updates, limiting a search to a specified file type, etc.

Troubleshooting -- Hit Highlighting is Off

The dtSearch Spider does not "capture" an indexed Web sites. To display a file indexed with the dtSearch Spider, dtSearch will return to the Web site to access the document. If the Web site has changed since the indexing, then hit highlighting will be on an incorrect word. To fix this, you will need to reindex the Web site.

For Web sites that do change content frequently, another alternative is to use a "Capture" program to grab the Web site. The result will be to store the contents of the Web site on your own PC. Then you can index and search it just like any other collection of HTML, PDF, XML, etc. files on your hard drive.

For capturing a Web site, two good products are:

SiteSnagger, www.pcmag.com (free)

Offline Explorer Pro, www.metaproducts.com ($50)

Troubleshooting - Passwords

Click Options > Preferences > Spider to set up user names and passwords for a site that requires logging in. The dtSearch Spider can handle password authentication but will not know how to log in with sites that require an HTML login form.

Related Topics

For more information on creating indexes, see "dtSearch Quick Start".

For more information on setting up dtSearch Web, see "dtSearch Web Quick Start".

For more information on using the dtSearch Spider to index dynamically-generated content, see "How to use dtSearch Web with dynamically-generated content".


How to make the '+' character searchable

dtSearch Home

Downloads

Quick Start

How to make the '+' (or '#') character searchable

Last Reviewed: September 6, 2001

Article: DTS0131

Applies to: dtSearch 6, dtSearch 5

Question

When I try to search for "C++" it finds "C". How do I make + a searchable letter?

How do I make # a searchable letter so I can search for "C#"?

Answer -- dtSearch Desktop 6

(1) Click Options|Preferences in dtSearch, select the Indexing Options tab, and click Edit Alphabet.

(2) Locate the "+" (or "#") character in the list of characters and click on it

(3) Set the Type to "Letter" and leave the two boxes under Letter Properties unchecked

(4) Click Save and then Close

(5) Rebuild your indexes with the Clear index before adding documents box checked.

The same procedure will work for most other punctuation characters. However, a few characters require special treatment: &, %, #, :, and ~. These characters are used in dtSearch search requests to indicate search features like fuzzy searching, phonic searching, etc. To make one of these characters searchable,

(1) Go through the steps above to make the character a "Letter".

(2) Exit dtSearch and run RegEdit to access your dtSearch registry entries. Open the key [HKEY_CURRENT_USER\Software\dtSearch Corp.\dtSearch\Settings]

(3) Change these registry settings to redefine the characters that you want to change:

"MacroChar"="@"

"FuzzyChar"="%"

"StemmingChar"="~"

"PhonicChar"="#"

"SynonymChar"="&"

"WeightChar"=":"

For example, to make the & character searchable, you could change the SynonymChar to $.

The following characters cannot be made searchable in dtSearch: (, ), *, and ?.

Answer -- dtSearch Engine 6

(1) Follow the steps above to customize the dtSearch alphabet file. The default name for the alphabet file is English.abc, but after changing the settings you can save the file under a different name.

(2) Use the Options object (in Visual Basic) or the DOptions or dtsOptions object (C++) to tell dtSearch to use the new alphabet file. These objects also provide properties that let you change the MacroChar, FuzzyChar, etc. as described above. (The Visual Basic Options object has these properties in version 6.04 but not in earlier versions.)

Answer -- dtSearch Desktop 5

(1) Click Options|Indexing Options|Edit Alphabet

(2) Locate the "+" character in the list of characters and click on it

(3) Set the Type to "Letter" and leave the two boxes under Letter Properties unchecked

(4) Click Save and then Close

(5) Rebuild your indexes with the Clear index before adding documents box checked.

The same procedure will work for most other punctuation characters. However, a few characters require special treatment: &, %, #, :, and ~. These characters are used in dtSearch search requests to indicate search features like fuzzy searching, phonic searching, etc. To make one of these characters searchable,

(1) Go through the steps above to make the character a "Letter".

(2) Exit dtSearch and open the file options.dat (in your dtSearch folder) in notepad.

(3) Look for these option settings:

MacroChar = @

FuzzyChar = %

PhoneticChar = #

StemmingChar = ~

SynonymChar = &

WeightChar = :

(4) Change the character that you want to make searchable to something else. For example, to make the & character searchable, you could change the SynonymChar to $. Save the file, exit notepad, and restart dtSearch.


How to segment long text, HTML, or XML files during indexing

dtSearch Home

Downloads

Quick Start

How to segment long text, HTML, or XML files during indexing

Last Reviewed: August 2, 2001

Article: DTS0170

Applies to: dtSearch 6.04 or later

Long text files often consist of many subsections that can be considered to be separate documents. For example, some email programs keep messages in a single long text file, with a marker such as "Date:" or a line of dashes separating the messages. Also, XML files with data converted from a database table will typically contain a series of records. The File Segmentation Rules feature in dtSearch provides a way to tell dtSearch to index each message or record as a separate document, without breaking up the original text file into numerous tiny files.

File Segmentation Rules work with Ansi text files, HTML, and XML. Word processor files such as Word or WordPerfect documents cannot be segmented using this method.

Creating File Segmentation Rules

To set up a file segmentation rule, click Options, Preferences in dtSearch and select the "File Segmentation Rules" tab. Each rule has the following parts:

Name

The name of a rule is used only to identify it in the File Segmentation Rules dialog box.

New document starts at

This is a marker that indicates when a new document begins. For email message files, this is often part of a message header such as "Date:" or "From:". To avoid incorrectly splitting a message, this marker should be as unique as possible.

How to check for document boundaries in text

Each line of the files a rule applies to will be compared against the marker under New document starts at. Three types of comparison are available:

  • Require exact match The entire line must exactly match the marker.

  • Match start of line The start of the line must match the marker.

  • Match regular expression The marker is interpreted as a regular expression. A document boundary occurs when the marker is found anywhere in a line. To require a marker to begin at the start of a line, precede it with the ^ character.

Ignore case

Match a document boundary even if the capitalization does not match.

First segment in a file is header for other segments

Check this box to have dtSearch insert the first segment in a file in every following segment. This option is useful when segmenting XML or HTML files, because it allows the HTML or XML header to be repeated for each segment.

Filename filters

For each rule, a filename filter determines which files the rule applies to. If more than one rule could apply to a particular file, the first one to match the filename is the one applied. A rule that does not have have a filter will be ignored.

Segmenting XML and HTML

If you use File Segmentation Rule with XML or HTML files, use the First segment in a file is header for other segments checkbox to make sure that the XML or HTML header is repeated for each segment. Otherwise, each segment will lack the <html> or <?xml> header that is necessary for correct identification of the file type.

Segment markers for XML and HTML are based on the raw text in the file, including tags and comments. For example, suppose you have an HTML file that looks like this:

<html>

<body>

<h3>Sample File</h3>

<!-- Segment -->

<p>This is the first segment</p>

<!-- Segment -->

<p>This is the second segment</p>

<!-- Segment -->

<p>This is the third segment</p>

You could use "<!-- Segment -->" the marker separating segments, even though these comments are not visible when you open an HTML file in dtSearch. Using this marker, and setting the option to make the first segment a header for the other segments, dtSearch would index the HTML file as three separate HTML documents:

<html>

<body>

<h3>Sample File</h3>

<!-- Segment -->

<p>This is the first segment</p>

<html>

<body>

<h3>Sample File</h3>

<!-- Segment -->

<p>This is the second segment</p>

<html>

<body>

<h3>Sample File</h3>

<!-- Segment -->

<p>This is the third segment</p>

Developer Information

File segmentation rules are stored in an XML file named fileseg.xml. You can create this file using the dtSearch Desktop GUI or by writing the XML in a program. To tell the dtSearch Engine to use file segmentation rules in your application, use the SegmentationRulesFile property of the Options object (VB, ASP, Delphi, or Java), or the segmentationRulesFile member of dtsOptions (C++).

To obtain the text of a segment, you can use the FileConverter object or DFileConvertJob and pass in the name of a segment, as retrieved in a search, as the filename.


How to use dtSearch or dtSearch Web with OCR

dtSearch Home

Downloads

Quick Start

How to use dtSearch or dtSearch Web with OCR

Last Reviewed: April 14, 2002

Article: DTS0167

Applies to: dtSearch Web 6

Scanned documents are usually stored as TIFF images, which are then converted using OCR (optical character recognition) into a text format such as HTML or Microsoft Word. Using dtSearch, dtSearch Desktop, or dtSearch Web, these text documents can then be indexed and made searchable. (See the dtSearch Web Quick Start for more information.)

In many cases, it is necessary to provide access both to the searchable text and the original image file, so that users can see exactly what the original document looked like. Because web browsers cannot display TIFF images without an image-viewing plug-in, the image files must be converted into another format if web access is needed.

Using PDF to combine images and text

The Adobe PDF file format provides two ways to combine images and text in a single file. First, the "image with hidden text" format stores the complete original TIFF images, along with the text obtained through OCR. The text is "hidden" because, when a user opens the PDF, the user only sees the scanned image, not the text. Because the text is also in the file, dtSearch can index and search it. After a search, dtSearch, or dtSearch Web, can highlight hits directly on the scanned image.

Another option for combining scanned images and text in a single PDF file uses small images for the parts of each scanned page that do not appear to be text. For example, a picture or a signature would be stored as a small image embedded in the page, while the rest of the page would be converted to text. This format produces much smaller files than the first alternative, because only a a few small images are stored for each page, instead of a complete image of the whole page. Additionally, the text detected through OCR often becomes more readable, because it is stored as text with font information rather than as an image.

The PDF format is ideal for use on the web because multiple pages of images and text can be combined into a single, compressed file; anyone with a web browser and the free Adobe Reader viewer can view the files; and the text can be searched using dtSearch Web.

Some OCR products that can generate PDF files from scanned images include:

  • Adobe Capture, www.adobe.com (Limited OCR capabilities are also available for Adobe Acrobat 5 in the "Paper Capture Plug-In", available for download from the Adobe web site.)

  • ScanSoft's TextBridge Pro Business Edition, http://www.scansoft.com (Note: The "Business Edition" is needed to generate PDF files. Other versions of TextBridge do not support creation of PDF files.)

  • Caere OmniPage Pro, http://www.scansoft.com

  • Abbyy Fine Reader, www.abbyy.com

  • DocuLex, www.doculex.com

  • AdLib, www.adlibsys.com

Searching PDF files with dtSearch Web, dtSearch Desktop, or dtSearch Network

Once you have created PDF files with one of the OCR products listed above, you can index and search them with dtSearch Web just like any other documents. After a search, dtSearch Web will display a list of retrieved documents. When a user clicks on one of the documents, dtSearch Web will display the PDF file, with hits highlighted. If the PDF file is in the "image with hidden text" format, dtSearch Web will highlight hits directly on the image.

For a demo, see http://support.dtsearch.com/ocr/


How to send large files to dtSearch Technical Support

dtSearch Home

Downloads

Quick Start

How to send large files to dtSearch Technical Support

Last Reviewed: December 4, 2000

Article: DTS0143

Applies to: All dtSearch Products

Small files (2 Mb or less) can be sent by email to tech@dtsearch.com.

Larger files will not work with our email system, so please post them to our public FTP site at ftp://support.dtsearch.com/incoming. Alternatively, if you have an FTP site, you can post the file on your FTP site and email the location of the file to tech@dtsearch.com.

1. Please ZIP the files if you are posting more than one, and use a ZIP file password if the files are confidential. While we have taken steps to make our FTP "incoming" folder more secure, a ZIP password will provide additional protection for your data.

2. Please send an email to tech@dtsearch.com when the file is posted so we know to copy the file and remove it from the FTP site.

"Access Denied"

The "incoming" folder is set up to allow public "write" access but no "read" access, so you will get an "Access Denied" error if you try to browse the incoming folder. The write-only access prevents other users from accessing your data after it has been posted. The write-only access is also necessary to protect our servers against abuse by companies that use open FTP sites to post illegal video and music files (a problem that we encountered repeatedly before we implemented the write-only access policy).

Using Internet Explorer for FTP

To use Microsoft Internet Explorer to send a file to our FTP site,

1. Open Internet Explorer

2. Enter the address ftp://support.dtsearch.com in the navigation bar

3. Open Windows Explorer

4. Arrange the windows so that you can see both Internet Explorer and Windows Explorer at the same time.

5. Find the file you want to upload in Windows Explorer

6. Drag the file from Windows Explorer onto the Internet Explorer window using the mouse, and drop it over the "Incoming" folder in Internet Explorer. Because of the security settings on our server, you may not be able to open the "Incoming" folder in Internet Explorer to confirm that the file was posted successfully. See the "Access Denied" note above for additional information.


Hyphenation options

dtSearch Home

Downloads

Quick Start

Hyphenation Options

Last Reviewed: December 20, 2000

Article: DTS0154

Applies to: dtSearch 5, dtSearch 6

The dtSearch Engine supports four options for the treatment of hyphens when indexing documents: spaces, searchable text, ignored, and "all three".

For most applications, treating hyphens as spaces is the best option. Hyphens are translated to spaces during indexing and during searches. For example, if you index "first-class mail" and search for "first class mail", "first-class-mail", or "first-class mail", you will find the phrase correctly.

Effect on Indexes

When an index is created, the hyphenation option currently in effect is stored in the index, and cannot be changed without re-creating that index. Therefore, the hyphenation option you select affects any indexes you create in the future, but it does not affect indexes that already exist.

When a user searches an index, the hyphenation option for that index applies to the user's search request.

How the options apply during indexing

During indexing, dtSearch extracts a stream of words from each document, and each word is assigned a number that represents that word's position in the file. The first word is assigned the position "1", the second word is assigned the position "2", and so forth. Consider a document that starts with the sentence, "I sent it by first-class mail". The following describes how the document would be treated under each of the hyphenation options:

1. Hyphens treated as spaces:
I (1), sent (2), it (3), by (4), first (5), class (6), mail (7)

2. Hyphens treated as searchable characters:
I (1), sent (2), it (3), by (4), first-class (5), mail (6)

3. Hyphens ignored:
I (1), sent (2), it (3), by (4), firstclass (5), mail (6)

4. All three:
I (1), sent (2), it (3), by (4), (5) first-class, (6) first-class, (5) first, (6) class, (5) firstclass
(6) firstclass, (7) mail

How the options apply during searching

During a search, dtSearch translates the search request according to the hyphenation option for the index being searched. For example, if you search for "first-class" in an index created with hyphens treated as spaces, the search request is translated into "first class".

During a search of an index created with the "all three" option, the search request is not modified. For example, if you search for "first-class", dtSearch will not search for "firstclass" or "first class".

Effects of the "all three" option

The "all three" option has one advantage over treating hyphens as spaces: it will return a document containing "first-class" in a search for "firstclass". Otherwise, it provides no benefit over treating hyphens as spaces, and it has some significant disadvantages:

1. The "all three" option generates many extra words during indexing. For each pair of words separated by a hyphen, six words are generated in the index.

2. It can produce unexpected results in searches involving longer phrases or words with multiple hyphens. With the "all three" option enabled, the sequence "a-b-c" would be indexed as: a (1), ab (1) a-b (1), ab (2), a-b (2), b (2), b-c (2), bc (2), b-c (3), bc (3), c (3). Thus, "a b c" would be found as would "a bc" or "ab c", but not "a-b-c" or "a-bc" or "ab-c". (To prevent the number of permutations from becoming excessive, dtSearch only permutes one hyphen at a time.)


Installation Problems

dtSearch Home

Downloads

Quick Start

Installation Problems

Last Reviewed: May 15, 2002

Article: DTS0163

Applies to: dtSearch 6

dtSearch 6 uses a new Microsoft component, the Windows Installer service, for installation. If you have Windows 2000 or Microsoft Office 2000, you already have the Windows Installer. If not, the dtSearch setup program should install it for you automatically.

Message: "Error 1719. The Windows Installer Service could not be accessed. You may be running in Safe mode or Windows Installer may not be correctly installed."

(1) If you are not logged in as an administrator, log in as Administrator and try running the setup again.

(2) You may have a registry error in your Windows Installer setup. Please see this article on the Microsoft Web site for more information:

Q315346 "Error 1719. The Windows Installer Service Could Not Be Accessed" Error Message When You Add or Remove Program

http://support.microsoft.com/default.aspx?scid=kb;EN-US;q315346

Message: "A Service Pack with a newer version of the Windows Installer is required"

If you get a message during installation that a service pack with a newer version of the Windows Installer is required, please do the following:

(1) Close the dtSearch Setup program, reboot, and run dtSearch setup again.

(2) If this does not solve the problem, click here for instructions on getting the Windows Installer from Microsoft and installing it directly. Once the Windows Installer is set up correctly, run dtSearch setup again.

Message: "Internal Error: 2755.1632"

Please see this article in the Microsoft KnowledgeBase for conditions that can cause this internal error in the Windows Installer, and a workaround to resolve the problem:

Q274635 Money 2001 Error Message: Internal Error: 2755.1632

http://support.microsoft.com/support/kb/articles/Q274/6/35.ASP

Message: "Internal Error 2381"

This can be caused by NTFS folder settings under Windows NT. Please see this article in the Microsoft KnowledgeBase for more information:

Q237578 OFF2000: Internal Error 2381 Running Office Setup on Windows NT

http://support.microsoft.com/support/kb/articles/q237/5/78.asp

Message: Error 1327, Invalid Drive: drive:\, Installation ended prematurely because of an error.

This message can be caused by changes to the location of system folders, changes to drive letters made in Disk Administrator, or lack of network access to a system folder. Please see these articles in the Microsoft Knowledgebase for more information:

Q282183 Works 2001: Error 1327, Invalid Drive

http://support.microsoft.com/support/kb/articles/Q282/1/83.ASP

Q217666 OFF2000: Error 1327 Invalid Drive During Office 2000 Installation

http://support.microsoft.com/support/kb/articles/q217/6/66.asp

Q290356 OFFXP: "Error 1327. Invalid Drive: C:\..." When You Run Office XP Setup

http://support.microsoft.com/support/kb/articles/q290/3/56.asp

Message: "The feature you are trying to use is on a network resource that is unavailable"

This message can appear during dtSearch Setup if you are upgrading from a previous version and your computer's Windows Installer database is damaged. When dtSearch Setup tries to uninstall the previous version of dtSearch, the Windows installer cannot find information about the previous installation and displays this message. The same message will appear if you try to uninstall dtSearch from the Add/Remove Programs tool in Control Panel.

Damage to the Windows Installer database can be caused by deletion of files in the Windows\Installer folder, or by registry "cleanup" utilities that may remove necessary file links in the registry. When Windows Installer cannot find its internal copy of the data, it looks for the data in the folder originally used to run setup, which may be a temporary folder that no longer exists. This is why the message often includes a reference to an MSI file in a non-existent folder.

To resolve this problem, use the Windows Installer Cleanup utility (see below) to remove dtSearch references from the Windows Installer database, and then run dtSearch Setup again.

Q238413 Windows Installer CleanUp Utility

http://support.microsoft.com/support/kb/articles/Q238/4/13.ASP

General Troubleshooting

Please see this Microsoft Knowledgebase article for troubleshooting suggestions if the Windows Installer setup does not work correctly.

Q260404 HOWTO: Troubleshoot Windows Installer Engine Problems

http://support.microsoft.com/support/kb/articles/Q260/4/04.asp

To clean up a damaged Windows Installer database, use the Windows Installer CleanUp Utility. The Microsoft article below describes how to obtain and use this tool:

Q238413 Windows Installer CleanUp Utility

http://support.microsoft.com/support/kb/articles/Q238/4/13.ASP


Index size

dtSearch Home

Downloads

Quick Start

Index size

Last Reviewed: July 4, 2004

Article: DTS0142

Applies to: dtSearch 5, dtSearch 6

Contents

Overview

How big can an index get?

How can the size of an index be minimized?

What is the effect of index size on searching performance?

What is the effect of index size on indexing performance?

What is the effect of document type on index size?

Overview

dtSearch works by building an index of your documents. You can create as many indexes as you want, and in a search you can search any or all of them by clicking on the ones you want to search.

An index is generally about 1/4 to 1/3 the size of the original documents. The ratio of the index size to the size of the original documents depends on the type of documents, the size of the documents, and certain indexing options (discussed below). The ratio is better for word processing documents (which contain less text per kilobyte than plain text files) and large files (which reduce the effect of the per-document overhead).

Each index can hold up to 4-8 gigabytes of documents. If you fill up an index, dtSearch will display an "Index is full" message and stop adding documents to the index. Larger document collections can be indexed using multiple indexes. The Forensic Indexer included in dtSearch Desktop (forix.exe) will automatically generate as many indexes as are needed to index a very large document collection.

A new dtSearch index format, currently in beta testing, will increase the capacity of a single index to over 1,000 gigabytes. For more information on this new format, see http://www.dtsearch.com/index7.html.

Apart from the 4-8 gigabytes per index limit, there is no specific limit on the number of documents you can have in an index or the number of words in an index.

How big can an index get?

A single dtSearch index can index about 4-8 gigabytes of text. When an index is full, indexing stops and no additional documents can be added to the index. Compressing an index may reduce the size of an index, making it possible to add more documents.

A dtSearch index consists of a series of index_*.ix files in a folder. The actual limit on index size is that none of these files can exceed 2 gigabytes in length.

Larger document collections can be indexed using multiple indexes. The Forensic Indexer included in dtSearch 6.06 (forix.exe) will automatically generate as many indexes as are needed to index a very large document collection.

How can the size of an index be minimized?

The following are some option settings that can be used to reduce the size of a dtSearch index.

Binary Files

The binary files option setting controls the way dtSearch treats files in a format that it does not recognize as a document. There are three options: (1) index the files completely, (2) filter out only the text of the files, and (3) skip the files entirely.

Filtering or skipping binary files can greatly reduce index size, and improve indexing speed. Indexing binary files completely can have a large effect on index size because binary files do not contain a normal mix of words. If treated as text, they produce a large number of unique, random text sequences, which then bloat the word list portion of the index disproportionately.

Filtering is also more effective at extracting text than simply indexing binary files entirely. This is because text may be present in a variety of formats (for example, blocks of Unicode text are often mixed with blocks of single-byte text), and the filtering algorithm can identify and decode these segments.

Exclude filters are another good way to minimize the number of binary files included in an index.

Title Size

If the documents being indexed are small, the per-document overhead in an index may be a relatively large portion of the total index size. For each document, dtSearch stores the filename and location, the modification date, size, and other properties, as well as a "title" which is usually the first 80 characters of text from the file. To reduce the size of the index, the title size can be changed to a smaller value. Additionally, reductions in the size of the filenames in the index, including the folder name, will save space.

Noise Words

For text files, a noise word list can reduce the size of an index by eliminating common words like "the" or "if". By default, dtSearch will index documents using a noise word list for the English language. The noise word list is a plain text file named noise.dat that can be edited using Notepad to add additional words.

Numbers

dtSearch has an indexing option to skip indexing numbers. If the documents being indexed contain many numbers, and if these numbers do not have to be searchable, this setting can reduce index size considerably.

Hyphens

The option setting to treat hyphens as spaces produces smaller indexes than any of the other option settings.

What is the effect of index size on searching performance?

A dtSearch search essentially consists of two steps: (1) looking up the words in the search request, and (2) enumerating the documents that match that request.

The word lookup step is usually very quick and takes a small fraction of the total time required for the search. A wildcard at the beginning of a search term (for example, a search for "*abc") can be slow because dtSearch uses letters at the start of a word to implement fast word searches, and dtSearch cannot do this if the start of the search term is unknown.

The time required for the second step, enumerating the documents, depends on the number of documents found rather than the size of the index. (For developers, using the dtsSearchDelayDocInfo flag can minimize the time required for this step, making searches that retrieve many files much faster. For more information, please see "Optimizing search performance with the dtSearch Engine")

What is the effect of index size on indexing performance?

The dtSearch Engine uses a mechanism for concurrent access that allocates all of the cost of concurrent access to the user who opens the index for write access, favoring search speed over indexing speed. Users who open an index for read access only (searching) will not have to wait for locked records to be released, no matter how many users are searching at a time and whether or not the index is being updated or compressed. To provide this type of concurrent access, about 20-30 percent of the index data must be copied to support concurrent access each time an index is opened for write access. This overhead is incurred only once at the start of an indexing job and depends on the size of the index when the job starts, not on the number of files to be indexed. Because of this per-update overhead, it is much more efficient to index documents in larger batches.

What is the effect of document type on index size?

The size of the index as a fraction of the original document size depends on how much text the document contains per kilobyte of data. The more text the document contains, the larger the index. For example, if you index a 20k Microsoft Word document and a 20k text file, the 20k text file will add much more data to the index than the Word document. A 20k Word document will consist mostly of formatting information, leaving only 10k or less of text, so it adds less than half as much data to the index as the text file.

In some cases, it is even possible for the index to be larger than the original documents. This can happen if the documents are in a compressed format such as ZIP archives or a PDF files (PDF files store text in a compressed stream).

Indexes of database files, such as MDB (Microsoft Access) or DBF (XBase) files, are also usually a large fraction of the original document size. dtSearch indexes each record of a database file as a separate document. As a result, the per-document overhead in the index becomes a much larger factor in the index size.


Ordering Information

dtSearch Home

Downloads

Quick Start

dtSearch Ordering Information

Last Reviewed: January 26, 2002

Article: DTS0171

How to Order

Click here to order dtSearch through our on-line store.

For information about upgrading from dtSearch 5, click here.

For information about U.S. Federal Government purchases, click here (dtSearch is on the GSA schedule).

Company Information

dtSearch Corporation

6852 Tulip Hill Terrace

Bethesda MD 20816

FEI# 54-157-7887

Phone 800-483-4637 (301-263-0731)

Fax 301-263-0781

Email: sales@dtsearch.com

GSA Contract GS-35F-4686G

Purchase Order Terms

Purchase orders may be sent by mail, email, or fax.

Orders are delivered both electronically and by mail. To obtain electronic delivery, please provide an email address with your order. The boxed product, including a CD and manual, will be delivered within one week.

Pricing includes standard shipping within the US

Our payment terms are Net 30 days.

Tendered and payable in US Funds only.


Security passwords on PDF files

dtSearch Home

Downloads

Quick Start

Security passwords on PDF files

Last Reviewed: July 6, 2002

Article: DTS0161

Applies to: dtSearch 6

PDF files can have two kinds of passwords:

(1) A password that prevents certain operations (printing, editing, etc.) only. This type of PDF file can be opened in Adobe Reader with no need to manually enter a password.

(2) A password that prevents any access to the file unless the user enters a password before opening it in Adobe Reader.

dtSearch versions 6.10 and later can index password-protected PDF files of the first type (the kind that can be opened in Adobe Reader without entering a password), as long as the encryption level is 40-bit RC4. If a password must be entered to open the file in Adobe Reader, or if the encryption level is 128-bits, then dtSearch cannot access the text in the PDF file..


'Too many words retrieved in index' message

dtSearch Home

Downloads

Quick Start

"Too many words retrieved in index" message

Last Reviewed: November 29, 2000

Article: DTS0119

Applies to: dtSearch 5, dtSearch 6

Symptoms

Executing a search causes dtSearch to display a "Too many words retrieved in index" message.

Cause

dtSearch limits the number of unique words (not hits) that can be retrieved in a search. A search for * or *a* that retrieves thousands of words would exceed this limit.

Additional Information

The limit on the number of unique words that can be retrieved in a single search is determined by the maxWordsToRetrieve option setting in dtSearch. In dtSearch 5, the default limit is 500. In dtSearch 6, the default limit is 64k.

How to change the maxWordsToRetrieve option setting:

dtSearch Desktop 5 Edit the options.dat file in notepad and change the "maxWordsToRetrieve = " line in the file.

dtSearch Desktop 6 A registry entry can be used to change this in dtSearch. The key is:

HKEY_CURRENT_USER\Software\dtSearch Corp.\dtSearch\Settings\maxWordsToRetrieve

dtSearch Engine (Visual Basic) Use the Options.MaxWordsToRetrieve property

dtSearch Engine (C/C++) Use the maxWordsToRetrieve member of dtsOptions

dtSearch Engine (Java) Use the maxWordsToRetrieve member of the Options object

The limit is necessary to ensure efficient use of memory in searching. For each unique word matched by a search term, dtSearch reserves about 1k for bookkeeping during the search. The 500 word limit means that dtSearch allocates 500k to process a search request.

In dtSearch 6, the space required for words in a search request is allocated as needed, so the maxWordsToRetrieve limit specifies the maximum amount of memory that will be allocated for a search, but dtSearch will only allocate what it actually needs during each search.

Unless you search for a term like *a*, it is very unusual to get anywhere near this limit even with huge databases, because the number of words you retrieve is generally a function of the specificity of the search rather than the size of the database.


Unicode support in dtSearch 6

dtSearch Home

Downloads

Quick Start

Unicode support in dtSearch 6

Last Reviewed: November 15, 2003

Article: DTS0140

Applies to: dtSearch 6

dtSearch 6 supports indexing and searching Unicode text. This article will describe what is and is not covered in this support, and will provide additional information about how dtSearch Unicode support works with different operating systems and document types.

Contents

Background

dtSearch Support for Unicode

File Formats

Language Issues

Alphabet Customization

Troubleshooting Encoding Problems

See also: International Language Features in dtSearch

Background

Unicode. Unicode is a specification that allows text in any language to be encoded in a consistent way. Instead of the 255 characters allowed by the Ansi character set, the Unicode character set can express over 65,000 characters. Detailed information on the Unicode specification is available at www.unicode.org.

UTF-8. UTF-8 is a widely-used, compact encoding of Unicode text that preserves all information in a Unicode string. For example, Java uses UTF-8 to provide Unicode support. In UTF-8, characters between 1 and 128 are encoded as Ansi characters 1 through 128. Other characters are encoded using character values greater than 128. UTF-8 encoded strings do not contain embedded NULL characters. Additional information on UTF-8 is available at www.unicode.org.

Operating Systems. Windows NT, Windows 2000, and Windows XP support Unicode, while Windows 95, Windows 98, and Windows ME do not. Unicode support in an operating system generally means that (1) filenames and folder names can use Unicode characters, and (2) the user interface supports display of Unicode characters. For example, under Windows 2000 you can enter Greek, Arabic or Hebrew characters into the text controls in a dialog box.

Fonts. If characters are appearing as small rectangles, your system font may not support display of characters in the language you are searching. Microsoft Office contains a useful "Arial Unicode MS" font with coverage of nearly every character in every language included in the Unicode standard. Use Windows "Display Options" to select this font for use in menus and message boxes. In dtSearch Desktop, use Options|Preferences, Display Options to select the font used to display documents in the dtSearch viewer window.

Keyboard and Character Sets. To add support for additional languages and keyboards to your Windows 2000, XP, or 2003 Server system, use the Regional Options tool in Control Panel. First, add the additional languages under the "General" tab. Second, select the Input Locales tab and click Change. This will let you add support for non-English keyboards and languages. For more information, see this article on the Microsoft web site (it covers both Windows NT and Windows 2000):

Q177561 HOWTO: Add and Enable Additional Languages in Windows

http://support.microsoft.com/default.aspx?scid=kb;EN-US;q177561

The Windows charmap.exe program provides another way to enter non-English text. To access it, click Start > Programs > Accessories > System Tools > Character Map. Another helpful tool for working with non-English text is the on-screen keyboard: Start > Programs > Accessories > Accessibility > On-screen Keyboard.

dtSearch Support for Unicode

dtSearch Unicode support means that dtSearch can index and search documents containing Unicode-encoded data. dtSearch Unicode support is built into the dtSearch Engine and works on all 32-bit versions of Windows, including Windows 95, Windows 98, and Windows ME (which do not themselves have Unicode support). dtSearch can support Unicode even under non-Unicode versions of Windows because the necessary data is built into the dtSearch Text Retrieval Engine.

dtSearch supports 8-bit (UTF-8) and 16-bit (UCS-16) encodings of Unicode. UCS-32, a 32-bit encoding of Unicode that can express characters beyond the original 65,000 character limit in Unicode, is not yet supported.

File Formats

Microsoft Office

dtSearch can automatically recognize Unicode data in Microsoft Word, Excel and PowerPoint files. Some Microsoft Word 6 documents contain text without encoding information, so dtSearch may not interpret these files correctly. Saving these files as Word 2000 or Word XP files will add the necessary encoding information.

HTML and XML

An HTML or XML file can include Unicode data if the HTML file uses the UTF-8 encoding. HTML files that are stored with the UTF-8 encoding contain a META tag in the beginning of the file that looks like this:

<meta http-equiv="Content-Type" content="text/html; charset=utf-8">

If the file uses a different encoding, the META tag will contain a different charset= value, like this:

<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">

HTML editors such as Microsoft FrontPage generally have an option that lets you control the encoding used to store HTML files.

dtSearch 6 can index and search Unicode data in UTF-8 encoded HTML files and can also recognize many other HTML encodings.

Text Files

Text files do not contain any encoding information, so dtSearch has to infer the encoding. Unless otherwise specified, dtSearch assumes that a text file uses the Windows Ansi encoding. Using the Options|File Types dialog box in dtSearch Desktop, you can set up rules to tell dtSearch to treat certain text files as UTF-8 or as DOS Text files (using the old "OEM" character set from from DOS programs).

WordPerfect

WordPerfect files use the WordPerfect Character Set to express non-English text. dtSearch 6 converts WordPerfect Character Set data to Unicode for indexing, so non-English text in WordPerfect files is supported.

PDF

dtSearch can index and search Unicode characters in some, but not all, PDF files. Unlike other document formats, which usually contain text in some form, PDF files are essentially drawing instructions that provide information necessary to print a document on a printer or to draw it on the screen. Many PDF files contain character encoding information in addition to the drawing instructions, so the content of the PDF file can be converted back to text. In these types of PDF files, you can use the Text Select tool in Adobe Reader to select a block of text, copy the text to the clipboard, and paste it into another program like Notepad or Microsoft Word. If you can you use the Text Select tool in Adobe Reader to copy and paste text from a PDF file, it means that the file does contain meaningful character encoding information, and so dtSearch will probably be able to index and search the file correctly.

In some PDF files, however, only the drawing instructions are present, and the encoding information is either absent or random. As a result, there is no way to convert the file back to text. In these types of PDF files, Adobe Reader's Text Select tool will either (a) fail to work entirely, or (b) will copy text to the clipboard that is meaningless. dtSearch cannot index or search this type of PDF file, because the file is really just a picture of text but does not contain any words.

Language Issues

Chinese, Japanese, Korean

Text in Chinese, Japanese, and Korean can be stored in, or converted to, Unicode, so dtSearch can search for words in these languages just as it can search for words in other languages. However, while dtSearch can search for literal word matches (or wildcard or fuzzy matches), there are some limitations on the support in dtSearch for Chinese, Japanese, and Korean text, described below.

(1) Dictionary-based word breaking

Some documents store text in a way that does not separate the words with spaces. Instead, all of the text in a document is run together and a language-specific dictionary is needed to find word breaks. dtSearch does not have the ability to identify word breaks in these documents, because it does not include any language-specific dictionaries.

(2) Variations in character forms and scripts

In these languages, the same text can be presented in different ways depending on the context. dtSearch will search for a word as it is provided in the search request and does not generate additional grammatical or script variations for words in Chinese, Japanese, and Korean.

For background information on handling text in these languages, and resources for software developers, see the CJK Institute site at www.cjk.org.

The dtSearch Engine has an API that can be used to integrate with dictionary-based language analyzers from companies such as Basis Technologies . For more information, see How to integrate the dtSearch Engine with a language analyzer.

Word Prefixes and Suffixes (Arabic)

In some languages such as Arabic, the surrounding context for a word (my, your, the, a, masculine/feminine, etc.) can be expressed as characters added in front of or behind the word. For example, "the apple" or "my apple" would not be two words but would be different prefixes or suffixes added to "apple". To search for text in these languages, adding a * in the front and back of the word will pick up most of the variants, like this: *apple*.

Accent-insensitive indexing

dtSearch 6 can create indexes that are either "accent-sensitive" or "accent-insensitive." An accent-insensitive index converts characters, wherever possible, to a "base" character which is either one of the letters A-Z or one of the digits 0-9. Accent-insensitive indexes are generally easier to use because they ensure that a document will be found even if the author omitted an accent, or if the user entering a search request omitted an accent, in typing a word. The following are examples of the character conversions done in an accent-insensitive index:

Character | Unicode value | "Base" Character
Å | U+00C5 | A
ç | U+00E7 | c
Superscript 8 | U+2078 | 8
Arabic-Indic digit 8 | U+0668 | 8

In an accent-sensitive index, each letter is converted to lower case where possible but otherwise characters are indexed using their Unicode values. In an accent-sensitive index, ç and c would be considered different letters, and a search for one would not find the other.

Alphabet Customization

dtSearch versions 5 and earlier used "alphabet" files with a .ABC extension to provide for customization of the handling of 8-bit characters. This made it possible to define, for each character in the range from 33 to 255, whether it was a letter or not and the rules for capitalization and accents. dtSearch 6 still uses .ABC files, but only for characters in the range from 33 to 127. All other characters are handled according to the definitions in the Unicode character tables.

Troubleshooting Encoding Problems

No accented letters appear in the indexed word list

Indexes are created accent insensitive by default. This means that all letters are converted to a-z whenever possible, and a search for é is considered equivalent to a search for e. Therefore, no accented letters will appear in the indexed word list. To make an accent-sensitive index, check the "accent sensitive" option in the Create Index dialog box when you create the index.

Text files appear incorrectly in dtSearch, and the words in the indexed word list have missing or scrambled accented characters

Text files do not contain any encoding information, so dtSearch has to infer the encoding based on the rules in the Options|File Types dialog box. By default, all text files are assumed to be in the Windows Ansi encoding. To change the way dtSearch handles a text file, set up a rule in the File Types dialog box telling dtSearch which files are covered by the rule and the way the files should be interpreted.

Non-English characters in HTML files appear incorrectly in dtSearch or in the indexed word list

HTML files should have a header tag that describes their encoding. (See the note above on HTML for examples.) If an HTML file does not have this header, dtSearch has to guess the encoding, and may guess incorrectly. To prevent this, save your HTML files with a defined encoding so they will be unambiguous. The most efficient encoding to use with dtSearch is UTF-8.


What file formats does dtSearch support?

dtSearch Home

Downloads

Quick Start

What file formats does dtSearch support?

Last Reviewed: December 12, 2004

Article: DTS0103

Applies to: dtSearch 6.4

Text and Email Formats

dtSearch can automatically recognize, index, search and display documents, including graphic marking of "hits" and multiple "hit" and file navigation options, in the following current formats. dtSearch uses its own built-in file viewers for document parsing and display, unless otherwise noted.

Adobe Acrobat (PDF) all versions through version 6

Ami Pro

Ansi Text

CSV (Comma-separated values)

EBCDIC

EML files (emails saved by Outlook Express)

Eudora MBX message files

HTML

MBOX email archives

MHT archives (HTML archives saved by Internet Explorer)

MIME messages

MSG files (emails saved by Outlook)

Microsoft Excel (through Excel 2003)

Microsoft Outlook Express 5 and 6 (*.dbx) message stores

Microsoft PowerPoint 97, PowerPoint 2000, PowerPoint XP, PowerPoint 2003

Microsoft Rich Text Format

Microsoft Word for DOS

Microsoft Word for Windows (through Word 2003)

Microsoft Works

Multimate Advantage II

Multimate version 4

Treepad HJT files

Unicode

WordPerfect (all versions from 5.0 through WordPerfect 2002)

WordStar versions 4, 5, 6

WordStar 2000

Write

XBase (including FoxPro, dBase, and other XBase-compatible formats)

XML

For information on indexing Microsoft Outlook messages, click here.

In the dtSearch search results viewer, HTML and PDF documents appear with all formatting and embedded images and links exactly as in the original document. Other file types are converted to HTML for display, with varying levels of formatting preserved.

Media Formats

Indexing of metadata (Author, Title, etc.) in MP3, ASF, and WMV files

Archive Formats

GZIP and TAR (.gz files)

ZIP versions 1.x and 2.0 (each file in a ZIP archive will be indexed as a separate document)

Database Formats

Using 32-bit ODBC, dtSearch can also index and display records in Access and Paradox databases. Each record is treated as a separate document. (XBase databases are indexed without using ODBC.)

For information on indexing SQL databases, click here.

Older Word Processor Formats

dtSearch can index and display, but cannot automatically recognize, documents in the following formats:

WordPerfect 4.2

WordStar versions before 4

XyWrite

Ascii Text

Choose File Types in the Options in dtSearch menu to tell dtSearch how to recognize these types of files.

Other File Formats

dtSearch will still index, search, and display other file formats, but they will be treated as binary file types. In other words, all binary codes, etc. will be displayed along with the text. dtSearch can also use a proprietary binary file filtering algorithm to clean up these file formats. For more information see Indexing Options in the dtSearch help file.

For legacy file types in which multiple messages or log entries are stored in one very large text file, use the dtSearch File Segmentation Rules feature to tell dtSearch how to break up the file into multiple logical subdocuments. For more information, see File Segmentation Rules in the dtSearch help file.

An AutoCAD searching product, based on the dtSearch Engine, is available from DocuPoint. For more information please see: http://www.dtsearch.com/CS_docupoint.html

Image Formats

dtSearch Desktop can display images in the following formats:

BMP

EPSF

GIF

IMG

JPEG

PCX

PNG

TIFF

Targa

WMF

WPG (WPG version 1.0 only)

When viewing multipage images, use PgUp and PgDn to navigate between the pages. The dtSearch image viewer also includes viewing options such as Zoom In, Zoom Out, Invert, Rotate, etc.


Accessibility -- dtSearch Desktop

dtSearch Home

Downloads

Quick Start

Accessibility -- dtSearch Desktop

Last Reviewed: March 20, 2002

Article: DTS0175

Applies to: dtSearch 6.06 and later

Customizing Fonts

The dialog box fonts in dtSearch Desktop are based on the standard Windows dialog box font, which can be changed using the Display Options applet in Control Panel.

The font used in the Search dialog box to display the search request and the scrolling word list can be changed to make these items more readable. To change these fonts, click Options > Preferences > Other Options and change the setting under Search dialog box fonts.

By default, the text display settings for Internet Explorer are used to display retrieved documents, and hits are highlighted in yellow. To change the font used to display a retrieved document, or to change the way hits are highlighted, click Options > Preferences > Document Display.

Keyboard Navigation

Document Windows

Key | Purpose
Spacebar | Next hit in document
Backspace | Previous hit in document
Tab | Switch to search results window
Ctrl+Spacebar | Next document in search results
Ctrl+Backspace | Previous document in search results
Ctrl+Home | Top of document
Ctrl+End | End of document
Ctrl+K | Advanced copy
Ctrl+P | Print document or, if text is selected, print selected block.

Search Results Windows

Key | Purpose
Enter | Open current document
F8 | Launch current document
Tab | Switch to document window
Ctrl+P | Print document or, if text is selected, print selected block.

Search Dialog box

Key | Purpose
Alt+1 | Select Search Request pane
Alt+2 | Select More Search Options pane
Alt+3 | Select Search History pane

Other Keyboard Shortcuts

Key | Purpose
Ctrl+S | Search
Ctrl+Shift+S | Search in a new window
Ctrl+H | Search history
Ctrl+I | Index manager
Ctrl+U | Update index


Document is not found after a search

dtSearch Home

Downloads

Quick Start

Document is not found after a search

Last Reviewed: January 5, 1999

Article: DTS0113

Applies to: dtSearch 5, dtSearch 6

Symptoms

Search fails to retrieve a file that should be found

Possible Causes

- A search setting is excluding the file

- The file was not indexed

- The file does not contain the text you are searching for

Troubleshooting Steps

1. In the Search dialog box, check the bottom of the dialog box for filename or other criteria, possibly left from a previous search, that could be excluding the file you are searching for.

2. Check that the file was indexed.

- Try searching for another word that you know is in the file.

- Click Index|Index Manager and check the folders and filename filters for the index, and also make sure that the number of documents in the index is not 0.

3. Check that the file contains the word you are searching for. In dtSearch, click File|View File to open the document. Some files contain images of words rather than the words themselves. For example, a PDF file may contain an image of a scanned page but no text. Similarly, an HTML file may display text in a banner image. To convert images into searchable text, use an OCR product such as Xerox TextBridge, Adobe Capture, or Visioneer Pro OCR 100.


Does dtSearch have any size limitations?

dtSearch Home

Downloads

Quick Start

Does dtSearch have any size limitations?

Last Reviewed: August 1, 1999

Article: DTS0134

Applies to: dtSearch 5.x, dtSearch 6

dtSearch works by building an index of your documents. You can create as many indexes as you want, and in a search you can search any or all of them by clicking on the ones you want to search.

Each index can hold up to 4-8 gigabytes of documents. If you fill up an index, dtSearch will display an "Index is full" message and stop adding documents to the index.

Apart from the 4-8 gigabytes per index limit, there is no specific limit on the number of documents you can have in an index or the number of words in an index.

For more information on factors affecting index size, see article DTS0142 "Index Size"


dtSearch 6 Quick Start

dtSearch Home

Downloads

Quick Start

dtSearch 6 Quick Start

Last Reviewed: November 3, 2003

Article: DTS0160

Applies to: dtSearch Desktop 6

dtSearch can search gigabytes of text in a second. It does this by building an index that stores the location of each word in your files. Therefore, to get started with dtSearch, the first step is to build an index of your documents.

Indexing Documents

1. Choose Create Index from the Index menu.

2. In the Create index dialog box, enter a name for the index and click OK.

3. dtSearch will ask if you want to add documents to the index. Click Yes to go to the Update Index dialog box.

4. Click Add folder... to add a folder to the list of folders to index.

5. Click Add web... to index a site using the dtSearch Spider.

6. Click Start Indexing to begin adding documents to your index.

dtSearch automatically recognizes popular file types, including word processor files, databases, spreadsheets, PDF, RTF, ANSI text, ZIP, XML, and HTML. For a complete list of the file formats that dtSearch supports, see Supported File Types.

Searching using the Index

1. Click the Search button [image] on the dtSearch button bar, or press Ctrl-S, to open the Search dialog box.

Indexes to search
The top right of the dialog box shows a list of the indexes you have created; select one or more to search.

Indexed word list
The top left of the dialog box shows a list of the words in the currently selected index -- "Adobe Acrobat" in the example above. If more than one index is selected for searching, you can select the index to display in the word list by clicking the down arrow above the word list.

2. Enter a search under Search Request.

3. Select any items under Search features (such as fuzzy searching) that you want to use.

4. Click Search to begin the search.

Search Types

Any words: use quotation marks around phrases, put + (plus) in front of any word or phrase that is required, and - (minus) in front of a word or phrase to exclude it. Examples:

banana pear "apple pie"

"apple pie" -salad +"ice cream"

All words: like an "any words" search except that all of the words in the search request must be present for a document to be retrieved.

Boolean search: a group of words, phrases, or macros linked by connectors such as AND and OR that indicate the relationship between them. Examples:

Search Request | Meaning
apple and pear | both words must be present
apple or pear | either word can be present
apple w/5 pear | apple must occur within 5 words of pear
apple not w/5 pear | apple must occur, but not within 5 words of pear
apple and not pear | only apple must be present
name contains smith | the field name must contain smith
apple w/5 xfirstword | apple must occur in the first five words
apple w/5 xlastword | apple must occur in the last five words

If you use more than one connector, you should use parentheses to indicate precisely what you want to search for. For example, apple and pear or orange juice could mean (apple and pear) or orange, or it could mean apple and (pear or orange).

You can use variable term weighting in a search request to weigh some words more heavily than others in ranking search results. Example: apple:5 and pear:3

Search Features

Stemming searches other grammatical forms of the words in your search request. For example, with stemming enabled a search for applies would also find apply, applying or applied.

Phonic search finds words that sound similar to words in your request, like Smith and Smythe.

Fuzzy search sifts through scanning and typographical errors. Fuzziness adjusts from 1 to 10 depending on the degree of misspellings. A search for alphabet with a fuzziness of 1 would find alphaqet; with a fuzziness of 3, it would find both alphaqet and alpkaqet.

Synonym searching tells dtSearch to use a thesaurus to automatically expand a search to include synonyms or related concepts, including three optional levels. (Click Browse thesaurus to browse the entire thesaurus.)

To see how stemming, phonic searching, fuzzy searching or wildcards will affect your search, click the Browse Words button.

To browse the thesaurus used for synonym searching, click the Browse Thesaurus button.

Click the Fields button for a list of searchable fields, if you want to limit your search to a particular field.

More Search Options

To search without an index, or to search by filename, date, or size, click the More search options tab.

Viewing Search Results

After a search, dtSearch will display the results of the search. The top half of the dtSearch window will list all of the files retrieved in the search, and the lower half will show the first document in the list, with hits highlighted in yellow.

1. To select a document to view from the search results list, double-click on it.

2. To jump to the next hit in a document window, click Next Hit on the button bar (or press SPACEBAR). Press Ctrl-SPACEBAR, or click the Next Doc button, to go to the next document.

3. To change the way search results are sorted, click on one of the column headers (Name, Score, Location, Date, etc.).

4. Click the Launch button to open a document in the application associated with it. For example, a Word document would be launched in Microsoft Word.

To view or reuse a prior search request, click the Search History tab in the Search dialog box.

Create a Quick Summary of Your Search Results

An easy way to see all hits in all retrieved documents is to build a searchreport. A search report shows all hits along with the amount of context that you request.

1. Choose Search Report from the Search menu. The Generate Search Report dialog box will appear.

2. Enter the number of words (or paragraphs) of context that you want dtSearch to include in your search report and click OK to generate the report.

3. The search report will open in your word processor so you can edit or print it

Updating an Index

If you edit or add to your documents, you will need to update your index to reflect the changes.

To update your index, choose Update Index in the Index menu (or press Ctrl-U). Check the Index new or modified documents box and the Remove deleted documents box, and then click the Start Indexing button.

To schedule automatic index updates, click Index > Index Manager and click the Schedule Updates.


dtSearch 6 Upgrade Information

dtSearch Home

Downloads

Quick Start

dtSearch 6 Upgrade Information

Last Reviewed: February 15, 2002

Article: DTS0159

How to get dtSearch 6

Ordering

For upgrade ordering information, click here or go to the dtSearch Web site and click the "Upgrades" link. For new orders, click here or go to the dtSearch Web site and click the "Orders" link. All sales come with a full 30-day money-back guarantee.

Free Upgrade

If you purchased dtSearch 5.x after January 1, 2000, you are entitled to a free upgrade to dtSearch 6. To obtain instructions to download your free upgrade, please send an email to sales@dtsearch.com with your name, product(s) purchased, and if possible, approximate date of purchase and place of purchase, if not purchased direct.

30-Day Evaluation Version

A free 30-day evaluation version of dtSearch Desktop 6 (including the dtSearch Spider) is available from the dtSearch web site. Click here or go to www.dtsearch.com and select the "Evaluate" link. The setup program will ask for a serial number to install dtSearch. Check the box to install the evaluation version only and click the Next button.

Upgrading from dtSearch 5

Installation

Before installing dtSearch 6, we recommend that you uninstall any previous versions of dtSearch.

If you want to keep both dtSearch 6 and an earlier version on the same computer, install dtSearch 6 in a separate folder from the earlier dtSearch version. dtSearch 5 and dtSearch 6 can coexist on the same computer, as long as each version has its own separate folder and indexes.

Indexes

dtSearch 6 uses a new index format and cannot use indexes created by earlier versions of dtSearch. Therefore, you will need to create new indexes to use with dtSearch 6 after upgrading.

Internet Explorer

dtSearch 6 uses some Windows components that are installed by Microsoft Internet Explorer 5 or 5.5, or 6. Therefore, Internet Explorer 5 or later must be installed in order to use dtSearch 6. Internet Explorer does not have to be your default browser; it just has to be present on the computer so that dtSearch will have the components it needs.

Batch Scripts

You can use your dtSearch 5 batch scripts with dtSearch 6. To run a script, use the scriptrun.exe utility, which is installed in the dtSearch BIN folder. For documentation, see Scripts.chm in the dtSearch HELP folder.

Installation Problems

dtSearch 6 uses a new Microsoft component, the Windows Installer service, for installation. If you have Windows 2000 or Microsoft Office 2000, you already have the Windows Installer. If not, the dtSearch setup program should install it for you automatically.

If you get a message during installation that a service pack with a newer version of the Windows Installer is required, please do the following:

(1) Close the dtSearch Setup program, reboot, and run dtSearch setup again.

(2) If this does not solve the problem, click here for instructions on getting the Windows Installer from Microsoft and installing it directly. Once the Windows Installer is set up correctly, run dtSearch setup again.

New Features in dtSearch 6

dtSearch 6 Adds FindPlus® Distributed Searching, a Web Spider, Unicode support, API enhancements (dtSearch Text Retrieval Engine) and other enterprise features. For details on new end-user features, please see the dtSearch help file or the user's manual (PDF).

Easier Window Management

In dtSearch 6, each search has its own dtSearch window, so there is no need to close search results before starting a new search. To keep search results open and search in a new window, click Search|Search in a new window, or press Ctrl-Shift-S.

PDF, HTML and other files are displayed directly in the dtSearch viewer rather than in a separate Adobe Reader or web browser.

Improved Display Options

Click Options|Preferences and select the Document Display tab to customize the way dtSearch displays search results. You can set up different options for different types of files. For example, you could specify that files named *.cpp, *.h, or *.bas are displayed using a fixed-pitch font like Courier, and that other files are displayed with Arial. You can also control whether long lines of text will wrap and the color and font effects used to highlight hits.

Task Scheduler Integration

See "Scheduling Index Updates" in the dtSearch Help for information on setting up automatic indexing.

FindPlus® Searching & Spider

FindPlus searching lets you combine searches of dtSearch Web servers, shared network indexes, and local indexes in a single search request.

For information on sharing indexes on a local area network, see "Creating a Shared Index" and "Sharing Option Settings" in the online help.

For information on using dtSearch Desktop to access dtSearch Web servers, see "Searching using dtSearch Web."

Finally, for information on building your own indexes of web sites using the dtSearch Spider, see "Indexing Web Sites."

Field Searching

Field searching is greatly enhanced in dtSearch 6. Document summary information fields are automatically recognized in Word, Excel, PowerPoint, WordPerfect, and PDF documents. In XML files, all fields and attributes are searchable, and multiple levels of fields can be specified in a search.

Search Dialog Box

Improvements to the Search dialog box include auto-completion of search terms and preview of search results in the Search History panel.

Unicode Support

dtSearch 6 uses Unicode for indexing and searching, eliminating the need for alphabet customization to handle different languages. (You can still use the alphabet customization feature in dtSearch to change the way Ascii characters are indexed.)


dtSearch 5 Quick Start

dtSearch Home

Downloads

Quick Start

dtSearch 5.x Quick Start

Last Reviewed: February 5, 1999

Article: DTS0122

Applies to: dtSearch 5

dtSearch can search gigabytes of text in a second. It does this by building an index that stores the location of words in your files. An index does not store the text of documents, so to view retrieved documents you will still need to keep the documents accessible.

Indexing Documents

1. Choose Create Index (Basic) from the Index menu.

2. Enter a name for the index and click OK.

3. dtSearch will ask if you want to add documents to the index. Click Yes and the Update Index dialog box will appear.

4. In the Update Index dialog box, click Add folder... to add a folder to the list of folders to index.

5. Click OK to begin adding documents to your index.

Disk Space. To build the index, dtSearch needs free space on the drive containing the index at least equal to the size of the documents to be indexed. (This is because dtSearch creates temporary files during indexing.) Once complete, an index is usually about one fourth the size of the original documents, though this can vary depending on the number and type of documents. In general, the more documents in an index, the smaller the index will be as a percentage of the documents.

File Formats. dtSearch automatically recognizes major word processor files, database files (through ODBC), PDF files, ANSI text files, ZIP files, and HTML files. For a complete list of the file formats that dtSearch supports, see "Supported File Types" in the online help.

Searching using the Index

1. Click Search on the button bar. The Search dialog box will appear. The dialog box will display a list of the indexes you have created and a scrolling list of the words in the currently selected index.

2. To select an index to be searched, click on it in the list of indexes.

3. Enter a search request in the space provided. There are two basic types of search requests:

A natural language search request consists of an unstructured natural language or "plain English" query. In a natural language search request, words such as AND and OR are disregarded. Any specific word ordering such as phrases will also be disregarded. A natural language search can rank retrieved files from most to least relevant according to the density and rarity of matching words in your documents.

A boolean search request consists of a group of words, phrases or macros linked by search connectors such as AND and OR to precisely indicate the relationship between them.

To enter a natural language search request, click Natural language under Search type and type in a plain English search request: Get me Sam's memo on the 1994 takeover of CorpX.

To enter a boolean search request, click Boolean under Search type and enter words or phrases linked by connectors.

AND | Both words must be present
OR | At least one of the words must be present
NOT | The next word must not be present (use with AND or OR)
W/5 | The words must occur not more than 5 words apart.
* | Matches any number of characters
? | Matches any single character

Examples:

apple AND pear

*apple W/6 pear

apple sauce W/27 grape ju?ce

apple AND (pear W/5 bana*a)

apple AND NOT pear

4. Select Search Features to use in the search.

Stemming searches other grammatical forms of the words in your search request. For example, with stemming enabled a search for apply would also find applies.

Phonic search finds words that sound similar to words in your request, like Smith and Smythe.

Fuzzy search sifts through scanning and typographical errors. Fuzziness adjusts from 1 to 10 depending on the degree of misspellings. (Try starting with 3.)

Synonym searching tells dtSearch to use a thesaurus to find synonyms of words in your search request. dtSearch provides three ways to perform synonym searching:

Check the User thesaurus box to find synonyms that you have defined in your own thesaurus.
Check the WordNet thesaurus box to find synonyms using the WordNet concept network included with dtSearch.
Check the WordNet related words box to find related words from the WordNet concept network.

To see how stemming, phonic searching, fuzzy searching or wildcards will affect your search, click the Browse Words button. To browse the thesaurus used for synonym searching, click the Browse Thesaurus button.

5. To search for fields in databases or other files you have indexed, click the Fields button for a list of searchable fields.

6. Click Search to start the search.

Viewing Search Results

1. To select a document to view from the Search Results list, doubleclick on it or move the cursor to it and press ENTER.

2. To jump to the next hit in a document window, click Next Hit on the button bar or press SPACEBAR.

3. To search for text in a retrieved document, click the Find Text button in the button bar, enter the text you want to find, and click OK.

4. For a menu of all options you can use in a document window or Search Results, click the right mouse button anywhere in the window.

5. Click the Launch button to open a document in the application associated with it. For example, an HTML document would be launched in your web browser. See "Launching Applications" in the dtSearch manual or online help for more information on associating documents and applications.

6. Click the Image button to view an image associated with a document file. See "Linking Images and Documents" in the manual or online help for more information on associating documents and images.

7. HTML and PDF files. dtSearch can display HTML files with hits marked using its own viewer or using your web browser. To tell dtSearch to use your web browser to view HTML files, choose External Viewers in the Options menu. When displaying PDF files, dtSearch uses the Adobe Reader.

See "Hotkeys" in the online help for a complete list of shortcuts.

Copying Retrieved Text

To copy a block of text from a document to the Windows clipboard or to a file, mark the text (click and drag with the mouse) and click the Copy button in the button bar.

To copy an entire file, choose Copy File in the Edit menu while you are viewing the file. If a file is stored in a ZIP archive, Copy File will extract it from the archive.

Create a Quick Summary of Your Search Results

An easy way to see all hits in all retrieved documents is to build a Search Report. A Search Report shows all hits along with the amount of context that you request. It also contains hypertext links to the hits in your files.

1. Choose Search Report from the Search menu. The Generate Search Report dialog box will appear.

2. Enter the number of words of context that you want dtSearch to include in your Search Report and click OK to generate the report.

3. In the Search Report, double-click on the highlighted hypertext links to jump to the location of hits in your documents. Search Reports contain two types of hypertext links: highlighted page and paragraph numbers like [Page 4 Paragraph 22] which take you right to the location of a hit, and highlighted document names, which take you to the top of the document.

dtSearch home page


How to add a list of files, folders, or web sites to an index

dtSearch Home

Downloads

Quick Start

How to add a list of files, folders, or web sites to an index

Last Reviewed: April 18, 2001

Article: DTS0164

Applies to: dtSearch Desktop 6

In the dtSearch 6 Update Index dialog box, you can add folders, files, or web sites to the list of items to be indexed by using the Add folder, Add file, and Add web buttons. If you have a long list of things to add to an index, this process can be time consuming. This article will describe two ways to add a large number of items quickly.

Pasting a list of items into the What to index list.

(1) Open notepad or another text editor and make a list of the items that you want to add. The items have to be in the same format that dtSearch displays in the What to index list.

Files: Include the full path and filename for each file.

Folders: Include <+> after the name of a folder to automatically include subfolders.

Web sites: Enter the complete URL to the site, including the http://, and include a number inside <> marks after the URL indicating the crawl depth. Example: http://www.federalreserve.gov<4>

(2) Once you have completed the list, click Edit|Select All to select the entire list, and then Edit|Copy to copy it to the clipboard.

(3) Click Index|Update Index in dtSearch.

(4) Select the index that you want to add the list to

(5) Right-click in the What to index box and select the Paste list option from the menu. The list of items from your notepad file will be added to the list.

Directly Editing the Index Settings

dtSearch Desktop stores index settings in an XML file named index_o.ix in the index folder. This is an example of what this file looks like:

<?xml version="1.0" encoding="UTF-8" ?>

<dtSearchIndexSettings>

<DocRoot></DocRoot>

<Filters>*.mbx phone*.txt</Filters>

<ExcludeFilters>*.MP3 *.SWF *.PSP *.BMP *.CHI *.CHM *.COM *.DLL *.EXE *.GIF *.HLP *.IVI *.IVT *.IX *.JPEG *.JPG *.MSI *.NLS *.OBJ *.OCX *.PST *.RMI *.SYS *.TIF *.TMP *.TTF *.VBX *.WAV *.WPG DRVSPACE.*</ExcludeFilters>

<Extra Files>Index_LastUpdate*.htm</Extra Files>

<DetailedLogging>0</DetailedLogging>

<Item>

<Name>G:\eudora</Name>

<Recursive>1</Recursive>

</Item>

</dtSearchIndexSettings>

For each item included in the index, there is an<Item> entry in the index_o.ix file that specifies the location of the item, crawl depth, whether to include subfolders (the "Recursive" field), etc. As long as you are careful to preserve the XML format of the data, you can edit this file directly in notepad. If you just want to add a list of items to an index, the "Paste List" approach described above will generally be easier because dtSearch will format the data as XML for you. Editing the index_o.ix file directly can be useful if you want to transfer a list of items from one index to another. To do this, you would just use notepad to transfer the <Item> blocks from one index_o.ix file to the other, and then update the index.


How to change dtSearch file locations

dtSearch Home

Downloads

Quick Start

How to change dtSearch file locations

Last Reviewed: February 12, 2002

Article: DTS0174

Applies to: dtSearch Desktop 6

The UserData Folder

Your dtSearch UserData folder is where indexes, settings, and search results are saved. To see which folder dtSearch is using, or to select a different folder, start dtSearch Desktop and click Options > Preferences > Change dtSearch Folder.

dtSearch saves the most recent 100 searches in your UserData folder. To change the number of searches saved, click Options > Preferences > Search Results and change the number under Number of search results lists to keep.

Indexes

Each time you create an index, you can specify the location for the index in the Create Index dialog box. The default location for new indexes is the UserData folder, but you can select a different folder as follows:

(1) Click Options > Preferences > Index Defaults

(2) Change the Default location for new indexes folder.

Temporary Files

The dtSearch Spider uses the Windows TEMP folder to store downloaded pages while they are being indexed. To use a different folder

(1) Click Options > Preferences > Index Defaults

(2) In the Spider options section, enter a folder name under Folder to use for temporary files


How to change the hit highlight color in PDF files

dtSearch Home

Downloads

Quick Start

How to change the hit highlighting color in PDF files

Last Reviewed: September 8, 2004

Article: DTS0182

Applies to: dtSearch Desktop 6, dtSearch Web 6

dtSearch uses Adobe Reader to highlight hits in PDF files, and Adobe Reader does not provide an option to change the highlight color. Instead, it always uses the highlight color you have set up in Windows (in Adobe Reader 5) or white-on-black (in Adobe Reader 6).

To change the option in Windows, which will affect Adobe Reader 5 only (not Adobe Reader 6):

1. Click Start > Settings > Control Panel > Display

2. Click the "Appearance" tab.

3. In Windows XP, click "Advanced" to access the settings for specific items.

4. Change the color setting for "Selected Items".

The default color setting for Selected Items is white-on-blue. This is the color used to highlight the selected item in menus and other lists in Windows. If you change it to black-on-yellow, Adobe Reader will show highlights in yellow. (And from then on your Windows menus and list boxes will use yellow highlights as well.)

dtSearch Web

When a user searches PDF files in dtSearch Web, the user's own Windows color settings will determine how hits are highlighted. Therefore, there is no way for the web site to change this setting. Instead, each user can follow the procedure above to change the hit highlight color.


How to customize the way dtSearch prints search results

dtSearch Home

Downloads

Quick Start

How to customize the way dtSearch prints search results

Last Reviewed: January 28, 2002

Article: DTS0104

Applies to: dtSearch 6, dtSearch 5

Question

How do I change the format or contents of search results printed from dtSearch? When I change the settings in Options|Search Results format, the screen display changes but the printed format does not.

Answer -- dtSearch 6

dtSearch 6 generates a printable list of search results based on a template file, SearchListTemplate.rtf. The list generated from this file is then launched in your word processor, so you can edit or print it. If you select the "Report" format rather than the "List" format, then SearchReportTemplate.rtf is used.

(Another way to save search results is to click File|Save As in dtSearch. This lets you save the search results list in XML, CSV (for importing into Excel), or DBF format.)

To customize the template used to generate a search results list in RTF:

1. WordPad or Microsoft Word to open the file SearchListTemplate.rtf in your <dtsearch>\Template folder. You will see a document that looks like this:

Search Request: %%Request%%

Total files: %%TotalFiles%%

Total hits: %%TotalHits%%

<<<ListItem

%%Filename%% (%%Hits%% hits)

Location: %%Location%% Size: %%Size%% Last modified: %%Date%%

>>>

2. Edit the file to change the format and save the file as an RTF file. Do not save it as a DOC file because then dtSearch will not see the changes. Save the file in your dtSearch UserData folder, not in the Template folder, so that your changes will not be lost if you reinstall dtSearch. (Click Options|Change dtSearch Folder in dtSearch to find out where your UserData folder is.)

dtSearch uses SearchListTemplate.rtf as a template to generate a printable version of the search results file. Text between the <<< and >>> markers defines the template for each search results item. Text before the first <<< marker is the header for printed search results. Text after the >>> marker is ignored.

For example, if you wanted to see the full path and filename for each document rather than just the name, you could edit the template and change %%Filename%% to %%Fullname%%.

The following symbols are used in this template:

Symbol | Meaning
%%Hits%% | Number of hits in the document
%%Filename%% | Name of the document. For PDF and HTML files, this will be the TITLE.
%%Fullname%% | Full path and filename of the document
%%Date%% | Modification date of the the document
%%Location%% | Location of the document
%%Title%% | First 80 text characters of the document
%%Request%% | Search request
%%Size%% | Size of the document
%%TotalFiles%% | Number of files retrieved in the search
%%TotalHits%% | Number of hits in all files retrieved in the search

Answer -- dtSearch 5

1. Use WordPad or Microsoft Word to open the file resprn.rtf. This file will be located in your dtSearch directory. You will see a document that looks like this:

Search Results: “$request$”

Hits Document Modified Location

<<< $hits$ $filename$ $date$ $location$

>>>

2. Edit the file to change the format and save the file as an RTF file. Do not save it as a DOC file because then dtSearch will not see the changes.

dtSearch uses resprn.rtf as a template to generate a printable version of the search results file. Text between the <<< and >>> markers defines the template for each search results item. Text before the first <<< marker is the header for printed search results. Text after the >>> marker is ignored.

The following symbols are used in this template:

Symbol | Meaning
$hits$ | Number of hits in the document
$filename$ | Name of the document
$date$ | Modification date of the the document
$location$ | Location of the document
$title$ | First 80 text characters of the document
$request$ | Search request
$size$ | Size of the document
$type$ | File type of the document

If you have created user-defined fields in your documents, you can insert these as well. For example, if you have a user-defined "Subject" field, add $Subject$ to the template to display these items.


How to display PDF and HTML titles in search results

dtSearch Home

Downloads

Quick Start

How to display PDF and HTML titles in search results

Last Reviewed: November 23, 2000

Article: DTS0141

Applies to: dtSearch 6

HTML and PDF files have a "Title" attribute that usually has a user-friendly name for the document. For example, usrsmnl.pdf might have a title "User's Manual", which is generally preferable to display in search results.

dtSearch 6 includes options to specify whether you want the HTML and PDF Title or the filename to appear in search results. To change this option in dtSearch 6, click Options|Preferences, and select the Search Results tab. After changing the setting, the change will take effect immediately (you do not need to rebuild your indexes).

To change this option in dtSearch Web 6, use the Search Results tab in the Form Builder dialog box.

Additional Information

For each document, dtSearch can show both a filename and a "title" in search results. For most files, the dtSearch "title" is the first 80 characters of text. For PDF and HTML files, the "title" is the PDF or HTML title unless that title is being displayed as the filename. If the title is being displayed as the filename, then the dtSearch title is the first 80 characters. The following example summarizes this.

Example file:

Filename | sample.html
Title | A Sample File
First 80 characters | A Sample File demonstrating how the title options work

With the option to use the title as the filename ON, dtSearch will display the following in search results:

Filename | A Sample File
Title | A Sample File demonstrating how the title options work

With the option OFF, dtSearch will display the following in search results:

Filename | sample.html
Title | A Sample File


How to exclude folders from an index

dtSearch Home

Downloads

Quick Start

How to exclude folders from an index

Last Reviewed: April 19, 2003

Article: DTS0135

Applies to: dtSearch 5.2 or later

Question

How can I exclude directories from indexing? For example, suppose I want to index everything under my "c:\docs" folder except for c:\docs\archive\oldstuff.

Answer

When you specify a folder to be indexed in dtSearch, by default all subfolders of that folder will also be indexed. You can change this and have dtSearch index just the top-level folder, but sometimes you may want to exclude a couple of folders from a large tree of folders and subfolders.

To do this, use the Exclude filters in the Update Index dialog box. In addition to the usual type of filters -- *.exe, *.tif, etc. -- you can include filters with directory information. For example, to exclude everything in the c:\docs\archive\oldfiles folder, you could add an exclude filter "c:\docs\archive\oldfiles\*"

If a filter contains a space, it must be quoted, like this: "* ame with space\*".

In dtSearch 6.1 and later versions, exclude filters will apply inside a ZIP archive. For example, if you exclude *.doc from an index, and a ZIP archive contains sample.doc, then the indexer will skip sample.doc because of the exclude filter, even though it is inside a ZIP.

Additional Information

When a filename filter includes a \, dtSearch will compare it to the full path and filename of a document, but if the filename filter does not include a \, dtSearch will compare it to the filename alone. Examples:

s*.doc | Matches c:\docs\sample.doc, but not s:\docs\other.doc
*\oldfiles\* | Matches any file in a folder named oldfiles
"*\Old Files\*" | Matches any file in a folder named "Old Files". The quotation marks are needed because "Old Files" has a space, and without the quotation marks, dtSearch will see the filter as two separate filters, *\Old and Files\*


How to index databases with dtSearch

dtSearch Home

Downloads

Quick Start

How to index databases with dtSearch

Last Reviewed: May 23, 2001

Article: DTS0110

Applies to: dtSearch 5.x and 6.x

dtSearch indexes, searches and displays Access, dBase, FoxPro and Paradox files using Microsoft ODBC drivers. If the ODBC drivers are installed, dtSearch will index a database file as a series of records. The name of each record includes the filename of the database, the name of the table, and a record identifier. If instead dtSearch indexes a database as a single text file, you probably do not have the ODBC drivers installed.

Database indexing works best when the ODBC drivers are able to use a primary key to access records in the database. A primary key is a field that is unique for each record and that is designated as a primary key in the database.

The dtSearch Text Retrieval Engine can also index databases using ODBC, but there are other options as well. Click here for more information.

Updated Microsoft ODBC drivers are available from the Microsoft web site. For download information, please see this article in the Microsoft Knowledgebase:

Q239114 ACC2000: Updated Version of Microsoft Jet 4.0 Available in Download Center

http://support.microsoft.com/support/kb/articles/Q239/1/14.ASP


How to index Outlook messages with dtSearch

dtSearch Home

Downloads

Quick Start

How to index Outlook messages with dtSearch

Last Reviewed: January 3, 2003

Article: DTS0130

Applies to: dtSearch 6

dtSearch versions 6.04 and later can index Outlook messages, contacts, tasks, and notes. Each attachment to an Outlook message is indexed as a separate document. (You can use the filename filters in the Update Index dialog box to select the attachments that you want to index.)

After a search, you can view retrieved items in dtSearch, with hits highlighted, and launch a message, contact, task, or note in Outlook. For example, you could search for a message in dtSearch, launch the retrieved message in Outlook, and then reply to the message using Outlook.

Outlook indexing and searching work with dtSearch Desktop only and cannot be used with dtSearch Web. Outlook 2000 or Outlook XP (not Outlook Express) must be installed for Outlook indexing to work because dtSearch uses the Outlook program to access Outlook data.

To index Outlook messages and other information with dtSearch 6,

1. Open dtSearch

2. Click Index > Create Index

3. Enter a name for the index and click OK

4. In the Update Index dialog box, click Add Outlook to add one or more Outlook folders.

5. Select the folders to index and click OK to close the Select Outlook Folders dialog box.

6. Click Start Indexing.

Outlook Express. The above procedure applies to Outlook only, not Outlook Express. Outlook Express messages are stored in archives with a .DBX extension, and dtSearch can index these files like any other files. dtSearch will index each message and attachment in a DBX archive as a separate document.

Troubleshooting Outlook Indexing

When I index my PST file, it does not find anything

Outlook messages are stored on client machines in a file with a .pst extension. (Messages can also be stored on a Microsoft Exchange server.) If you try to index this file directly in dtSearch, even in dtSearch 6, it will not work because the file is encrypted. Text in a .pst file cannot be accessed except through Outlook.

Instead, add the PST file to your list of Outlook folders. To do this in Outlook,

(1) Click File|New, Outlook Data File

(2) Select "Personal Folders File (*.PST)" in the dialog box that appears and click OK.

(3) Browse for the PST file that you want to index.

Once a PST file is known to Outlook, it will appear in the list of Outlook folders to index in dtSearch.

Message: "Outlook Application could not initialize"

Message: "Outlook Error 800401f3: Invalid Class String"

These errors indicate that scripting support is disabled or damaged in your Outlook installation. If Outlook was installed without scripting support, the following steps will fix the problem:

(1) Click Start > Settings > Control Panel > Add/Remove Programs

(2) Locate Microsoft Office 2003 in the list of installed programs

(3) Click the "Change" button

(4) Click "Add or Remove Features" and then click "Next"

(5) Check the box "Choose advanced customization of applications" and then click "Next"

(6) Click the box next to "Microsoft Office Outlook" and select "Run all from my computer" from the drop-down list that appears.

(7) Click the "Update" button.

If Outlook was fully installed but the installation is damaged, the following steps will fix the problem:

(1) Click Start > Settings > Control Panel > Add/Remove Programs

(2) Locate Microsoft Office 2003 in the list of installed programs

(3) Click the "Change" button

(4) Click "Reinstalll or Repair" and then click "Next"

(5) Select "Detect and Repair errors in my Office installation"

(6) Click "Install"

Norton AntiVirus warns of viruses detected when dtSearch indexes Outlook messages

When dtSearch indexes Outlook messages, it extracts attachments for each message to a temporary folder, indexes, the attachments, and then deletes the temporary copies. There is no opportunity for any viruses in an attachment to become activated, but Norton AntiVirus or other scanning programs may detect the virus and warn about it. To find the messages that contain viruses: (1) record the filenames of any files that our anti-virus software warns about, and (2) search for these filenames in dtSearch, after the Outlook data has been indexed. When you retrieve a message, press the green "Launch" button to open it in Outlook, where you can delete the message from your Outlook message store.

Message: "A program is trying to access e-mail addresses you have stored in Outlook. Do you want to allow this?"

To prevent this message from appearing when dtSearch tries to index or search your Outlook messages, get the latest version of dtSearch Desktop. dtSearch Desktop versions 6.2 and newer will prevent this problem with all Outlook versions through Outlook 2003.

Message: "Either there is no default mail client or the current mail client cannot fulfill the messaging request. Please run Microsoft Outlook and set it as the default mail client."

Outlook indexing will not work unless Outlook is your "default" mail client. (You can still have other email programs installed, but Outlook has to be the default.) There are two places this setting has to be changed:

(1) In Outlook, click Tools > Options > Other, and check the box, "Make Outlook the default program for E-mail"

(2) In Internet Explorer, click Tools > Options > Programs, and make Outlook the default program for email.

"The Server is Unavailable"

No messages or items found during indexing

If you have multiple Outlook profiles, either (1) Outlook must be set to always use one of the profiles, without prompting, or (2) Outlook must already be running when dtSearch is started. The reason for this is that dtSearch may not be able to prompt you for the profile to use during indexing (for example, during an unattended indexing job). To set an Outlook profile as your default,

(1) Click Start > Settings > Control Panel > Mail

(2) Click Show Profiles

(3) Select the profile to use with dtSearch from the list at the end of the dialog box, and

(4) select Always use this profile.

(There is a checkbox in the "Select Profile" dialog box that appears when you start Outlook that supposedly does this, but it seems to have no effect.)

"Out of Memory" errors indexing large message stores.

We have received a small number of reports of this problem, occurring only with users who (a) have PGP Desktop Security, and (b) have installed it with the option to integrate with Microsoft Outlook. For these users, Outlook runs out of memory while dtSearch is indexing messages. Uninstalling PGP Desktop Security, and reinstalling it without Outlook integration, should resolve the problem.

Message: "The custom form could not be opened. Outlook will use an Outlook form instead."

Based on information in the Microsoft KnowledgeBase and Outlook newsgroups, this problem appears to be caused by either a bug in Outlook 2000 or data corruption in the forms cache.

As described in the articles cited below, Microsoft's recommended solution is to install the SR-1 or SR-1a update to Office 2000. A link to instructions for obtaining this update is also below. Note that if you get the SR-2 update instead, or any other update including the Outlook Security Update, you will encounter the problems created by the Security Update, described above. Therefore, it is important to install only SR-1 or SR-1a if you want to be able to index Outlook messages with dtSearch.

If the SR-1 or SR-1a update does not help, a frequently-recommended solution on Outlook-related newsgroups is to delete your Outlook forms cache. To avoid damaging your Outlook data, we recommend renaming the file instead so that you can recover it if necessary. To do this, (1) exit Outlook if it is open, (2) find the file "frmcache.dat" on your computer and rename it to frmcache.dat.save, (3) restart Outlook.

Q251415 "OL2000: Custom Form Produces Error Message and Does Not Load"

http://support.microsoft.com/support/kb/articles/Q251/4/15.ASP

Q245025 "OFF2000: How to Obtain and Install the Microsoft Office 2000 SR-1/SR-1a Update"

http://support.microsoft.com/support/kb/articles/Q245/0/25.ASP

Q223481 "The custom form could not be opened. Outlook will use an Outlook form instead."

http://support.microsoft.com/support/kb/articles/Q223/4/81.ASP

Office 2000 Customizable Alerts Info Button

http://officeupdate.microsoft.com/2000/downloadDetails/alerts.htm

Downloadable feature adds additional diagnostic information to "the most frequently occurring and

difficult to troubleshoot Office alerts."

The ScanPST Inbox Repair Utility

The ScanPST utility is a Microsoft tool for fixing problems with Outlook .PST and .OST files. Please see the links below for details applicable to your Outlook version.

Q287497 OL2002: How to Use the Inbox Repair Tools to Recover Messages

http://support.microsoft.com/support/kb/articles/q287/4/97.asp

Q272227 OL2000: How to Find and Run the Inbox Repair Tool

http://support.microsoft.com/support/kb/articles/Q272/2/27.ASP

Q181167 OL98: How to Find and Run the Inbox Repair Tool

http://support.microsoft.com/support/kb/articles/Q181/1/67.ASP


How to link images and documents

dtSearch Home

Downloads

Quick Start

How to link images and documents

Last Reviewed: January 5, 1999

Article: DTS0106

Applies to: dtSearch 5.x

When you click the Image button while viewing a document in dtSearch, dtSearch attempts to find and display the image file associated with the document. To find the image file, dtSearch looks for a file with the same name as the document and an extension that indicates it is an image file. For example, if you are viewing SMITH.DOC and click Image, dtSearch will look for SMITH.TIF. dtSearch will first check in the directory where the document is located and then, if the image is not found, in the other directories that you can list under Image Directories in the Imaging Options dialog box.

Linking Multiple Images to a Document

When dtSearch looks for images associated with a document, dtSearch automatically checks for multiple files having sequential names, such as SMITH001.TIF, SMITH002.TIF, etc. Some scanning (or other) software stores the pages of a document in multiple files with names ending in 001, 002, 003, etc. For example, a three page file named EXAMPLE.DOC could have image files EXAMP001.TIF, EXAMP002.TIF, and EXAMP003.TIF.

If dtSearch does not find an exact match to the document filename, dtSearch will look for and collect image files containing truncated versions of the filename. It will display the first of these files as "page 1" of the image, the second as "page 2," etc. Use the PgUp and PgDn keys to get from one page to another.


How to make common words like 'the' searchable

dtSearch Home

Downloads

Quick Start

How to make common words like 'the' searchable

Last Reviewed: December 11, 2000

Article: DTS0115

Applies to: dtSearch (all versions)

Question

When I try to search for words like "the" or "that", they show up in the index as "noise" words and dtSearch does not find any hits.

Answer

These words are not searchable because dtSearch considers them to be "noise" words. A noise word is a word such as the or if that is so common that it is not useful in searches. To save time, noise words are not indexed and are ignored in index searches. You can modify or remove the noise word list if you would prefer that these words be handled differently.

dtSearch 5

Edit the file dtsearch.noi in your dtSearch directory using Windows Notepad or another plain-text editor and delete the words that you want to make searchable. If you want all words to be searchable, just delete or rename the dtsearch.noi file. After you make changes to dtsearch.noi, create a new index. Existing indexes will not be affected because each index makes its own copy of the noise word list at the time the index is created.

The words in DTSEARCH.NOI do not have to be in any particular order, and can include wildcard characters such as * and ?. However, noise words may not begin with wildcard characters.

dtSearch 6

In dtSearch 6, the noise word list is named noise.dat and is located in your UserData folder. To edit the file,

1. Choose Preferences in the Options menu

2. Select the Indexing Options tab

3. Click the Edit... button next to the noise word list, or click the "..." button to select a different file.

Changes to the noise word list will only affect new indexes. When an index is created, it gets a private copy of the noise word list. Therefore, any existing indexes will continue to use the noise word list that was in effect when the index was created.


How to prevent dtSearch from indexing a word

dtSearch Home

Downloads

Quick Start

How to prevent dtSearch from indexing a word

Last Reviewed: October 15, 1999

Article: DTS0138

Applies to: dtSearch (all versions)

Question

How can I prevent dtSearch from indexing certain words?

Answer

The file noise.dat in your dtSearch folder contains a list of words that dtSearch will exclude from indexes. To modify this file, just open it in Notepad and add or remove words. The words in the list can contain the wildcard characters * and ? but must begin with a letter.

When you create an index, dtSearch stores a copy of the noise word list in the index. Changes to the noise word list will, therefore, only affect indexes created after the changes were made.

Indexing Numbers

To prevent dtSearch from indexing numbers, click Options > Preferences > Indexing Options. The Indexing Options dialog box contains a setting to prevent numbers from being indexed.


How to run multiple copies of dtSearch on the same machine

dtSearch Home

Downloads

Quick Start

How to run multiple copies of dtSearch on the same machine

Last Reviewed: November 20, 2000

Article: DTS0136

Applies to: dtSearch 5.x

Question

How can I have several copies of dtSearch running on one computer, each with its own separate option settings?

Answer

Note: This information is valid for dtSearch 5.x only. dtSearch 6 uses one set of option settings for each user.

Option 1. Multiple dtSearch installations

Each copy of dtSearch needs its own "private" directory for option settings. If you just install several copies of dtSearch in different folders on the same machine, though, the different dtSearch copies may still use the same private directory because dtSearch uses a single registry entry to identify the private directory. Instead, you can use a command-line switch to force each dtSearch copy to use its own private directory, like this:

(1) Install dtSearch in c:\dtsearch1, c:\dtsearch2, and c:\dtsearch3

(2) For each copy of dtSearch, make a Windows shortcut with the command-line switch /dir (directoryname), like this:

c:\dtsearch1\dtswin.exe /dir c:\dtsearch1

c:\dtsearch2\dtswin.exe /dir c:\dtsearch2

c:\dtsearch3\dtswin.exe /dir c:\dtsearch3

You can use this approach to run different versions of dtSearch on the same machine, without conflicts between the versions.

Option 2. One dtSearch installation, multiple working directories

If you want to have only one copy of dtSearch on the machine but still have different option settings (for example, for three different users who share the same machine), you can use the same procedure described above but with a single copy of the dtSearch program.

(1) Install dtSearch in c:\dtsearch

(2) For each user, make a Windows shortcut with the command-line switch /dir (directoryname), like this:

c:\dtsearch\dtswin.exe /dir c:\dtsearch1

c:\dtsearch\dtswin.exe /dir c:\dtsearch2

c:\dtsearch\dtswin.exe /dir c:\dtsearch3

Additional Information

How can I tell what directory dtSearch is using?

Click Help|About in dtSearch. Two directories will be displayed in the "About dtSearch" dialog box: the "Home" directory (where the dtSearch program is installed), and the "Private" directory, which is where option settings will be saved.

How does dtSearch decide which directory to use?

When dtSearch starts up, this is how it decides the private directory to use for option settings:

(1) If there was a /dir command-line switch, dtSearch will use that directory.

(2) If a registry entry identifies a dtSearch working directory, dtSearch will use that directory.

(3) If the directory where dtswin.exe (the dtSearch program) is located can be locked for read/write access, dtSearch will use that directory.

(4) Finally, dtSearch will create a temporary directory and use that directory as its private directory. The name of this directory will be stored in the registry so dtSearch will find it next time dtSearch runs.

In the Options|Preferences dialog box in dtSearch, there is a place to specify the dtSearch private directory. Changes to this setting are stored in the registry.


How to search for connector words like 'and' and 'not'

dtSearch Home

Downloads

Quick Start

How to search for connector words like 'and' and 'not'

Last Reviewed: December 11, 2000

Article: DTS101

Applies to: dtSearch (all versions)

Question

How can I search for a phrase that contains one of the dtSearch connector words like and, or, and not. For example, how can I search for the phrase not relevant?

Answer

(1) Put a tilde (~) after a connector word to tell dtSearch that it is a search term, not a connector. Examples:

clear and~ convincing evidence

not~ relevant

to~ be or~ not~ to~ be

(2) Connector words such as and and not are also included in the noise word list. Therefore, to make dtSearch index these files, you will need to change the noise word list to remove the words that you want to make searchable. For more information, see "How to make common words like 'the' searchable".

Additional Information

The dtSearch connector words are: and, or, not, to, contains.

Adding a ~ after a connector word prevents dtSearch from recognizing the word as a connector but does not otherwise affect the search. The ~ character after a word tells dtSearch to apply the stemming rules to it. Because the stemming rules included with dtSearch do not modify short words, the ~ does not change the outcome of a search for and, or, not, or to.


Launching files from dtSearch

dtSearch Home

Downloads

Quick Start

Launching files from dtSearch

Last Reviewed: May 31, 2001

Article: DTS0168

Applies to: dtSearch Desktop 6

Adobe PDF Files

dtSearch uses either Adobe Reader or Adobe Acrobat to view PDF files. Which program is used, and how it is opened in dtSearch, depends on the software that is installed and on your dtSearch settings.

Adobe Reader vs. Adobe Acrobat

dtSearch uses Windows' filename association for the .PDF extension to decide which program to launch to view PDF files. If both Adobe Reader and Adobe Acrobat are installed, the filename association will be controlled by the setup programs for these products. When both Adobe Acrobat 5 and Adobe Reader 5 are installed on the same machine, Adobe Acrobat will have the .PDF filename association. However, if Adobe Reader 5 is already open and you launch a PDF file, Adobe Reader, not Acrobat, will open the PDF file. Therefore, if you want to override the default association and open PDF files with Adobe Reader, one way to do this is to leave an Adobe Reader window open. (Another advantage of leaving Adobe Reader open is that you skip the slow Adobe Reader startup process.)

Separate windows vs. embedded window

By default, dtSearch displays PDF files in the document pane in the dtSearch window. To open the PDF in a separate window, you can click the Launch button [image] in dtSearch, or press F8. To have PDF files open in a separate window by default, click Options|Preferences, select the Search Results tab, and check the box to "Display PDF files in a separate window".

Whether you launch PDF files in a separate window or in the dtSearch document pane, there will be two buttons on the Adobe Reader button bar to navigate hits:

Other Files

When you click the Launch button or press F8 in dtSearch to launch a document, Windows' filename associations will determine which program is used to launch the file. For example, a .DOC file will usually open in Microsoft Word, and an .HTML file will open in your web browser.

Windows filename associations are stored in the system registry. A useful tool for changing these associations is "Freedom of Association", a free utility from the PC Magazine web site (www.pcmag.com).


'ODBC Resource DLL Is a Different Version' Message

dtSearch Home

Downloads

Quick Start

ODBC Resource DLL Is a Different Version Message

Last Reviewed: January 9, 1999

Article: DTS0120

Applies to: dtSearch 5

Symptoms

The message "ODBC Resource DLL Is a Different Version" when you try to run dtSearch.

Cause

ODBC is not installed correctly.

Many different Microsoft products install ODBC versions (dtSearch does not install any ODBC files) and sometimes the versions of the various components can get confused. The message appears in dtSearch because dtSearch uses ODBC to access databases, but this is not a dtSearch error message.

Resolution

Microsoft KnowledgeBase article Q170769 describes how this can happen and the steps to fix the problem.


PDF files open slowly with Acrobat 6 installed

dtSearch Home

Downloads

Quick Start

PDF files open slowly with Acrobat 6 installed

Last Reviewed: June 18, 2003

Article: DTS0188

Applies to: dtSearch 6, Adobe Acrobat 6 or Adobe Reader 6

Symptom

After installing Adobe Acrobat 6 or Adobe Reader 6, PDF files open very slowly in dtSearch Desktop, sometimes taking 30 seconds or more to open.

Workarounds

When you open a PDF file in dtSearch Desktop, Adobe Reader must load and initialize, which can take a long time. In Adobe Reader and Adobe Acrobat 6, the delay has become much longer. To prevent the delay, there are two options:

(1) Open Adobe Acrobat or Adobe Reader and leave it open in the background while dtSearch is running.
This eliminates the need for Adobe Reader or Adobe Acrobat to initialize to open each PDF file, so PDF files will open much more quickly. In dtSearch 6.21, there is an option to have dtSearch automatically open Adobe Reader in the background to make PDF files open more quickly. To enable this option, click Options > Preferences > External Viewers, and check the box labelled "Automatically open Adobe Reader to speed viewing of PDF files."

(2) Change the dtSearch option settings to display PDF files in a separate Adobe Acrobat window. To do this,

- Click Options > Preferences > Search Results

- Check the box "Display PDF files using Adobe Reader in a separate window"

- Click OK.

After making this change, dtSearch will launch PDF files in Adobe Reader or Adobe Acrobat rather than inside the dtSearch window. Hits will still be highlighted, and PDF files should open much more quickly.


Phrase or boolean search is not working

dtSearch Home

Downloads

Quick Start

Phrase or boolean search is not working

Last Reviewed: February 23, 2002

Article: DTS0114

Applies to: dtSearch

Symptoms

- A search for a phrase like "first class mail" finds the search words but not as part of the phrase

- A search for a boolean expression like "apple and pear" finds "apple" even when "pear" is not in the document

- A search for a phrase like "this is a house" finds all instances of "house"

Possible Causes

(1) Natural language searching is selected instead of boolean searching

(2) The search request contains noise words

(3) Some of the words found are in hidden document properties

Natural Language Searching

In a natural language search, all "boolean" connectors like AND, OR, NOT, W/7, etc. are ignored in your search request. Also, all phrases like New York are broken up into their constituent words, so it is as if you are searching for two separate words: New and York.

dtSearch does this because the purpose of natural language searching is to find a collection of documents, ranked by full relevancy, that best matches your search request. So, for example, if you are searching for: Get me Tom's memo on the 1998 takeover of CorpX by Megahugecorp, and 1998 appears in 30,000 documents and Megahugecorp only appears in three documents, the documents containing Megahugecorp would be ranked much higher, bringing you that much closer to your memo.

Technically, weighting of retrieved documents takes into account: the number of documents each word in your search request appears in (the more documents a word appears in, the less useful it is in distinguishing relevant from irrelevant documents); the number of times each word in the request appears in the documents; and the density of hits in each document. For purposes of this relevancy ranking, and hence for purposes of your search request, noise words, as well as search connectors like NOT and OR, are ignored. All words are treated as "individuals," ignoring phrases.

Noise Words

dtSearch ignores "noise" words like "if" or "the" when searching. Therefore, if you search for a phrase like "This is a house", dtSearch will ignore "this", "is", and "a", and just search for "house". If you need to search for noise words in your documents, you can build your indexes without a noise word list. To do this, click Options > Preferences > Indexing Options in dtSearch Desktop, edit the noise word list, and either delete all of the words or delete only the ones that you want to be able to search for.

After you change the noise word list, you will need to delete and rebuild your indexes, because each index keeps its own copy of the noise word list when the index is created.

Hidden Document Properties

Most documents contain a document "Summary Information" or "Properties" area that is not normally displayed but that is still searchable. These fields may contain some of the words matched by your search. To find the document properties, open the document in the application that created it, and look in the File menu for a Document Properties or Document Summary Information item. For example, to find the properties of an Adobe Acrobat (PDF) file, click File > Document Properties > Summary in Adobe Reader.

You can usually see the document properties in dtSearch Desktop as well. This chart explains how to find document properties in commonly-used file types:

File Type | How to find Document Properties
PDF | Open the PDF file in dtSearch or in Adobe Reader, and press Ctrl+D
DOC, XLS, PPT, WPD | dtSearch will list the document properties at the end of the document text. In Word, Excel, PowerPoint, or WordPerfect, click File > Document Summary Information.
HTML | Open the HTML file in dtSearch or in your web browser, right-click in the document window and select "View Source". Document properties will appear in "META" tags, like this: <META name="Keywords" content="Apple Pear Grape Banana">


Troubleshooting -- HTML files are indexed as text

dtSearch Home

Downloads

Quick Start

Troubleshooting -- HTML files are indexed as text

Last Reviewed: September 5, 2003

Article: DTS0191

Applies to: dtSearch 6, dtSearch Engine 6

Symptoms

- HTML tags are visible in dtSearch Desktop or dtSearch Web when opening a file

- Variable names in scripts, comments, or URLs are searchable after indexing HTML files

- META tags and other fields in an HTML file are not searchable

Troubleshooting Steps

(1) Check that dtSearch is interpreting the file as HTML..

If HTML tags are visible in the file when you open it in dtSearch Desktop or dtSearch Web, then dtSearch is interpreting the file as plain text. dtSearch detects HTML files by checking for a .html or .htm extension, a <HTML> header, or other common HTML header strings. If none of these are present, dtSearch may conclude that a file is plain text, in which case tags will be indexed like other text.

To make dtSearch Desktop treat a group of files as HTML even though dtSearch thinks the files are text,

(1) Open dtSearch Desktop version 6.21 or later and click Options > Preferences > File Types

(2) Click New... to create a new rule and give the rule a name (the name does not matter)

(3) Under File type" select HTML

(4) Under Filename filters, enter filter expressions like *.asp to identify the files that are to be covered by the rule.

(5) Check the box labelled Override all other file type detection methods for these files (this will make dtSearch treat a file as HTML even if the HTML file parser does not recognize it as HTML based on the header).

(6) Click OK

The File Types dialog box can also be used to make dtSearch index text files as XML, or to make dtSearch index HTML or XML files as plain text.

Note for dtSearch Engine Developers: To use these settings in a custom application developed using the dtSearch Engine, use FileTypeTableFile property of the Options object to tell dtSearch to use the filetype.xml file that dtSearch Desktop creates after following the procedure described above.

(2) Check the option settings used to index HTML files

Even if a file is indexed as HTML, non-text content such as scripts, styles, links, comments, or the filename may be searchable because of option settings. To change these option settings, start dtSearch Desktop and click Options > Preferences > Indexing Options.

To prevent words in the filename or directory name from being searchable, un-check the Index filenames as text box.

To prevent words in scripts, styles, links, and comments from being searchable, un-check the Index HTML scripts, styles, links and comments box.

To prevent META tags from being searchable, un-check the Index document properties box. (This will also affect Microsoft Office Document Summary Information fields.)


Troubleshooting database indexing

dtSearch Home

Downloads

Quick Start

Troubleshooting database indexing

Last Reviewed: March 18, 2002

Article: DTS0107

Applies to: dtSearch (all versions)

Symptoms

When an Access, Paradox, FoxPro, or DBase database file is indexed, dtSearch treats it as one long text file instead of as multiple records.

Possible Cause

ODBC drivers are not installed.

dtSearch indexes, searches and displays Access, dBase, FoxPro and Paradox files using Microsoft OCBC drivers. If the ODBC drivers are installed, dtSearch should index a database file as a series of records. The name of each record includes the filename of the database, the name of the table, and a record identifier. If instead dtSearch indexes a database as a single text file, you probably do not have the ODBC drivers installed.

Resolution

If you have Office 97, Office 2000, or Office XP, reinstall it with the option to install "Data access" components selected.

If you do not have Microsoft Office or another source of the ODBC drivers, you can download them from the Microsoft web site. Two separate downloads are required:

(1) The Microsoft Data Access Components (MDAC), available here:

http://www.microsoft.com/data.

(2) The Microsoft "Jet" database engine, available here:

http://support.microsoft.com/default.aspx?scid=kb;en-us;Q239114&id=Q239114

Note: Even with both of the above installed, some users have reported that ODBC still did not work until Microsoft Office was also installed. It may no longer be possible to use ODBC to access desktop databases without installing Microsoft Office.

For more information on installing MDAC and the Jet Engine, please see these articles on the Microsoft web site:

MDAC Installation

HOWTO: MDAC Setup Troubleshooting Guide (Q232060)


Troubleshooting hit highlighting problems

dtSearch Home

Downloads

Quick Start

Troubleshooting hit highlighting problems

Last Reviewed: March 20, 2002

Article: DTS0112

Applies to: dtSearch 6

Symptoms

After a search, dtSearch highlights the wrong words in a retrieved file

No hits are highlighted in a file

Possible Causes

The document was modified since it was indexed

An indexing setting was changed since the document was indexed

The document was indexed with a different version of dtSearch

The document was indexed with the dtSearch Spider, and you are searching using dtSearch Web

Browser security settings disabled JavaScript

Troubleshooting steps: incorrect words highlighted

(1) Perform the search in dtSearch Desktop

(2) After the search, click on the document that shows incorrect hit highlighting

(3) Click View > Document Information, or press Ctrl-Y.

A diagnostic dialog box will appear with a brief explanation of possible reasons for the hit highlighting problems.

To resolve the problem, update the index you searched with the "Clear index before adding documents" box checked. This will completely rebuild the index, ensuring that the index will reflect current indexing settings.

If the diagnostic dialog box does not identify any problems, check the search options you used in the search, and try repeating the search with no search options enabled. Synonym searching, phonic searching, and higher levels of fuzziness (3 or more) can generate a large number of word matches. If two of these search features are combined, the search may be so broad as to be useless. To see the words matched by different combinations of search features, click the Browse Words button in the Search Dialog box and try searching for words in your search request with various combinations of search features.

Additional information

After a search, dtSearch uses information in the index that you searched to highlight hits in a document. The index stores the location of each word in your documents, and dtSearch uses these word locations for hit highlighting. If the information in the index is out of date, the wrong words will be highlighted. Rebuilding the index fixes the problem by updating the index so that it is consistent with your documents and your dtSearch version and settings.

Troubleshooting steps: no words highlighted (dtSearch Desktop)

A document can be retrieved with no words highlighted if the hits matched are in document properties that are not displayed.

PDF files: Open the PDF file in Adobe Reader and click Document Properties > Summary or press Ctrl+D (Ctrl+D will also work inside the dtSearch viewer) to see summary information fields that may contain your search terms.

HTML files: Right-click the HTML file in dtSearch, or in your web browser, and select "View Source". Look for a "META" tag containing the words you searched for.

Document properties in Word, Excel, PowerPoint, and WordPerfect files will appear in dtSearch at the end of the document.

Troubleshooting steps: no words highlighted (dtSearch Web)

Try the same search in dtSearch Desktop. If no hits are highlighted in dtSearch Desktop, refer the to the dtSearch Desktop troubleshooting steps above.

If hits are highlighted in dtSearch Desktop and not in dtSearch Web, there are two possible reasons:

(1) The documents were indexed using the dtSearch Spider, and one of the following applies: (a) you have dtSearch Web 6.10 or earlier (older versions of dtSearch Web could not highlight hits in documents indexed using the dtSearch Spider); (b) the "HighlightHttpDocs" setting is disabled in the dtsearch_options.html file is either absent or set to 0, or (c) dtSearch Web was unable to download the page to apply hit highlight markings to it.

(2) Check the search results template file (the _options.html file generated by dtSearch Web Setup) to see if it is set to generate hit highlighted links or direct links. Change the %%DirectLink%% symbol to %%HighlightLink%% to switch to highlight links. See the "Generated Files" topic in dtSearch_Web.chm for more information on the _options.html file.

Troubleshooting steps: Next Hit/Prev Hit buttons do not work

If the documents are PDF files, you have to use the Next Hit/Prev Hit buttons on the Acrobat Reader toolbar rather than the dtSearch Next Hit/Prev Hit buttons. For more information, see Troubleshooting PDF hit highlighting problems in dtSearch Desktop.

If the problem affects other documents, and if you can see hit highlighting but clicking the Next Hit button has no effect, your Internet Explorer security settings may have disabled JavaScript. dtSearch Desktop and dtSearch Web use JavaScript to provide hit navigation.

1. Check to see if JavaScript is disabled

To check JavaScript, click on this link: http://support.dtsearch.com/testjavascript.html

If you can click the button and get a message to appear, JavaScript is working. If not, JavaScript is disabled.

2. Fix the JavaScript setting in Internet Explorer.

In dtSearch Desktop, the security settings for the "Internet Zone" apply to dtSearch Desktop, so if JavaScript is disabled in the Internet Zone settings, dtSearch Desktop will not be able to implement Next Hit/Prev Hit. Setting the security settings to any of the predefined levels (low, medium, or high) will enable JavaScript.

3. Repair Internet Explorer

Sometimes Internet Explorer loses its JavaScript support files after JavaScript has been disabled. To repair Internet Explorer

Click Start > Settings > Control Panel > Add/Remove Programs

Find "Microsoft Internet Explorer" in the list and double-click it.

In the window that opens, choose "Repair" and click OK.

4. Install a newer version of Internet Explorer

Alternatively, or if step 3 does not help, install the latest version of Internet Explorer. The dtSearch CD includes Internet Explorer in the other\msie folder.


Troubleshooting missing indexes

dtSearch Home

Downloads

Quick Start

Troubleshooting missing indexes

Last Reviewed: May 7, 1999

Article: DTS0132

Applies to: dtSearch 5.x

Symptoms

- dtSearch cannot access an index when I try to search

Possible Causes

- The index location changed due to network drive mapping changes

- The index library no longer contains a reference to the index

- The index is damaged

Troubleshooting Steps

(1) Is there a reference to the index in the index library?

Your index library is a list of references to the indexes that dtSearch knows about. To see which indexes are listed and where dtSearch thinks they are, Click Index|Index Manager.

If the index is not in the list, click the Recognize Existing button, browse to the location of the index, and click on any of the index_*.ix files. dtSearch will add the index to your index library.

If the index is on the list, go to step (2).

(2) Can dtSearch access the index?

Click on the name of the index in the list in Index Manager. On the left side of the Index Manager, you should see information about the index: the location, the number of words, number of documents, folders indexed, etc.

If the number of documents is not zero, then dtSearch is able to access the index and you should be able to search it.

If the number of documents is blank or zero, this means that dtSearch cannot access the index. The index may be damaged, or the index location may have changed, possibly because of a network drive mapping change since the index was created. For example, if the "G:" drive has become the "H:" drive, dtSearch may be looking for the index in the wrong place. Check the folder name under Index Location and open that folder in Windows Explorer.

If Windows Explorer shows files in the index folder, the index is present but damaged. In dtSearch, use Index Manager to delete the index and make a new index to replace it.

If the index folder is empty or missing in Windows Explorer, go to step (3).

(3) Did the index get relocated by a change in network settings?

If the index was located, you will probably want to find it so you can either try to recover the index or delete the index so it does not take up space on your hard disk.

A dtSearch index consists of a folder with a group of files named index_a.ix, index_d.ix, etc. To find all of the indexes on a drive, in Windows, click Start, Find, Files or Folders, and enter *.ix under Named. Check each of the drives where the index might be, especially network drives. The folder names listed will tell you where each index is.

If the index was not found, you will need to create a new index to replace the missing index.

If the index was found, go back into dtSearch, click Recognize Existing in Index Manager and browse to the folder where the index was found. After recognizing the index, exit Index Manager, click the Search button, search the index, and try to view documents. You may find that even though you can search the index, when you try to view documents, dtSearch cannot open the documents. If this is the case, you will need to delete and rebuild the index. (This can happen if the change in drive mapping that changed the index location also changed the document locations, making the information in the index obsolete.)


Troubleshooting missing DLL problems

dtSearch Home

Downloads

Quick Start

Troubleshooting missing DLL problems

Last Reviewed: August 30, 2003

Article: DTS0108

Applies to: dtSearch 6.1, 6.11, 6.2

OLEACC.DLL

Click here to download oleacc.dll, and save the file in your dtSearch BIN folder (usually c:\program files\dtsearch\bin). This version of oleacc.dll will only work with dtSearch, so you should not use it with any other program; it is just a placeholder so that dtSearch 6.x will run under Windows 95 or Windows 98.

Important Note 1: Please do not copy the file anywhere else, like your Windows or Windows\System folder, as this could cause other programs on your computer to stop working.

Important Note 2: This file can only be used with Windows 95 or Windows 98.

Upgrading to dtSearch 6.21 or later will also fix this problem.


Troubleshooting PDF hit highlighting problems in dtSearch Desktop

dtSearch Home

Downloads

Quick Start

Troubleshooting PDF hit highlighting problems in dtSearch Desktop

Last Reviewed: November 2, 2004

Article: DTS0166

Applies to: dtSearch Desktop 5 or dtSearch Desktop 6; Adobe Reader or Adobe Acrobat versions 5 or 6

1. "Next Hit" and "Prev Hit" buttons greyed out in dtSearch Desktop

When a PDF file is displayed in dtSearch Desktop, the "Next Hit" and "Prev Hit" buttons on the dtSearch Toolbar are disabled, and instead you can use the "Next Hit" and "Prev Hit" buttons on the Adobe Reader toolbar. To find the buttons on the Adobe Reader toolbar, look for like those pictured on the right.

If the buttons on the Adobe Reader toolbar are grey, it means that either the PDF file has no hits that can be highlighted (for example, if you searched on the filename), or the hits may all appear on the same page. In Adobe Reader, the "Next Hit" and "Prev Hit" buttons take you to the next or previous page with hits, so if all of the hits are on one page, these buttons cannot do anything.

2. Missing "Next Hit" and "Prev Hit" buttons in Adobe Reader

If the dtSearch "Next Hit" and "Prev Hit" buttons do not appear in the Adobe Reader toolbar, then the dtSearch plug-in to highlight hits may be disabled. This can happen in Adobe Reader or Adobe Acrobat if the "Certified Plug-ins Only" setting is enabled.

To solve the problem, do the following:

[Adobe Reader or Adobe Acrobat 6]

(1) Open Adobe Reader or Adobe Acrobat

(2) Click Edit > Preferences > Startup to access startup options

(3) Un-check the "Use only certified plug-ins" box and click OK.

(4) Close Adobe Reader or Adobe Acrobat

[Adobe Reader or Adobe Acrobat 5]

(1) Open Adobe Reader or Adobe Acrobat

(2) Click Edit > Preferences > Options. (In Adobe Acrobat, click Edit > Preferences > General.)

(3) Un-check the "Certified Plug-ins Only" box.

(4) Close Adobe Reader or Adobe Acrobat

If you have Adobe Acrobat, the same procedure must be done to change the setting for Adobe Acrobat. (In Adobe Acrobat, click Edit > Preferences > General to access the Preferences dialog box.) Additionally, you may need to reboot for the option change to take effect. This may be necessary if there is a hidden instance of Adobe Acrobat or Adobe Reader running on your computer.

3. Hits in a PDF file are not highlighted in dtSearch Desktop 5

(1) Follow the procedure described above to disable the "Certified Plug-ins Only" setting.

(2) Run dtSearch setup again to install the dtSearch hit-highlighting plug-in for Adobe Acrobat and Adobe Reader

Additional Information

dtSearch uses a "plug-in" component to highlight hits in Adobe Reader. In Adobe Reader 5, Adobe added a new default setting, "Certified Plug-ins Only" that prevents any plug-ins not written by Adobe from working. (Only Adobe's own plug-ins can be certified, so "Certified" really means "Adobe".) Changing the option setting allows the dtSearch plug-in to be loaded by Adobe Reader 5, so that hit highlighting will work correctly.


Troubleshooting PDF indexing

dtSearch Home

Downloads

Quick Start

Troubleshooting PDF indexing

Last Reviewed: September 9, 2004

Article: DTS0108

Applies to: dtSearch 6

dtSearch reports file is encrypted, but it can be opened in Adobe Reader

If a PDF file has a security password, dtSearch may not be able to open it to extract the text for indexing.

A PDF file may have a security password even if no password is needed to open it in Adobe Reader. For example, the password may prevent printing the document, changing the document, adding annotations, etc.

To see if a PDF file has a security password, open it in Adobe Reader and click File|Document Info|Security. A dialog box will appear that will tell you if the file has a password. For information on indexing PDF files that have security, please see "Security passwords on PDF files."

dtSearch reports file is corrupt, but it can be opened in Adobe Reader

Adobe Reader and Adobe Acrobat will automatically fix some file corruption problems in PDF files when a PDF file is opened. In Adobe Acrobat 6, you will see a brief "Rebuild" message when this occurs.

To fix a single PDF file, open it in Adobe Acrobat 5 or later and save it using File > Save As. This will usually fix any problems in the file and will also optimize the file for much faster viewing over the web. After saving the PDF file in Adobe Acrobat, try to index it again in dtSearch.

To fix a large number of PDF files at once,

(1) Start Adobe Acrobat Professional 6 or later

(2) Click Advanced > Batch Processing...

(3) Select "Fast Web View"

(4) Click "Run Sequence"

(5) Select the files to process

These steps will repair and optimize all of the PDF files selected.

PDF file is indexed without errors but no text is searchable

Some PDF files contain either pure image data or text but no encoding information. In either of these cases, there is no text in the PDF file that can be indexed, and OCR is needed to add text to the PDF file. For information on OCR tools that can add text to a PDF file, see How to use dtSearch or dtSearch Web with OCR.

Steps to check whether a PDF file contains text

Some PDF files look like they have text in them but do not. Because the PDF file format is a graphical format, a PDF file can contain a picture of text with no words. To see if a PDF file contains text,

1. Open the file in Adobe Reader

2. Select the Text Select Tool (press "V" or click on the [image] icon in the toolbar)

3. Try to select some text.

If you are unable to select test with the Text Select Tool, the file probably has no text in it. You will need to use an Optical Character Recognition (OCR) Program like TextBridge or Acrobat Capture to convert the image to text. Click here for more information.

Steps to check whether a PDF file contains valid encoding information

Some PDF files contain text but use an encoding that is meaningless outside of the PDF file. For each character, the PDF file contains embedded font information that describes how to draw the PDF file, but the characters do not correspond to an encoding that can be used to extract text from the file. As a result, the PDF file looks like a normal document but there is no meaningful text in the file. For more information, see the "PDF" section in Unicode support in dtSearch 6.

To see whether a PDF has valid encoding information,

1. Open the file in Adobe Reader

2. Select the Text Select Tool (press "V" or click on the [image] icon in the toolbar)

3. Select a block of text

4. Click Edit > Copy

5. Open Notepad, Microsoft Word, or another program that can accept pasted text.

6. Click Edit > Paste

If you see what looks like random letters instead of the text you copied from the PDF file, the PDF file lacks encoding information. If a PDF file lacks encoding information, there is no way to index it with dtSearch.


Troubleshooting Scheduled Updates

dtSearch Home

Downloads

Quick Start

Troubleshooting Scheduled Index Updates

Last Reviewed: February 28, 2003

Article: DTS0123

Applies to: dtSearch 6

dtSearch uses the Windows Task Scheduler to schedule automated index updates.

Symptom: No Task Scheduler dialog box when you try to schedule a task

Check that Task Scheduler is installed. In Windows, click Start > Settings > Control Panel. You should see an icon on Control Panel for Scheduled Tasks. If this icon is not present, Task Scheduler is probably not installed. To install Task Scheduler, install Microsoft Internet Explorer 6, and make sure that Task Scheduler is selected to be installed when you run the setup program.

Symptom: Scheduled task never runs

Check the user account for the task. After you schedule a task in dtSearch Indexer, click Start > Settings > Control Panel > Scheduled Tasks to open your list of scheduled tasks. Right-click on the task you created in dtSearch Indexer and select "Run Now". This will test whether the user name and password are set up correctly.

If the message "Could not start" appears, it means that the user name and password are incorrect. Double-click the task to open its properties and check the "Run as" user name and password to make sure they are correct.

Sometimes Task Scheduler forgets a password after a while, so if a task was working for a long time, the password might need to be re-entered.

A password is required for any Scheduled Task. If your user account does not have a password, Windows will not allow it to run a Scheduled Task. See:

Scheduled Tasks Cannot Run with a Blank Password

http://support.microsoft.com/default.aspx?scid=kb;en-us;310715

Symptom: Scheduled task runs but the index is never updated

Check that the user account for the task has a dtSearch folder. Start dtSearch Desktop and click Help > dtSearch Diagnostic Tools. A dialog box with a list of dtSearch-related log files will appear. Look for a file named dtSearch_Errors_T.log and if you find it, click on it and click the Edit button to open the file. This file should provide information about why dtSearch was unable to run in the scheduler. Usually this is due to a mismatch between the user account assigned to the task and the account used to set up the dtSearch indexes.

If the user account for a task has never been used to run dtSearch interactively, dtSearch will not have any registry settings when run as that user, so it will not be able to update the index. To prevent this from happening, log in with the user name and password assigned to the dtSearch task, and run dtSearch Indexer to verify that the dtSearch indexes are visible to the user and that the user has the correct dtSearch folder.

Symptom: Scheduled task runs, and the index is updated, but the index is empty

This can occur if you are indexing mapped network drives and the scheduled task runs when you are not logged on. Drive mappings may not be available when a scheduled task runs even if you set up the task to log in with your user name and password. To index network resources in a scheduled task,

(1) Set up the folders to index as UNC paths rather than mapped drive letters. In the dtSearch Index Manager, you can convert a mapped drive to a UNC path by right-clicking on the path and selecting "Make UNC" from the menu that appears.

(2) Verify that the user name and password for the scheduled task will enable the task to access the network resources.


'Unable to create a directory' message installing dtSearch

dtSearch Home

Downloads

Quick Start

'Unable to create a directory' message installing dtSearch

Last Reviewed: March 7, 1999

Article: DTS0126

Applies to: dtSearch 5, 5.1, 5.11

Symptoms

When installing dtSearch, the following message appears: "Unable to create a directory under ___. Please check write access to this directory."

Cause

Adobe Acrobat 4 is installed on the machine and you attempted to install dtSearch Adobe Acrobat (PDF) indexing support. Versions of dtSearch prior to dtSearch 5.2 are not able to install the dtSearch Adobe Reader plug-in for Adobe Acrobat 4.

Resolution

Obtain dtSearch 5.2, a free upgrade from dtSearch 5.11, from www.dtsearch.com.


dtSearch Web Quick Start

dtSearch Home

Downloads

Quick Start

dtSearch Web Quick Start

Last Reviewed: April 21, 2003

Article: DTS0162

Applies to: dtSearch Web 6

dtSearch Web is a search engine that you can install on a Web server to publish documents on your web site. It can perform fast indexed searches using the same search features that dtSearch Desktop supports -- fuzzy searching, phonic searching, natural language searching, boolean logic, proximity, etc. Indexed documents can be in any format that dtSearch supports, such as HTML, PDF, XML, Word, Excel, PowerPoint, WordPerfect, RTF, and ZIP archives.

After a search, dtSearch Web will convert documents as needed to HTML for display, with hits highlighted. If a retrieved document is already in HTML, dtSearch Web will display it with hits highlighted while preserving the HTML attributes (so links and images will work in your browser).

dtSearch Web requires Microsoft Internet Information Server (IIS) version 4 or later. Internet Information Server is included in all versions of Windows 2000 and is available from Microsoft as part of the Windows NT Option Pack.

This Quick Start will describe how to get a basic search form running on your web site with minimal effort. You can create any number of search forms, each with its own set of option settings and index selections.

1. Install the dtSearch and dtSearch Web program files on your web server.
If you have dtSearch Web on CD, run the setup program to install. If you downloaded dtSearch Web from the internet, follow the download instructions to open the dtSearch Web archive and install the files.

2. Build an index of your documents
dtSearch Web uses the same indexes as dtSearch, so if you already have a dtSearch index of the documents that you want to make searchable, you can use this index with dtSearch Web. If not, follow the dtSearch Quick Start to set up an index. If you are indexing a dynamically-generated web site (such as a web site generated by a content manager) see "How to use dtSearch web with dynamically-generated content" for information on using the dtSearch Spider to set up your index.

3. Run dtSearch Web Setup
In dtSearch, click "dtSearch Web Setup..." in the File menu. dtSearch Web Setup will display a drop-down list of the web sites on your server, and below it a list of the virtual directories defined for each site.

4. Select the web site to use with dtSearch Web
If you have more than one web site on this server, you can install dtSearch Web on each of them, or just one. (To install dtSearch Web on additional web sites, repeat the procedure described in this Quick Start for each site.)

5. Select the folder where dtSearch Web should be installed
The "Scripts" folder is recommended but you can choose another folder by selecting it from the drop-down list. dtSearch Web Setup will create a "dtSearch" folder under the folder you select where the dtSearch Web files will be installed. Click the Install or Upgrade button to install the dtSearch Web files in the folder you selected.

If the folder you selected does not already have 'Execute' permission enabled, dtSearch Web Setup will ask if it can enable 'Execute' permission in the folder so that dtSearch Web can run from that folder.

Note: Under Windows 2003 Server an additional step is needed to enable dtSearch Web. Please see this article for instructions: How to install dtSearch Web on Windows 2003 Server

6. Build a Search Form for your site
Click the Build Search Form... button to build a search form to use with your site. You can make as many search forms as you want for each site, and after dtSearch Web Setup has generated a search form, you can use an HTML editor such as Microsoft FrontPage to edit it to fit into your web site.

In the Form Builder dialog box, click the Indexes tab to select the indexes that you want to put on the search form. Check the box next to each index to include it on your search form.

7. Click OK to build the search form.
After the search form is built, dtSearch Web will open it in your browser so you can try out a search. Once you have a basic search form working, you can run Form Builder again to customize the search form, the appearance of search results, and other options.

Making Documents Accessible

To provide access to documents on your web server, designate each folder that contains documents to be published as a virtual directory. To do this with Internet Information Server, use the Internet Service Manager to add an "Alias" for each folder.

After you add virtual directories to a site, run dtSearch Web Setup to update dtSearch Web with the new information. You do not need to reinstall dtSearch Web or run Form Builder again -- just start dtSearch Web Setup and exit it, and dtSearch Web will know about the new directories.

Customizing the search form and search results

Once you have a basic search form working, you can use your HTML editor to modify the search form and the appearance of search results. For more information, see the dtSearch Web User's Manual, dtSearch_Web.chm.


dtSearch Web Demos

dtSearch Home

Downloads

Quick Start

dtSearch Web Demos

Last Reviewed: April 10, 2004

Article: DTS0188

Applies to: dtSearch Web 6.32

Demo Search forms

Search form | What it shows
Search dtSearch Support Articles | Searching of HTML and PDF dtSearch technical support articles, using a simple search form.
Simple search | Searching of all of the demo document collections, using a simple search form. Also demonstrates use of cookies to save search form state from one search to the next.
Advanced search | Search form with advanced searching features.
Field searching | Search form with advanced searching features and date and field searches, implemented using JavaScript on the search form.
Shakespeare XML | Demonstrates highlighting of hits in XML, using an .xsl stylesheet to format the XML for display. See FAQ article dts0183.htm for more information.
ASP Search | Demonstrates how to use the dtSearch Engine's COM interface to search using ASP. For VBScript source code to the demo, click here. For a similar demo implemented in C#, see FAQ article dts0172.htm.
ASP Search with synopsis | Demonstrates how to use the dtSearch Engine's COM interface to search using ASP, with hits in context included in the search results list. Also demonstrates a simplified, no-frames search results interface. For VBScript source code to the demo, click here.
ASP Template | Demonstrates how to use the dtSearch Engine's COM interface to search using ASP, with an ASP template file used for search results. For VBScript source code to the demo, click here.
Embedded search form | Search form that can be embedded in another web page so that search results will pop up over the original page.

Document collections

These document collections are used in the sample search forms listed above.

Document Collection

What it shows

dtSearch Support Articles (HTML)
Current articles on the dtSearch Web site

Searching of HTML files with:
• highlighted hits
• embedded images intact in the display
• embedded links fully operational in the display
• multiple hit and file navigation options

dtSearch Web site (HTML and PDF)

Searching a web site indexed with the dtSearch Spider

Tax Forms & Publications (PDF)
IRS tax forms, instructions and publications in PDF format. These documents are posted for demonstration purposes only and are provided "as is." They may be out-of-date or incorrect. For current tax information, see www.irs.gov.

Searching of PDF files with:

• highlighted hits

• embedded images intact in the display

• multiple hit and file navigation options

OCR Demo (PDF)

dtSearch "white papers", scanned from paper copies.

Searching of OCR files with "hidden text" highlighting:

• PDF files display complete original images of the OCR'ed documents.

• The text is "hidden" in the sense that simply opening the PDF file displays only the scanned image, not the underlying OCR'ed text.

• Because the OCR'ed text is "hidden" in the file, however, dtSearch can index and search it.

• After a search, dtSearch Web can highlight hits in the OCR'ed text so that the highlights appear directly on the scanned image.

Shakespeare (XML)
Works of William Shakespeare, converted to XML.

Searching of XML data with:
• searches limited to precise combinations of fields and subfields in hierarchical XML data (as well as full-text searching)
• highlighted hits
• multiple hit and file navigation options
Note: dtSearch Web / Engine also supports field searching in other database formats, such as in HTML, PDF and SQL. See Support for more information


dtSearch Web document access problems

dtSearch Home

Downloads

Quick Start

dtSearch Web document access problems

Last Reviewed: April 8, 2002

Article: DTS0129

Applies to: dtSearch Web 5.2, dtSearch Web 6

Symptoms

1. dtSearch Web says it cannot display a document because the document is not in a virtual directory folder.

2. Users are prompted for a password when attempting to search with dtSearch Web, or when attempting to view a document in search results.

Cause

1. dtSearch Web says it cannot display a document because the document is not in a virtual directory folder.

Documents on an internet site are usually placed in virtual directory folders. These are folders that have been designated as part of your site and that have been given a "virtual directory" or "alias" such as /Docs. By default, dtSearch Web will only display documents that are located in a virtual directory folder, and will display an error message if a user tries to access documents located in other folders. The purpose of this is to provide an additional layer of protection against unauthorized access to documents.

To create a virtual directory,

1. Start Internet Services Manager

2. Open the web site

3. Click Action > New > Virtual Directory

4. Set up the virtual directory to point to the folder with the documents that you want to make accessible

5. Close Internet Services Manager

6. Click Start > Programs > dtSearch > dtSearch Web Setup

7. As soon as the dtSearch Web Setup dialog box appears, close it

You do not need to reindex your documents after creating a new virtual directory. However, you do have to run dtSearch Web Setup again so that dtSearch Web will know about the new virtual directory. (It is not necessary to reinstall dtSearch Web or to create a new search form. All that is needed is to start dtSearch Web Setup and then to close the program. This will refresh the registry tables that dtSearch Web uses to identify virtual directories.)

Allowing access to non-virtual folders. dtSearch Web includes an option to allow documents outside of a virtual directory to be displayed. To enable access to non-virtual folders, check the Allow access to non-virtual folders box in the Options tab in the dtSearch Web Setup dialog box. If this option is checked it is very important that all files accessible from the web server be properly secured against unauthorized access. Please see the "Security" topic in the dtSearch Web help file for more information. Generally, instead of allowing access to non-virtual folders, it is better to create new virtual directory for any additional documents that you want to make available.

2. Users are prompted for a password when attempting to search with dtSearch Web, or when attempting to view a document in search results.

A password prompt means that dtSearch Web tried to access the document or index using the rights of the user currently accessing the site, and access was denied. When dtSearch Web receives an "access denied" error trying to open an index or a document, it returns a code to the web browser that tells the web browser to prompt for a user name and password.

A password prompt that appears when a user clicks the Search button indicates that access was denied to the index. A password prompt that appears when a user clicks on a document in search results indicates that access was denied to the document.

Troubleshooting Access Problems

1. If the document or index that cannot be accessed is on a different server, see "Delegation" under "Additional Information," below. The remainder of this troubleshooting section will assume that the resource is on the same server as dtSearch Web.

2. In Internet Service Manager, create a virtual root for the folder containing the document or index with the access problem, if one does not already exist.

3. Put a sample HTML file in the same folder as the document or index with the access problem.

4. Attempt to access the sample HTML file from a client machine. You should see the same password prompt that appeared in dtSearch Web.

5. Adjust the security settings for the folder until the HTML file can be accessed from the client machine. To change the security settings for the folder, right-click it in Explorer, select "Properties," and then click the Web Sharing tab.

Example 1. Problems accessing the file c:\docs\abc.doc. Create a virtual root making "/docs" an alias for "c:\docs", and put a sample html file, sample.htm, in c:\docs. Then try to access /docs/sample.htm through a browser, and adjust the security settings until the file can be accessed.

Example 2. Problems searching the index c:\docs\index. Create a virtual root making "/docs" an alias for "c:\docs", and put a sample html file, sample.htm, in c:\docs\index. Then try to access /docs/index/sample.htm through a browser, and just the security settings until the file can be accessed.

Once you are able to access the sample HTML file from a client machine, you should be able to access the dtSearch Web index and documents as well.

Additional Information

Delegation

Delegation problems occur when a user on a client workstation tries to use Web Server A to access a resource on Server B. For example, Web Server A may contain dtSearch Web and Server B may have the documents and indexes to be searched. When the administrator tries a search while sitting at Server A, the administrator is able to access documents and indexes on Server B. However, when sitting at a client workstation, the administrator cannot get access to the Server B indexes and documents through dtSearch Web on Server A. The same problems will occur with any other resource on Server B, such as a web page. The administrator is unable to access the documents or indexes even after logging in with the Administrator user name and password.

The KnowledgeBase articles and the Stabbert article cited below under "Resources" explain delegation problems and provide workarounds for them. The problem is that the Windows NT 4 security model does not allow Web Server A to use a logged-in client's authentication to access secured resources on Server B using Windows NT Challenge/Response authentication. As the article puts it, "Using Windows NT Challenge/Response, there is no way a process relying on impersonation can access so much as a text file on another Windows NT box." (Impersonation is the process that Internet Information Server (IIS) uses to respond to internet requests. When a user requests information from an IIS web site, the IIS process "impersonates" the user when accessing secured resources.)

The workarounds all involve stripping away some of the security protection around the resources on the two servers. One workaround is to switch from Challenge/Response authentication to Basic Authentication, which results in passwords being sent through the network without encryption. An alternative is to open up both web Web Server A and Server B to anonymous access, and to ensure that Server B has an anonymous user and password that match the anonymous user and password for anonymous access to Web Server A. The Stabbert article contains details on both workarounds.

Workarounds that do not involve compromising security are:

(1) Transfer the resources from Server B to Web Server A so that they can be accessed directly,

(2) Put dtSearch Web on Server B and give it its own search page,

(3) Link directly to the documents in search results (so hits will not be highlighted). To do this, change %%HighlightLink%% to %%DirectLink%% in your dtsearch_options.html file.

(4) Build an index that caches documents, so dtSearch Web can highlight hits by using the cached copy of each document instead of accessing the original server. For more information, see:

dtSearch v.7 index format

http://www.dtsearch.com/index7.html

Resources for Windows 2000 and Windows NT Web Site Security

How to Create Virtual Directories to a Remote Novell NetWare Share

Microsoft KnowledgeBase Article Q285159

http://support.microsoft.com/support/kb/articles/Q285/1/59.asp

HOWTO: Accessing Network Files from IIS Applications

Microsoft KnowledgeBase Article Q207671

http://support.microsoft.com/support/kb/articles/Q207/6/71.ASP

Security Ramifications for IIS Applications

Microsoft KnowledgeBase Article Q158229

http://support.microsoft.com/support/kb/articles/Q158/2/29.asp

HOWTO: Impersonate a User from Active Server Pages

Microsoft KnowledgeBase Article Q248187

http://support.microsoft.com/default.aspx?scid=kb;EN-US;248187

P. Enfield, Developer Support Engineer, Microsoft, Implementing a Secure Site with ASP (October 24, 1997), MSDN CD.

S. Stabbert, Microsoft Internet Developer Support, Authentication and Security for Internet Developers (October 10, 1997), MSDN CD.

M. Streebe, C. Perkins, and M. Moncur, NT 4 Network Security (Sybex Network Press 1999) ISBN 0-7821-2425-9.


dtSearch Web 6 system requirements and performance

dtSearch Home

Downloads

Quick Start

dtSearch Web 6 system requirements and performance

Last Reviewed: December 20, 2004

Article: DTS0139

Applies to: dtSearch Web 6

For information on dtSearch Desktop system requirements, please see this article:

dtSearch system requirements

dtSearch Web 6 currently only supports Internet Information Server 4, Internet Information Server 5, or Internet Information Server 6. This effectively limits it to Windows 2003, Windows XP, Windows 2000 or Windows NT. It will work with both the Workstation and the Server versions of these platforms.

dtSearch Web 6 cannot be installed on a web server that uses Windows 95, Windows 98, Windows ME. It also cannot be installed on a web server other than Internet Information Server, even if that web server is advertised as supporting ISAPI.

Additional Information

The dtSearch Text Retrieval Engine (dten600.dll) will run on any 32-bit x86-based version of Windows, including Windows 95, 98, ME, NT, 2000, XP, and 2003 Server. However, dtSearch Web 6 (which includes the engine and an ISAPI-based component to respond to search requests) requires Internet Information Server 4 or later.

The reason for this limitation is that dtSearch Web 6 uses information that Internet Information Server maintains in the Windows Metabase to detect the web server configuration. The ISAPI specification does not provide a way to obtain this information. ISAPI only covers processing of queries, and does not provide a sufficiently functional mechanism for detecting the virtual directory mappings on the server. Therefore, even if another web server is fully ISAPI-compliant, dtSearch Web 6 will not be able to detect its web server configuration.

Linux. Support for Apache under Linux is planned but is not yet available. Currently the Linux version of the dtSearch Text Retrieval Engine has a Java and C++ API only, so some programming is needed to make any use of it.

Hardware Requirements

dtSearch Web does not have any specific hardware requirements. As long as Windows can run efficiently (without a lot of disk swapping) little additional memory is needed for dtSearch.

While a search is executing, dtSearch Web may allocate between 1 and 10 Mb of memory to process that search. Once the search is done (most searches are completed in substantially less than a second), the memory is released. Therefore, unless a site receives thousands of searches per hour, the amount of memory needed for dtSearch Web will be very small.

Recommendations for best performance:

1. At least 256 Mb of memory (so Windows will run efficiently)

2. Defragmented hard disk maintained using a tool such as Diskeeper

3. At least 25% of the hard disk space unused (so the defragmentation utility will run efficiently -- dtSearch Web does not create any temporary files).

4. The dtSearch index folder should not be compressed using NTFS compression, because this can slow index access considerably. NTFS compression will provide little benefit in reducing the size of a dtSearch Index, because most of the index data is stored in a highly compressed format.

5. Index updates will be faster and more reliable if dtSearch Desktop runs on the machine where the index is stored. It matters much less where the documents are located, because dtSearch just needs to read each document once to index it.

Shared Servers

dtSearch Web 6 can be run from a shared web server, because it supports installation on web servers with multiple web sites. When dtSearch Web Setup is started, it will display a drop-down list of the web sites defined on the server. Select the site to use and click the Install button to install dtSearch Web for that site.

dtSearch Web cannot be installed or configured through a web browser interface, though. Therefore, you will need either physical access to the server where dtSearch Web will be installed, or you will need to use remote access software such as PC Anywhere.

Scalability and Clustering

dtSearch Web is designed to operate in a completely "stateless" manner, meaning that no information about a user is retained between requests. This makes adding capacity to a site easier because you can simply add more servers, without the need to tie each user session to a particular server. Therefore, the recommended way to add capacity to a dtSearch Web search site is to clone the site on multiple, separate servers, and to use web server load-balancing software to allocate user requests among the servers.

dtSearch Web uses an efficient threading model so that it can handle multiple concurrent queries simultaneously under IIS, and can make efficient use of multiple processors where present. There is no built-in limit on the number of concurrent users the dtSearch Engine can handle. Searching is done without any need for file or record locks, so aside from the need to share CPU and other hardware resources, one search user has no effect on another concurrent search user.


How to add custom fields to dtSearch Web search results

dtSearch Home

Downloads

Quick Start

How to add custom fields to dtSearch Web search results

Last Reviewed: April 18, 2001

Article: DTS0158

Applies to: dtSearch Web 6

In addition to basic information about a file, such as the filename, date, and size, dtSearch Web can also display customized fields based on document properties. For example, you could display the Subject property of a PDF file or the Author property of a Word document.

To set up a custom field to appear in search results, there are two steps: (1) Tell dtSearch the name of the field so that when dtSearch indexes the documents, it will get the contents of the field and store them in the index so they can be displayed in search results. (2) Customize the search results format in dtSearch Web Setup to include the field.

Suppose you want to display the Subject property in search results. First, tell dtSearch that the Subject property should be stored in the index so that it can appear in search results. To do this,

(1) Select the Text Fields tab in the Preferences dialog box.

(2) Click New... and enter Subject as the name of the field.

(3) Check the Display field in search results box.

(4) Click OK to save your changes. (You do not have to enter anything for the filename filters or other spaces in the Text Fields page.)

(5) Index the documents that have the Subject field. (dtSearch looks for field data when a document is indexed, so it will not have the Subject field for previously-indexed documents. If you have already indexed the documents, update the index with the Clear index before adding documents box checked in the Update Index dialog box.)

To see if dtSearch found the Subject field in your documents, do a search in dtSearch Desktop. There should be a new Subject column in the search results list.

The next step is to tell dtSearch how the Subject field should appear in dtSearch Web. dtSearch Web keeps the search results format with each search form, so you will need to create a new search form. (The following steps assume that you have already set up dtSearch Web. If you have not, please see the dtSearch Web Quick Start for instructions on setting up dtSearch Web.)

(1) Click dtSearch Web Setup in the File menu

(2) Click the Build Search Form button

(3) Click the Indexes tab and select the indexes that you want to include on this search form.

(4) Click the Search Results tab

(5) Click the Add... button to add an item to search results. The Search Results Item dialog box will appear.

(6) Under Name enter Subject

(7) Under Label to appear in search results enter <B>Subject: </B>

(8) Under Content enter %%Subject%%

(9) Click OK to close the Search Results Item dialog box, then click OK again to build the search form.

When you search using the form you just built, the search results list will include the Subject field.

Additional Information

For information on customizing search results generally in dtSearch Web, please see the Customizing the search results format topic in the dtSearch Web help file (dtSearch_Web.chm).

For information on editing the dtSearch_options.html file directly to customize search results format, please see the Generated files topic in the dtSearch Web help file.

Related Topics

How to get field data in search results

How to add fields to documents


How to display links to the original files in dtSearch Web

dtSearch Home

Downloads

Quick Start

How to display links to the original files in dtSearch Web

Last Reviewed: February 17, 2003

Article: DTS0193

Applies to: dtSearch Engine 6

dtSearch Web can create two types of links in search results: highlighted links, which tell dtSearch Web to convert a retrieved file to HTML with hits highlighted, and un-highlighted links, which cause a document to open in its native application.

Changing the link type for all files

To change the link type with the dtSearch "Build Search Form" tool, click the "Search Results" tab

(1) Click the Search Results tab,

(2) Click on the item in search results that will have the link to the original document, and

(3) Click the Modify button to change the link type

You can also change the link type by editing the dtsearch_options.html file. Open the file in notepad and look for the ResultsTableItem section. When dtSearch Web generates search results, it uses the text in this section as a template for the HTML that appears in search results. The %%HighlightLink%% symbol will be replaced by a link that produces hit highlighting, and the %%DirectLink%% symbol will be replaced by a direct link to the document (which will open the document in its native viewer). Therefore, to convert a link from highlighted to direct, just replace %%HighlightLink%% with %%DirectLink%%. You can also include both link types in each search results item, if you want to give users a choice to view documents either way.

Changing the link type for specified file types

dtSearch Web has an option to specify that certain file types should always be displayed without conversion to HTML. In the Build Search Form tool, click "Document Display Options" and, under "File types to display without conversion to HTML", list the extensions of the file types that you want to appear in their native viewer. For example, to make Excel and Word files appear in Excel and Word, include DOC and XLS in the list.

In the dtsearch_options.html file., this is the "UnconvertedTypes" setting.


How to install dtSearch Web on Windows 2003 Server

dtSearch Home

Downloads

Quick Start

How to install dtSearch Web on Windows 2003 Server

Last Reviewed: April 21, 2003

Article: DTS0186

Applies to: dtSearch Web 6

Windows 2003 Server has a new "Web Service Extension" manager that prevents a web application from running unless it is specifically enabled. When an application is invoked and it has not been enabled, a 404 ("Not Found") error results. This article describes steps to register dtSearch Web with the Web Service Extensions manager.

Before you install

(1) Make sure that the Internet Information Service (IIS) is installed. By default, IIS is not installed in Windows 2003, so you must either use Add/Remove Programs or Configure Your Server to set it up.

(2) Log in as the Administrator or a user with equivalent rights. Other user accounts will not have sufficient permissions to enable a web application.

Installing dtSearch Web

(1) Run dtSearch Web Setup as usual. When you install dtSearch Web, note the location where dtisapi6.dll was copied (you will need it in step 7 below).

(2) Open Internet Services Manager

(3) In Internet Services Manager, open "Web Service Extensions"

(4) Click "Add a new Web service extension"

(5) Under "Extension name" enter "dtSearch Web"

(6) Click "Add..." to add a file under "Required Files"

(7) Locate the dtisapi6.dll file under the c:\inetpub folder, and select it.

(8) Click OK.


How does dtSearch Web handle security?

dtSearch Home

Downloads

Quick Start

How does dtSearch Web handle security?

Last Reviewed: March 27, 2003

Article: DTS0127

Applies to: dtSearch Engine 6, dtSearch Web 6

Limiting access to documents

dtSearch Web does not alter Windows security settings and only provides access to documents when the user seeking access has the necessary permissions. To secure a site, or to make a site open to the public, use Explorer and Internet Service Manager to set the permissions you want. dtSearch Web will recognize those permissions automatically. There is no need to rebuild your indexes after changing security settings.

If a user has access to an index, but not to the documents, dtSearch will let the user search the index but will not enable the user to open documents from the search results list. Therefore, it is preferable to make the security settings on each index match the security settings on the documents.

Multiple collections with different security settings

A common way to handle groups of documents with different security settings is to set up a different index for each group of documents, and to give each index the same security rights as the documents that it covers. This way only users who have access to the documents will be able to search the index.

Implementing complex security categories

The dtSearch Engine's developer API has a SearchFilter object that can be used to implement more complex security schemes, such as access rights stored in a database that specify the users who can access each document. A SearchFilter object specifies, for one or more indexes, which documents can be returned in a search. SearchFilter objects can be constructed from query results or directly from document ids, and can be saved to disk and read from disk.

To use SearchFilters to implement security, an application would first create a series of SearchFilters, one for each security category, based information in the database (or other repository) that specifies security rights. When a user submits a query, the application would select the SearchFilter that corresponds to the user's security category and attach it to the search. When the search executes, it will only return documents that the SearchFilter permits.

For more information on SearchFilter objects, see the dtSearch Engine help file, dtengine.chm, and this article:

Implementing Document Classification.

Documents on multiple servers

It is much easier to work with security settings if the indexes, documents, and dtSearch Web are all on the same IIS server. Otherwise, a security issue known as "delegation" makes limiting access to the documents much more difficult. Because of the difficulty involved in dealing with delegation, it is recommended that documents, indexes, and dtSearch Web all be kept on the same server, even if this means keeping an extra copy of the documents. For more information on resolving delegation issues, please see the Microsoft KnowledgeBase articles listed at the end of this article: "dtSearch Web document access problems"

Virtual directories

Documents on an internet site are usually placed in virtual directories. These are folders that have been designated as part of your site and that have been given an "alias" such as /Docs. dtSearch Web will only display documents that are located in a virtual directory, and will display an error message if a user tries to access documents located in other folders. The purpose of this is to provide an additional layer of protection against unauthorized access to documents.

There is an option to override this setting in the Document Display Options tab of the dtSearch Web Setup dialog box, but this option is not recommended, because of the risk that documents could be inadvertently made available on the web through dtSearch Web. In nearly all cases, you can create a virtual directory for the documents, which is a much safer way to make them available.


How to use dtSearch Web with dynamically-generated content

dtSearch Home

Downloads

Quick Start

How to use dtSearch Web with dynamically-generated content

Last Reviewed: October 26, 2004

Article: DTS0180

Applies to: dtSearch Web 6.4 and later

To use dtSearch Web to search a web site, you must first create an index of the web site with the dtSearch Indexer. If the web site consists of static web pages (HTML files, PDF files, etc.), you can just click Add folder in the dtSearch Indexer to add the folders with the web pages to the index. However, if the web site is dynamically generated, by a database, ASP or ASP.NET program, or content manager such as Microsoft Content Manager Server (CMS), there will be no folder with web pages to index. Instead, you can use the dtSearch Spider to crawl the web site.

To index your web site with the dtSearch Spider, click Add Web in the dtSearch Indexer and provide the starting address for the crawl (usually your site's home page). The dtSearch Indexer will traverse the web site by following the links connecting the pages. Because the Spider follows the same links that a web browser would use to navigate your site, it will be able to index the dynamically-generated content just as it is presented on your web site.

Highlighting hits

You will need dtSearch Web 6.10 or later to highlight hits in content indexed using the dtSearch Spider. The option to control whether hits are highlighted in content indexed using the Spider is in dtSearch Web Setup's Form Builder dialog box, in the Search Results tab, You can also change this setting after a search form is created by editing the dtSearch_options.html file. The option setting is controlled by this item in the dtSearch_options.html file:

<BR><HR><I>Highlight documents indexed via HTTP: </I>

<!-- $Begin HighlightHttpDocs -->

1

<!-- $End -->

If the option is set to 0 (off), then dtSearch Web will return direct links to any pages indexed by the spider, so the page will be displayed just as it appears normally. If the option is set to 1 (on), then dtSearch Web will request the page itself, add hit highlight markings, and then display the page with hits highlighted.

Excluding sections of web pages

Often pages generated by a content manager will contain sections of HTML that you would not want to be indexed, such as the table of contents and navigation menus. To tell dtSearch not to index parts of an HTML file, add HTML comments around the text to be excluded, like this:

<!--BeginNoIndex-->

... nothing here will be searchable...

<!--EndNoIndex-->

The BeginNoIndex and EndNoIndex tags must look exactly as they do in this example. dtSearch will skip everything between the two markers when it indexes web pages.

Excluding sections of web sites

In the dtSearch Indexer, you can use filename filters and exclude filters to limit indexing by filename or folder name. For example, you could use a filter of */OnlyThisFolder/* to limit indexing to documents in a folder named OnlyThisFolder, or you could use an exclude filter of */NotThisFolder/* to prevent anything in the folder named NotThisFolder (or subfolders) from being indexed. For more information on filename filters, see: How to exclude folders from an index.

Additionally, the dtSearch Spider checks for robots.txt and robots META tags in web pages, so you can use a robots.txt file or embedded tags in web pages to specify whether they should be indexed, and whether the Spider should check them for links when indexing the site. For more information on robots.txt and the robots META tag standard, see:

http://www.robotstxt.org/wc/meta-user.html

http://www.robotstxt.org/wc/exclusion.html

Related Topics

For more information on creating indexes, see "dtSearch Quick Start".

For more information on setting up dtSearch Web, see "dtSearch Web Quick Start".

For more information on using the dtSearch Spider to index web sites, see "How to index a web site with the dtSearch Spider".


How to use dtSearch Web with ODBC

dtSearch Home

Downloads

Quick Start

How to use dtSearch Web with ODBC

Last Reviewed: March 18, 2002

Article: DTS0121

Applies to: dtSearch Web 5.x, 6

dtSearch and the dtSearch Engine use the Microsoft ODBC database drivers to access desktop database files (Access, Fox Pro, Paradox, DBase) through ODBC.

The default installation of dtSearch Web does not include support for access to desktop database files using ODBC. As a result, you will get a "dtSearch Engine Unable to Process Request" message from dtSearch Web if you try to click on the name of a database row in search results.

To enable ODBC database support in dtSearch Web,

1. Install Microsoft Office on the server. If Microsoft Office is not installed, ODBC may not work and dtSearch will not be able to access rows from a database.

2. Locate the dtSearch Web folder on the server. To do this, start dtSearch and click File|dtSearch Web Setup. In the dialog box, look for the box labelled "/Scripts directory on server." The name will be something like c:\inetpub\scripts. dtSearch Web will be in a subfolder of this folder named "dtsearch" (for example, c:\inetpub\scripts\dtsearch).

3. Exit dtSearch.

4. Create a folder named "viewers" under the dtSearch Web folder (for example, c:\inetpub\scripts\dtsearch\viewers)

5. Copy the file dtv_odbc.dll from the bin\viewers subfolder of the folder where dtSearch is installed into the "viewers" folder you just created.

6. Reboot the server to ensure that dtSearch Web will be reloaded. Once dtSearch Web restarts, it will see the dtv_odbc.dll file and use it to access database files.

ODBC Driver Issues

dtSearch Web uses the Microsoft ISAPI web server interface and so operates in a multi-threaded environment, meaning that the program may simultaneously handle requests from many users. Some versions of the ODBC database drivers are not "thread safe" and will fail if run from a web server. Therefore, before you set up ODBC database support in dtSearch Web, you should check the documentation for the ODBC database drivers you have installed to see if they are thread safe. For more information, see this article on the Microsoft web site:

TN067: Database Access from an ISAPI Server Extension

Unfortunately, Microsoft Knowledgebase Article Q167907, which had more specific information about using ODBC in multi-threaded applications, apparently no longer exists.

For information on updating your ODBC drivers to the latest versions, please see "Troubleshooting Database Indexing."


How to use dtSearch Web with PDF files

dtSearch Home

Downloads

Quick Start

How to use dtSearch Web with PDF files

Last Reviewed: April 10, 2003

Article: DTS0116

Applies to: dtSearch Web 6

See also: How to use dtSearch or dtSearch Web with OCR

dtSearch Web can index and search PDF files, and can highlight hits in retrieved files. (There is also a developer API that provides a way to do this using Active Server Pages. For more information on this API, see the dtsearch.asp sample included with dtSearch Web.)

Creating PDF Files from Documents

To convert individual documents to PDF, you can use the PDF Writer tool that is included with Adobe Acrobat (www.adobe.com)

Other tools such as AdLib (www.adlibsys.com) can convert larger volumes of documents automatically. AdLib eXpress Server can convert over 300 file formats to PDF (or TXT) via watched folders, incoming emails, instructions sent from workflow processes, API calls or other methods, and also includes features such as PDF merge/split, PDF hyperlinks, watermarking, scripting and email.

Server Requirements

1. Install dtSearch Web.

2. Make sure the web server has Microsoft Internet Information Server 4 or later. Older web servers do not handle PDF files correctly.

3. Put the PDF files you want to publish in a folder that has a virtual root (or "Alias") name, like /docs or /webfiles. A subdirectory of the root folder for your web site (i.e., c:\inetpub\wwwroot\docs) will work.

4. Index the PDF files with dtSearch Desktop 6. See the dtSearch 6 Quick Start for more information on how to index documents with dtSearch.

5. Use dtSearch Web's Search Form Builder to build a search form. See the dtSearch Web Quick Start for more information on setting up dtSearch Web.

Setting up the Clients

PDF hit highlighting works best with current versions of the web browser and Adobe Reader plug-ins. The following are recommended:

- Adobe Reader 4.05 or later

- Internet Explorer 5.01 (or later) or Netscape 4.7 (or later)

To test your client setup, you can use the dtSearch Web demo site (go to www.dtsearch.com and click on the link to the dtSearch Web Demo). Adobe Reader plug-in files are available from www.adobe.com.

Changing the Highlight Color

The color used to highlight hits in Adobe Reader is controlled by the client machine's Display Options in Windows. The specific option is the "Selected Items" color, which defaults to white-on-blue. If you change it to black-on-yellow, Adobe Reader will show highlights in yellow. (This will also make your Windows menus and list boxes use yellow highlights.) There is no way to control this from the server.

Optimizing Page Downloads

When a user clicks on a retrieved PDF file in the search results list, the user will initially see the first page of the document that contains hits. Downloads of large files are optimized so that only the page(s) that a user views are downloaded. For example, if a user clicks on a 1000-page document with a hit on page 87, page 87 will be downloaded initially and displayed. If the user navigates to the next page in Adobe Reader, page 88 will then be downloaded and displayed.


PDF file appears as blank page in dtSearch Web

dtSearch Home

Downloads

Quick Start

PDF file appears as blank page in dtSearch Web

Last Reviewed: April 8, 2002

Article: DTS0118

Applies to: dtSearch Web 5.2, dtSearch Web 6

Symptoms

After a search, click on a document in the search results list and a blank page appears in the document frame.

Possible Causes

- PDF files are not in a virtual directory

- Missing Adobe Reader plug-in

- Older version of Internet Explorer

- Older web server version that handles PDF files incorrectly

- Directing PDF files to a new window using Target= links

Troubleshooting Steps

1. Virtual Directory. PDF files cannot be displayed over the web if they are not ina virtual directory. (This is because Adobe Reader must have a web URL that points directly to a PDF file in order to display it.) If the "Location" item in the search results list shows either a drive letter ("c:\something") or a UNC path ("\\server\share\something") then the PDF files are not in a virtual directory. For information on setting up a virtual directory for your documents, click here.

2. Client Software. Web browsers use an Adobe Reader plug-in to display PDF files. If this plug-in is not installed, or if an older version is not installed, PDF files may not display correctly.

To use Adobe Reader in Internet Explorer, you need:

- Internet Explorer 5.5 or later (older versions have problems displaying PDF files)

- Adobe Reader 4.05 or later, available from www.adobe.com.

First install Internet Explorer 5.5, then Adobe Reader.

To use Adobe Reader in Netscape, you need:

- Netscape Communicator 4.5 or later

- Adobe Reader 4.05 or later, available from www.adobe.com.

First install Netscape, then install the Adobe Reader.

3. Server Software. Some web servers process PDF files incorrectly, which can also produce the blank page problem. These web servers in particular are known to have problems:

- Microsoft Internet Information Server version 2

- Microsoft Personal Web Server (included with FrontPage 98)

If you have Windows NT, upgrade to Microsoft Internet Information Server version 4 or later. If you have Windows 98 or Windows 95, the Windows NT 4 Options Pack contains an upgrade to Personal Web Server that you can install (for some reason, this Windows 98/95 product is included only with the Windows NT 4 Options Pack).

4. Link type. This potential problem applies only to developers who are using the dtSearch ASP interface to generate their own web-based interface. Links to PDF files that ask the browser to open a new window seem to be more prone to the "blank page" problem than links that reference an existing window. dtSearch Web creates links that look like this:

<a href="...url goes here..." target=doc>

The target=doc attribute tells the browser to use the "doc" frame in the dtSearch Web frameset, which is initialized by the search form. If you use a "target=new" attribute instead to direct the PDF file to a new, separate window, we have found that the "blank page" problem is likely to occur.

Additional Information

Microsoft Q177321 - Adobe Acrobat .pdf Files Appear as a Blank Window or Frame

http://support.microsoft.com/support/kb/articles/q177/3/21.asp

Adobe Document 315029 - Configuring Internet Explorer and AOL for Windows to Display PDF Files
http://www.adobe.com/support/techdocs/98fe.htm

Adobe Document 325875 - PDF Pages Don't Appear in Web Browser Window

http://www.adobe.com/support/techdocs/29776.htm

Configuring Windows browsers for Adobe Reader

http://www.adobe.com/support/techguides/acrobat/rdrwinbrwsr/main.html

Troubleshooting Problems Byteserving PDF Files on the Web

http://www.adobe.com/support/techguides/acrobat/byteserve/byteservmain.html


PDF files appear without hit highlighting in dtSearch Web

dtSearch Home

Downloads

Quick Start

PDF files appear without hit highlighting in dtSearch Web

Last Reviewed: September 3, 2002

Article: DTS0117

Applies to: dtSearch Web

Symptoms

After a search, click on a PDF file in the search results list and the document appears but hits are not highlighted.

Alternatively, the PDF file may be listed in search results but when you click on it, you get an error page and the file is not displayed.

Possible Causes

- PDF files are not located in a virtual root folder on the web server

- Old version of Adobe Reader or web browser installed on client machine

Troubleshooting Steps

1. Check that PDF files are in a virtual root folder.

PDF files must be in a "virtual root" folder for web hit highlighting to work. A virtual root folder is a folder that is visible to internet users. For example, you might have c:\docs mapped to a virtual root "/docs".

To tell if the files are in a virtual root folder, look at the list displayed in search results in dtSearch Web. If the location of a file is shown as a letter drive, like "c:\docs\sample", then the files are not in a virtual root folder. If the location of a file is shown as a virtual root, like "/docs/sample", then the files are in a virtual root folder.

To make a folder into a virtual root folder:

(1) Open Windows Explorer.

(2) Browse to the folder.

(3) Right-click the folder name and select "Properties".

(4) Click the "Web Sharing" tab.

(5) Check on the "Share this folder" option.

(6) Enter an "Alias" for the folder (this will be the name of the virtual root).

After creating a virtual directory, the next step is to refresh the dtSearch Web virtual directory tables:

(1) In dtSearch Desktop, click File > dtSearch Web Setup

(2) Close the dtSearch Web Setup program (there is no need to install)

(3) Stop and restart the www service (or just reboot the web server)

Now try searching again. The PDF files in the folder should be listed in search results with the alias that you provided as their location, and when you click on one of the files it should appear with hits highlighted.

2. Check the software installed on the client machine.

Web browsers use an Adobe Reader plug-in to display PDF files. If this plug-in is not installed, or if an older version is not installed, PDF files may not display correctly.

To use Adobe Reader in Internet Explorer, you need:

- Internet Explorer 4 or later (version 3 has problems displaying PDF files)

- Adobe Reader 4 or later, available from www.adobe.com.

First install Internet Explorer 4, then Adobe Reader.

To use Adobe Reader in Netscape, you need:

- Netscape Communicator 4.5 or later

- Adobe Reader 4.

First install Netscape, then install the Adobe Reader.


Troubleshooting -- dtSearch Web search results formatting

dtSearch Home

Downloads

Quick Start

Troubleshooting -- dtSearch Web search results formatting

Last Reviewed: March 27, 2002

Article: DTS0176

Applies to: dtSearch Web 6

Symptom: Search results appear formatted incorrectly in a horizontal table format

Troubleshooting. The format of search results in dtSearch Web is controlled by a file with the same name as the search form but with _options instead of _form at the end of the name. For example, if the search form is named dtsearch_form.html, the options file will be named dtsearch_options.html. dtSearch Web uses the HTTP "referrer" variable to locate the search form that initiated a search, and then replaces _form with _options to find the options file. If you moved or renamed the search form, copy the _options.html file so that it continues to follow the naming convention described above and try searching again.

If this does not solve the problem, either the browser may not be providing a referrer, or the search form may have been customized to the point that the referrer variable is not meaningful (for example, if the search form is being dynamically generated by an ASP script). In this situation, you can add a hard-coded path to the search form to tell dtSearch Web where to look for the options file, as described below.

Symptom: Message "%%Error: include failed because file not found" at end of search results

Troubleshooting. dtSearch Web has an option to include the search form at the end of the search results list, so that users can easily search again. This option is implemented with a %%INCLUDE line at the end of the search form. After a search, dtSearch Web uses the HTTP "Referrer" variable to locate the original search form, and appends the form to the search results list. If the search form is not found, a message will appear indicating where dtSearch Web thought the form was located.

Most commonly, this problem occurs when the search form was moved or is generated dynamically using an ASP script. To eliminate the error message:

(Option 1) Remove the %%INCLUDE line from the dtsearch_options.html file so that dtSearch Web will not try to insert the search form into the search results list. To do this, edit the dtsearch_options.html file in notepad and remove the following text:

%%Include{%%SearchForm%%}%%

(Option 2) If you want the search form to be inserted into search results, edit the original search form to add a hidden variable specifying the form's location. To do this, find the hidden SearchForm variable that dtSearch Web Setup generates, which will look like this:

<input type="hidden" name="SearchForm" value="%%SearchForm%%">

Change the value to specify the location of the search form to be inserted, like this:

<input type="hidden" name="SearchForm" value="c:\inetpub\wwwroot\search_form.html">


Troubleshooting -- dtSearch Web server error codes

dtSearch Home

Downloads

Quick Start

Troubleshooting -- dtSearch Web server error codes

Last Reviewed: September 2, 2004

Article: DTS0178

Applies to: dtSearch Web 6

Symptom

An error page is returned when you try to search or access a document in dtSearch Web

Troubleshooting Steps

1. Find the error code on the error page

HTTP Error 401 - "Access Denied" or "Unauthorized"

Check the NTFS folder permissions for the /Scripts/dtSearch folder, the index folders, and the documents. If the indexes or documents are located on another server, please see this article for information on accessing documents between servers: dtSearch Web Document Access problems

HTTP Error 404 - "Not Found"

A 404 error means that the dtSearch Web files were not found. Possible reasons for a 404 error when you try to search include:

(1) The search form created for one web site is being used with another. If you have more than one web site on a server, you may have installed dtSearch Web on site "A" but not on site "B", so when a user tries to use the search form with site "B", the dtSearch Web application is not found. To fix this problem,

(a) Run dtSearch Web Setup,

(b) Select site "B",

(c) Click the Install button to install dtSearch Web

(d) Generate a new search form for site "B"

(2) You have installed dtSearch Web under Windows 2003 Server and it is not registered as a Web Services Extension. See this article for instructions to resolve this problem: How to install dtSearch Web on Windows 2003 Server

HTTP Error 405 - Method Not Allowed

A 405 error means that the folder where dtSearch Web is installed does not have "Execute" permission enabled. To fix the problem:

(1) Start dtSearch Web Setup
(2) Select the web site where dtSearch Web is to be installed.

(3) Click the Install button. This will install the dtSearch Web files and will also set the "Execute" permission for the folder where dtSearch Web is installed.
(4) Open Internet Services Manager and check the folder where dtSearch Web was installed to verify that "Execute" permission is enabled.

dtSearch Web error -- the file could not be accessed because it is not in a virtual directory

See dtSearch Web Document access problems.

Http Error 500 - Internal Server Error

To interpret a 500 error, check the Event Viewer to obtain more information about the error that occurred (see below).

If there is nothing in the Event Viewer, check the location of dtisapi6.dll on your web server and on the search form. If dtisapi6.dll is not in the location given in the search form, the server may return a 500 error without logging anything. For example, if the search form references /scripts/dtSearch/dtisapi6.dll and dtSearch Web is really in /dtSearch/dtisapi6.dll, the server will not find dtisapi6.dll when a search is started and will return the 500 error. If the server has more than one web site, check that the virtual directory where dtisapi6.dll is located is defined for the server that is being used for the search.

2. Check the Event Viewer

To access the Event Viewer in Windows 2000 or Windows XP, click Start > Settings > Control Panel > Administrative Tools > Event Viewer. Look in the System log for an error at the same time the message appeared. The log entry will contain an Event ID that you can use to determine the source of the problem.

Event ID 36 The server failed to load application '/LM/W3SVC/5/Root'. The error was 'General access denied error'.

See these articles on the Microsoft web site for troubleshooting steps:

Minimum NTFS Permissions Required for IIS 5.0 to Work

http://support.microsoft.com/directory/article.asp?ID=KB;EN-US;Q271071&

PRB: Logon Failure: Unknown User Name or Bad Password When You Run Out-of-Process Webs (Q255770)

http://support.microsoft.com/default.aspx?scid=kb;EN-US;q255770

Event ID 36 Server failed to load application ' '. The error was '80004005'.

This error occurs when an application does not have a name specified or does not have execute permission enabled. To fix the problem,

(1) Open Internet Services Manager

(2) Open the web site

(3) Open the /scripts/dtSearch application

(4) Under Application Configuration, click "Create"

(5) Enter "dtSearch" as the name

(6) Make sure "Execute" permission is set.

(7) Close the Properties dialog box

For more information, see:

Err Msg: Server Failed to Load Application, 80004005

http://support.microsoft.com/default.aspx?scid=kb;en-us;Q200514

Event ID 36 The server process could not be started because the configured identity is incorrect. Check the username and password.

Event ID 10004 DCOM got error "<varies>" and was unable to logon <account name> in order to run the server: <numeric id>

This error occurs when the IWAM_machine identity is "out of sync" in the metabase. A script, Synciwam.vbs, must be run to correct the problem. For more information, see:

PRB: Logon Failure: Unknown User Name or Bad Password When You Run Out-of-Process Webs

http://support.microsoft.com/default.aspx?scid=kb;en-us;255770

If Synciwam.vbs does not work or generates an error, see:

http://www.iisfaq.com/Default.aspx?tabid=3095

Event ID 36 No such interface supported

Solution 1

(1) Click Start > Settings > Control Panel > Administrative Tools > Internet Information Services

(2) Locate the /Scripts/dtSearch folder under the default web site

(3) Right-click the dtSearch folder and select Properties

(4) Change the "Application Protection" setting to "Low"

(5) Restart IIS

If this fixes the problem, it apparently indicates a synchronization problem with an IIS-related password. This article explains how to fix the underlying cause:

http://www.iisfaq.com/default.aspx?View=A463&P=170

Solution 2

If Crystal Reports 8 was installed, see:

http://support.businessobjects.com/library/kbase/articles/c2007581.asp

Additional Resources

Microsoft support site (search for "Event ID NN" where NN is the event ID from the Event Viewer).

http://search.microsoft.com/advanced_search.asp?siteid=us/dev

General information on fixing problems in Internet Information Server

http://www.iisfaq.com

Event ID diagnostic tips, searchable by Event ID code, with links to Microsoft support articles

http://www.eventid.net

Reference information on IIS Event IDs

http://www.iisfaq.com/Default.aspx?tabid=2568


Troubleshooting: 'Encrypted paths' error message from dtSearch Web

dtSearch Home

Downloads

Quick Start

Troubleshooting -- 'Encrypted Paths' error message from dtSearch Web

Last Reviewed: December 17, 2004

Article: DTS0199

Applies to: dtSearch Web 6.4 and later

Symptom

dtSearch Web displays an error message indicating that "encrypted paths" were enabled and so an index could not be accessed to search or display documents.

Cause

dtSearch Web has an option to encrypt index locations on the search form. If this option is enabled, dtSearch Web will refuse to accept unencrypted index paths in a search request. Encrypted paths are enabled in dtSearch Web if (a) the search form has encrypted paths enabled in its dtsearch_options.html file, or (b) no dtsearch_options.html file could be found for the search form.

Encrypted paths are resolved using a table that is stored in the registry when a search form is generated, so if a search form with encrypted paths is created on one server, it will not work if copied to another server.

Resolution

To resolve this problem, generate a new search form on the affected server. If the form is generated without encrypted paths, dtSearch Web will accept index references from the form without encryption.

To fix the problem on an existing form,

(1) Locate the dtsearch_options.html file that is associated with the search form, and
(2) Add this setting to the file:

<BR><HR><I>Encrypt index paths in search form: </I>

<!-- $Begin EncryptIndexPaths -->

1

<!-- $End -->

If the dtsearch_options.html file for the search form is not being found, see this article for troubleshooting steps:

Troubleshooting -- dtSearch Web search results formatting


What web servers does dtSearch Web 5.x work with?

dtSearch Home

Downloads

Quick Start

What web servers does dtSearch Web 5.x work with?

Last Reviewed: August 1, 1999

Article: DTS0133

Applies to: dtSearch Web 5.x

dtSearch Web works with Microsoft Internet Information Server (IIS) 4.0 or later. This is the version of Internet Information Server that is included with the Windows NT Option Pack. If you have Windows 95 or Windows 98, you can still install IIS 4.0 -- it will just appear in menus as "Personal Web Server."

You may be able to use dtSearch Web with earlier versions of IIS, but we recommend upgrading to IIS 4.0 or later for best results.

Installing dtSearch Web on non-Microsoft Web Servers

dtSearch Web is an ISAPI extension. ISAPI is a Microsoft web server interface that some other web servers support to varying degrees. The automated dtSearch Web Setup process in dtSearch Web is designed to work with IIS and may not work with other web servers. However, you may be able to manually install dtSearch Web with these other web servers. To do this,

(1) Put the documents that you want to make searchable into virtual root folders on your web site.

(2) Follow the documentation accompanying your web server to set up support for ISAPI extensions.

(3) Make a virtual folder on your web site named /Scripts/dtSearch. Copy these files from the dtSearch Engine BIN folder to the /Scripts/dtSearch folder: dtengine.dll, dtisapi.dll, english.abc, stemming.dat. (Note: For versions 5.3 and later of the dtSearch Engine, copy dten53u.dll instead of dtengine.dll.)

(4) Use the Form Builder tab in dtSearch Web Setup to create an HTML search form with the controls and indexes that you want to make accessible. You may need to edit the resulting HTML file to fix the link to dtisapi.dll. The "Action" parameter of the <FORM> tag should point to: "http:www.YourServerNameHere.com/Scripts/dtSearch/dtsiapi.dll".

Active Server Pages Interface

In addition to the ISAPI interface, dtSearch Web also has a programmable Active Server Pages (ASP) interface. A sample ASP application built with dtSearch Web is posted at support.dtsearch.com, with source code. Although the ASP interface does not have the easy point-and-click setup of dtSearch Web, it gives you complete control over the presentation of search results and retrieved files.

Hardware Requirements

dtSearch Web does not have any specific hardware requirements. As long as Windows can run efficiently (without a lot of disk swapping) little additional memory is needed for dtSearch.

While a search is executing, dtSearch Web may allocate between 1 and 10 Mb of memory to process that search. Once the search is done (most searches are completed in substantially less than a second), the memory is released. Therefore, unless a site receives thousands of searches per hour, the amount of memory needed for dtSearch Web will be very small.

Recommendations for best performance:

1. At least 256 Mb of memory (so Windows NT or Windows 2000 will run efficiently)

2. Defragmented hard disk maintained using a tool such as Diskeeper

3. At least 25% of the hard disk space unused (so the defragmentation utility will run efficiently -- dtSearch Web does not create any temporary files).


How to generate a synopsis to display in search results

dtSearch Home

Downloads

Quick Start

How to generate a synopsis to display in search results

Last Reviewed: November 10, 2004

Article: DTS0194

Applies to: dtSearch Engine 6.4 and later

SearchReportJob

The SearchReportJob object in the dtSearch Engine API provides a way to extract a few words of context around each hit. For example, see:

http://support.dtsearch.com/Support/demos/forms/SimpleSearch/dtsearch.html

For more information on the SearchReportJob object, see:

h ttp://support.dtsearch.com/webhelp/dtengine/net/doc/dtSearch.Engine.SearchReportJob.html (.NET)

http://support.dtsearch.com/webhelp/dtengine/searchreportjob.htm (VB/COM/ASP)
http://support.dtsearch.com/webhelp/dtengine/javadoc/com/dtsearch/engine /s earchreportjob.html (Java)
http://support.dtsearch.com/webhelp/dtengine/dsearchreportjob.htm (C++)

Generating a Synopsis

Use the following settings in SearchReportJob to generate a brief hits-in-context string to show in search results:

1. Use MaxContextBlocks to limit the number of blocks of context included in the report. For example, if MaxContextBlocks = 1, then only the first hit will be included.

2. Use MaxWordsToRead to generate the context more quickly. For example, if MaxWordsToRead=5000, then the search report will only include hits that occur in the first 5000 words in each document. This setting can be used to generate faster (but not necessarily complete) reports by limiting the search report to hits near the beginning of each document. Because SearchReportJob has to extract text by scanning the original document for each item in search results, it is important to make this process as fast as possible.
Note: If the index was generated using the version 7 format and the dtsIndexCacheText flag, there is no need to use MaxWordsToRead because the index caching will provide efficient access to all hits.

3. Use WordsOfContextExact instead of WordsOfContext -- otherwise SearchReportJob will return whole paragraph-sized blocks.

4. Set the output format to text (it_UTF8), not HTML, so formatting from the original document will not appear in the search results list.

5. Set the dtsReportStoreInResults flag in SearchReportJob, which causes the synopsis to be stored in each search results item, making it easier to access the individual synopsis items.

6. Set the BeforeHit and AfterHit marks to some unusual pair of strings like "{{{" and "}}}". HTML tags like <b> will not work because the synopsis output is assumed to be plain text, not HTML, and so it will have to be HTML-encoded (see below).

7. Select a range of items to include in the search report that corresponds to the range items to be displayed. For example, if you are displaying the first ten items, select items 0 through 9.

Displaying the Synopsis

After the search report has been generated, for each item in search results, there will be a UTF-8 string containing the extracted synopsis with {{{ and }}} around each hit.

To access the synopsis in each item in search results:

C/C++: Use dtsSearchResultsItem.synopsis

.NET: Use SearchResultsItem.Synopsis

VB/ASP: Use SearchResults.DocDetailItem("_synopsis")

Java: Use SearchResults.getDocDetailItem("_synopsis")

To mark hits:

(1) HTML-encode the text (otherwise a < in the original text will result in incorrect HTML in your output), and

(2) Use string operations to replace the {{{ and }}} with "<B>" and "</B>".

For sample code and a demo, see:

http://support.dtsearch.com/support/demos/forms/asp_Synopsis (demo)

http://support.dtsearch.com/support/demos/forms/asp_Synopsis/aspSample2.zip (source code)

Web pages

SearchReportJob cannot access remote web pages, so if a document was indexed using the Spider, SearchReportJob will not be able to generate a hits-in-context display from it.

A workaround for this limitation is to use the dtSearch 7 index format, which is currently available as a beta feature in dtSearch . For more information, see:

http://www.dtsearch.com/Index7.html

The new format lets you cache document text in the index, so the synopsis can be generated quickly from the cached text. This makes it possible to generate a synopsis from any data, and also makes synopsis generation much faster.


How to run dtSearch from a CD or DVD with a web browser interface

dtSearch Home

Downloads

Quick Start

How to run dtSearch from a CD or DVD with a web browser interface

Last Reviewed: November 9, 2004

Article: DTS0165

Applies to: dtSearch 6

dtSearch and the dtSearch Engine are often used to publish documents on a CD, DVD, or other media. This article will describe a way to do this using a web-browser based user interface, so users can access the documents just as they would access a web site. (Note: a dtSearch Text Retrieval Engine distribution license is required before media containing the dtSearch Engine can be distributed to users.)

Some advantages of a browser-based user interface include: it provides high-quality display of HTML files, so web sites will appear just as they do on the original site; it can be customized simply by changing some HTML files; it is easy for customers to use; and no software has to be installed on the user's hard disk to access the content.

If the documents are currently on a web site, you can use a web site download utility to download the HTML and image files into a local directory. Two good products for downloading web sites are:

HTTrack, http://www.httrack.com/ (free)

Offline Explorer Pro, www.metaproducts.com ($50)

In addition to downloading the files, these products will also fix the links in the HTML files so they will work when the files are opened locally.

Once you have the documents, you can use the "CD Wizard" included with the dtSearch Text Retrieval Engine that automates the process of setting up a master folder. For more details on what the CD Wizard does, click the Help button in the CD Wizard and see the "Implementation Details" topic. The CD Wizard helps you to create one or more CD master folders. Each folder contains a set of documents, software, and dtSearch indexes that is ready to be transferred to a CD or DVD. You can create any number of master folders, and each folder can contain any number of document folders and indexes.

To create a CD master folder,

1. Install the dtSearch Publish program files on your computer.
If you have dtSearch Web on CD, run the setup program to install. If you downloaded dtSearch Web from the internet, follow the download instructions to open the dtSearch Web archive and install the files.

2. Start the dtSearch CD Wizard
In dtSearch, click "dtSearch CD Wizard..." in the File menu.

3. Make a new CD master folder
Click New... to make a new CD master folder. Enter the location where you want the CD master folder to go and the title, copyright notice, and technical support contact for the CD. This information will be displayed in a dialog box that will pop up when the CD is inserted. To change this information later, just click Edit... and enter your changes. When you click OK, dtSearch will set up the CD master folder with your settings and the software that will run from the CD.

4. Add documents to the CD
Click Add Documents... to add documents to the folder. The Add Documents dialog box will appear. When the CD master folder is set up, a root\data folder will be created where the documents should be stored. To add documents to the CD, you can copy them into this folder using Windows Explorer, or you can use the Add Folder... button to have the CD Wizard do this automatically. When you click OK after selecting a folder to add, all of the documents in the folder will be copied into the CD master folder.

5. Create an index for the documents
Click Create Index... to create an index for the documents. You can create any number of indexes on each CD (just click Create Index... for each one). The process of creating and updating an index works exactly as it does in dtSearch Desktop. (Note: When you click Add Folder to add document folders to your index, select only folders that are under the CD master folder you are creating. If you try to select other folders, the "OK" button will be disabled in the Add Folder dialog box.)

6. Build a search form
Click the Build Search Form... button to build a search form to use with your site. You can make as many search forms as you want for each site, and after dtSearch Web Setup has generated a search form, you can use an HTML editor such as Microsoft FrontPage to edit it to fit into your web site.

After the search form is built, dtSearch Web will open it in your browser so you can try out a search. Once you have a basic search form working, you click Build Search Form again to customize the search form, the appearance of search results, and other options, and to create addtional search forms.

7. Make a "home" page for the CD
The home page is the first page that users will see when they insert the CD. The home page is named index.html and is located in the root\data subfolder of the CD master folder.

Compatibility

There are two compatibility requirements for the CDs that the dtSearch CD Wizard generates:

(1) The hit navigation buttons (Next Hit, Next Doc, etc.) are implemented using JavaScript, so users will need to have a web browser that supports JavaScript (Internet Explorer 4 or later, or Netscape 4 or later).

(2) The web server that runs from the CD uses WinSock2. Windows 95 did not come with WinSock2, but installing a recent browser will add WinSock2.

A CD Wizard-generated CD will check that requirements (1) and (2) are satisfied before starting the web server, and will advise the user if the machine needs a browser upgrade to access the CD. Software dependency checking can be customized easily to install any additional plug-ins or other components that your CD requires.

Why is a server needed?

The dtSearch CD Wizard puts an HTTP server on the CD, set up to work with dtSearch Web. A server is needed to use HTML forms from a CD because of the way HTML forms work. When you click on the Search button in an HTML search form. the web browser collects the search request and other information from the form and sends it to the web server. The server then sends back the search results as a stream of HTML, and the browser displays this.

An alternative approach that is sometimes taken is to put a link to a search program on the CD, but most browsers will display a warning message when the link is clicked. Another approach is to write a program that embeds the web browser control. The vbsearch sample application included with the dtSearch Engine is an example. The web browser control provides high-quality display of HTML files, and embedding it in a program lets you integrate searching using standard Windows dialog box. However, embedding the web browser control means that CD users will have to have a recent version of Internet Explorer installed, which may be an unacceptable requirement depending on the target audience for the CD. Also, if the program is written in Visual Basic, then the dtSearch Engine will have to be registered and the Visual Basic runtime files installed on the user's computer. This may also be an unacceptable requirement for some CD publishers.

Can the web server run on my network as well as from the CD?

dtSearch Publish can run from a CD or DVD, a hard disk folder or a shared network folder. The HTTP server that comes with dtSearch Publish does not allow HTTP access from anywhere other than "localhost" (127.0.0.1), so it cannot be executed as a true multi-user server, but it can be called from a desktop machine that accesses the CD content from a shared network folder.

However, using the standard dtSearch Web installation on Internet Information Server (IIS) is easier to set up and also more efficient. dtSearch Web Setup can install dtSearch Web on an IIS server, and generate a search form, with a few mouse clicks. (See the dtSearch Web Quick Start for more information.) An IIS-based installation will also provide much better searching performance because IIS can keep dtSearch Web loaded in memory between requests.

If you have created customized search forms for your CD, you can use those forms with dtSearch Web as well. To adapt the search forms for use with dtSearch Web, change the references to dtcgi2is.exe to point to dtisapi6.dll.


Automation errors accessing dtSearch Engine methods or properties

dtSearch Home

Downloads

Quick Start

Automation errors accessing dtSearch Engine methods or properties

Last Reviewed: February 26, 1999

Article: DTS0125

Applies to: dtSearch Engine 5.x

Symptoms

- After upgrading to a new version of the dtSearch Engine, you get Automation errors trying to access certain properties and methods.

- Automation errors occur after attempting to access new methods or properties of the dtSearch Engine.

Possible Causes

- Visual Basic is using an out-of-date type library to access the dtSearch Engine.

- The wrong version of the dtSearch Engine is registered.

- Method or property is incorrectly spelled in the Visual Basic or script code.

Troubleshooting Steps

- Use the Object Browser in Visual Basic to check whether the method or property that is triggering the error is listed. If the method or property is not listed, an out of date type library is being used. You can also use Object Browser to check that the method or property is spelled correctly in the calling code.

- Click Project|References and verify that the most recently-installed version of dtengine.dll is the one shown as the reference for the dtSearch Text Retrieval Engine

Resolution

To resolve type library version problems:

(1) Register the dtSearch Engine. Open a DOS window, go to the folder where the current dtSearch Engine version is located, and execute the command:

regsvr32 dtengine.dll

(2) Open your Visual Basic project.

(3) Click Project|References and remove the reference to the dtSearch Engine.

(4) Close the project

(5) Open the project

(6) Click Project|References and click on the item for the dtSearch Engine. Make sure that the path and filename match the dtengine.dll that you just registered. Check the box if they match, or click Browse to find the right DLL if they do not.

(7) Use Object Browser in Visual Basic to check that the methods and properties you are using are present in the selected type library.

(8) Once you have the program running in the VB debugger, build the .EXE again so it will also know about the new type library.


CreateObject errors creating dtSearchEngine.Server

dtSearch Home

Downloads

Quick Start

CreateObject errors creating dtSearchEngine.Server

Last Reviewed: July 1, 2003

Article: DTS0124

Applies to: dtSearch Engine 6

Symptoms

- CreateObject("dtSearchEngine.Server") fails.

- Error "424 Object Required" in an ASP script.

- ASP Error 8002801d "Library Not Registered"

- ASP Error 'ASP 0223' "TypeLib Not Found"

- ASP Error "METADATA tag contains a Type Library specification that does not match any Registry entry"

Possible Causes

- dtSearch Engine not registered

- Wrong object name used

- Registry security blocking access to type library information

- Crystal Reports 8 installation

Troubleshooting Steps

1. Check that the dtSearch Engine is registered

To check that the dtSearch Engine is registered, run the dtdemo.exe program in your dtSearch BIN folder (usually c:\program files\dtSearch\bin\dtdemo.exe). If dtdemo.exe does not run, the dtSearch Engine is not registered. To register the dtSearch Engine, execute the following command, replacing <path> with the full path to dtengine.dll:

regsvr32 "<path>\dten600.dll"

After this is done, the dtdemo.exe program should work.

2. Run the dttest.asp diagnostic script

The dttest.asp diagnostic script is included with the dtSearch Engine in <dtsearch>\examples\asp\dttest.asp. To use the script, copy it into a folder on your web server that has "Execute (including script)" permission enabled, and then navigate to dttest.asp in a browser. The dttest.asp script will tell you if the dtSearch Engine could be accessed from ASP.

If you see the error "424 Object Required" when the the dtSearch Engine is accessed, or ASP Error 8002801d "Library Not Registered", it usually means either (a) the dtSearch Engine is not registered (see step 1 above), or (b) registry security is blocking access to the type library entry for dtSearch. If you are able to run the dtdemo.exe program in step 1 above, then registry security is probably the problem. Please see this article on the Microsoft web site for instructions to fix the registry security problem:

PRB: ASP Error 8002801d "Library Not Registered" (Q274038)

http://support.microsoft.com/default.aspx?scid=kb;EN-US;q274038

If you see a 500 error, check the Event Viewer (click Start > Control Panel > Administrative Tools > Event Viewer) and check the System log to see the most recent error. For information on interpreting Event Viewer entries, see Troubleshooting -- dtSearch Web server error codes.

An 80070005 error means that access was denied to the dtSearch Engine DLL. To fix the error,

(1) Open Windows Explorer

(2) Locate dten600.dll, which is usually installed in C:\Program Files\dtSearch Developer\bin

(3) Right-click dten600.dll and select "Properties"

(4) Add "Read" and "Execute" access for "Everyone".

3. Check the CreateObject command you are using

The command is different for each version of dtSearch, as follows:

Version 5.2: CreateObject("dtSearchEngine.Server")

Version 5.3: CreateObject("dtSearchEngine53.Server")

Version 6.0: CreateObject("dtSearchEngine6.Server")

If you are able to get dttest.asp to work (see step 2 above), you can cut and paste the CreateObject command from dttest.asp into your script to ensure that it is correct.

4. Was Crystal Reports 8 recently installed?

Crystal Reports 8 changes a registry permission setting that can cause type library problems with other applications. Please see this article on the Microsoft web site for more information:

PRB: Error "Library Not Registered" with Crystal Reports 8.0 (Q266621)

http://support.microsoft.com/default.aspx?scid=kb;en-us;Q266621


Diagnostic tools for .NET developers

dtSearch Home

Downloads

Quick Start

Diagnostic Tools for .NET Developers

Last Reviewed: September 1, 2004

Article: DTS0196

Applies to: dtSearch Engine 6.4 and later

The dtSearch Text Retrieval Engine includes several diagnostic tools to help isolate problems that arise during execution of a program. These include: (1) detailed error reporting through the JobErrorInfo object, (2) debug logging to a text file, and (3) generation of a stack trace when a program crashes.

Error Reporting

Most of the dtSearch Engine's API functions involve a "Job" structure of some type (a SearchJob, an IndexJob, etc.). After a Job has been executed, the Errors property of the job will contain a JobErrorInfo that provides the text and error codes of each error that occurred. Documentation on the meaning of the error codes appears in the "Error Codes" topic in the dtSearch Engine help file.

Debug Logging

dtSearch Engine can generate a debug log that provides detailed information about internal processing of an API call. To enable debug logging in the dtSearch Engine, call the SetDebugLogging method of the Server object.

SetDebugLogging(LogName as String, flags as Long)

SetDebugLogging tells the dtSearch Engine to start recording a debug log in the specified file. The flags argument controls optional logging behavior. To turn off debug logging, call SetDebugLogging with a blank LogName.

Note: Debug logging will not work if the process creating the log lacks write access to the folder containing the log file. For example, an ASP application running inside IIS may not be permitted to write to some folders. If SetDebugLogging does not seem to work,

(1) Create an empty folder with a simple name like c:\log

(2) Right-click the c:\log folder in Windows Explorer and select "Permissions"

(3) Set the permissions for c:\log to grant "Everyone" "Full Control" access

(4) Use c:\log\debuglog.txt as the name of the log file

Flag | Effect
dtsLogTime | Each line in the log will have include the time, in hundredths of a second, since the start of execution.
dtsLogCommit | The log will be committed to disk after each line is written. This slows execution considerably but ensures that the log will survive a system crash.
dtsLogAppend | Log data is appended to the file if it already exists. Otherwise the file will be overwritten.

Crash Diagnostics

The dtSearch Engine can set up a crash handler so that if a General Protection Fault or other unhandled exception occurs during execution of the program, a stack trace file will be created.

To set up a crash handler, call SetDebugLogging (as above) with the dtsCrashLog flag, like this:

Server.SetDebugLogging "c:\error.txt", dtsCrashLog

After a crash occurs, the text file will contain a stack trace file identifying the state of the calling program and the dtSearch Engine when the crash occurred.


Diagnostic tools for C++ developers

dtSearch Home

Downloads

Quick Start

Diagnostic tools for C++ developers

Last Reviewed: December 1, 2000

Article: DTS0145

Applies to: dtSearch Engine 6

The dtSearch Text Retrieval Engine includes several diagnostic tools to help isolate problems that arise during execution of a program. These include: (1) detailed error reporting through the dtsErrorInfo structure, (2) debug logging to a text file, and (3) generation of a stack trace when a program crashes.

Error Reporting

Most of the dtSearch Engine's API functions involve a "Job" structure of some type (a dtsSearchJob, a dtsIndexJob, etc.). The Job structures all include an optional pointer to a dtsErrorInfo structure in the errorHandler member. After the job is done, the dtsErrorInfo will contain one or more strings and error codes identifying any errors that occurred. A default dtsErrorInfo will just store these strings and codes as they are added, but a callback function can also be added to the dtsErrorInfo to provide immediate notification. Documentation on the meaning of the error codes appears in the "Error Codes" topic in the dtSearch Engine help file (dtengine.chm).

A simple example demonstrating use of a dtsErrorInfo:

dtsSearchJob searchJob;

dtsErrorInfo errorInfo;

searchJob.errorHandler = &errorInfo;

// ... set up searchJob ...

dtssDoSearchJob(searchJob, result);

// errorInfo will have a list of any errors that occurred

If you are using the C++ Support Classes included with the dtSearch Engine, an error handler is automatically attached to every job and is accessible through the job's GetErrors() method. A FormatErrors() method will generate a string containing the text of all errors that occurred during a job.

Debug Logging

dtSearch Engine can generate a debug log that provides detailed information about internal processing of an API call. To enable debug logging in the dtSearch Engine, call the dtssDebugLogEx() function. The prototype for dtssDebugLogEx is:

void dtssDebugLogEx(char *filename, long flags)

dtssDebugLogEx tells the dtSearch Engine to start recording a debug log in the specified file. The flags argument controls optional logging behavior.

Flag | Effect
dtsLogTime | Each line in the log will have include the time, in hundredths of a second, since the start of execution.
dtsLogCommit | The log will be committed to disk after each line is written. This slows execution considerably but ensures that the log will survive a system crash.
dtsLogAppend | Log data is appended to the file if it already exists. Otherwise the file will be overwritten.

Note: Debug logging will not work if the process creating the log lacks write access to the folder containing the log file. For example, an ASP application running inside IIS may not be permitted to write to some folders. If debug logging does not seem to work,

(1) Create an empty folder with a simple name like c:\log

(2) Right-click the c:\log folder in Windows Explorer and select "Permissions"

(3) Set the permissions for c:\log to grant "Everyone" "Full Control" access

(4) Use c:\log\debuglog.txt as the name of the log file

Crash Diagnostics

The dtSearch Engine can set up a crash handler so that if a General Protection Fault or other unhandled exception occurs during execution of the program, a stack trace file will be created. The functions to install (and remove) the crash handler are:

void dtssInstallCrashDiagnostics(dtsCrashHandlerInfo& info)

void dtssRemoveCrashDiagnostics()

struct dtsCrashHandlerInfo {

long structSize;

void *pData;

void (*pNotifyFn)(void *pData, const char *logName);

char appName[40];

char appVersion[40];

char logName[256];

long reserved[5];

dtsCrashHandlerInfo();

void copy(const dtsCrashHandlerInfo& other);

};

pData

An optional value to be passed to the pNotifyFn.

pNotifyFn

An optional callback function to be called when a crash occurs.

appName

appVersion

Information identifying the application, which will be written in the header of the log file.

logName

The name of the log file to create with the stack trace.

The stack trace file will list each item on the stack as an address. A utility included with the dtSearch Engine, StackTrace.exe, can be used to convert these addresses to symbol names. To use StackTrace.exe, you will need a .map file for your program. For example, if your program is MySearch.exe, you will need a map file named MySearch.map. The Visual C++ linker does not generate map files by default so you will need to enable this option to get a map file. Also, the map file is only valid for the particular build for which it was generated. If you modify and rebuild the program, a new map file will be needed. Once you have a map file and the text file created by dtssInstallCrashDiagnostics, you can get the symbols as follows:

StackTrace MySearch.map Error.txt > ErrorWithSymbols.txt

If your program includes multiple modules, you can list the map files for each module before the error file, like this:

StackTrace MySearch.map MyAdditionalDll.map Error.txt > ErrorWithSymbols.txt


Diagnostic Tools for Java Developers

dtSearch Home

Downloads

Quick Start

Diagnostic Tools for Java Developers

Last Reviewed: December 1, 2000

Article: DTS0147

Applies to: dtSearch Engine 6

The dtSearch Text Retrieval Engine includes several diagnostic tools to help isolate problems that arise during execution of a program. These include: (1) detailed error reporting through the dtsErrorInfo structure, (2) debug logging to a text file, and (3) generation of a stack trace when a program crashes.

Error Reporting

Most of the dtSearch Engine's API functions involve a "Job" structure of some type (a SearchJob, an IndexJob, etc.). After a Job has been executed, the Errors property of the job will contain a JobErrorInfo that provides the text and error codes of each error that occurred. Documentation on the meaning of the error codes appears in the "Error Codes" topic in the dtSearch Engine help file (dtengine.chm).

Debug Logging

dtSearch Engine can generate a debug log that provides detailed information about internal processing of an API call. To enable debug logging in the dtSearch Engine, call the SetDebugLogging method of the Server object.

setDebugLogging(LogName as String, flags as Long)

SetDebugLogging tells the dtSearch Engine to start recording a debug log in the specified file. The flags argument controls optional logging behavior. To turn off debug logging, call SetDebugLogging with a blank LogName.

Flag | Effect
dtsLogTime | Each line in the log will have include the time, in hundredths of a second, since the start of execution.
dtsLogCommit | The log will be committed to disk after each line is written. This slows execution considerably but ensures that the log will survive a system crash.
dtsLogAppend | Log data is appended to the file if it already exists. Otherwise the file will be overwritten.

Note: Debug logging will not work if the process creating the log lacks write access to the folder containing the log file. For example, a web application running as the logged-on user may not be permitted to write to some folders. If debug logging does not seem to work,

(1) Create an empty folder with a simple name like c:\log

(2) Right-click the c:\log folder in Windows Explorer and select "Permissions"

(3) Set the permissions for c:\log to grant "Everyone" "Full Control" access

(4) Use c:\log\debuglog.txt as the name of the log file

Crash Diagnostics

The dtSearch Engine can set up a crash handler so that if a General Protection Fault or other unhandled exception occurs during execution of the program, a stack trace file will be created.

To set up a crash handler, call SetDebugLogging (as above) with the dtsCrashLog flag. After a crash occurs, the text file will contain a stack trace file identifying the state of the calling program and the dtSearch Engine when the crash occurred. Crash diagnostics can be combined with a stack trace as long as the filenames provided are different for each. Example:

server.setDebugLogging("c:\\dtsjava.log", Constants.dtsLogCommit);

server.setDebugLogging("c:\\dterror.txt", Constants.dtsCrashLog);


Diagnostic Tools for Visual Basic and ASP Developers

dtSearch Home

Downloads

Quick Start

Diagnostic Tools for Visual Basic Developers

Last Reviewed: December 1, 2000

Article: DTS0146

Applies to: dtSearch Engine 6

The dtSearch Text Retrieval Engine includes several diagnostic tools to help isolate problems that arise during execution of a program. These include: (1) detailed error reporting through the dtsErrorInfo structure, (2) debug logging to a text file, and (3) generation of a stack trace when a program crashes.

Error Reporting

Most of the dtSearch Engine's API functions involve a "Job" structure of some type (a SearchJob, an IndexJob, etc.). After a Job has been executed, the Errors property of the job will contain a JobErrorInfo that provides the text and error codes of each error that occurred. Documentation on the meaning of the error codes appears in the "Error Codes" topic in the dtSearch Engine help file (dtengine.chm).

Debug Logging

dtSearch Engine can generate a debug log that provides detailed information about internal processing of an API call. To enable debug logging in the dtSearch Engine, call the SetDebugLogging method of the Server object.

SetDebugLogging(LogName as String, flags as Long)

SetDebugLogging tells the dtSearch Engine to start recording a debug log in the specified file. The flags argument controls optional logging behavior. To turn off debug logging, call SetDebugLogging with a blank LogName.

Note: Debug logging will not work if the process creating the log lacks write access to the folder containing the log file. For example, an ASP application running inside IIS may not be permitted to write to some folders. If SetDebugLogging does not seem to work,

(1) Create an empty folder with a simple name like c:\log

(2) Right-click the c:\log folder in Windows Explorer and select "Permissions"

(3) Set the permissions for c:\log to grant "Everyone" "Full Control" access

(4) Use c:\log\debuglog.txt as the name of the log file

Flag | Effect
dtsLogTime | Each line in the log will have include the time, in hundredths of a second, since the start of execution.
dtsLogCommit | The log will be committed to disk after each line is written. This slows execution considerably but ensures that the log will survive a system crash.
dtsLogAppend | Log data is appended to the file if it already exists. Otherwise the file will be overwritten.

Crash Diagnostics

The dtSearch Engine can set up a crash handler so that if a General Protection Fault or other unhandled exception occurs during execution of the program, a stack trace file will be created.

To set up a crash handler, call SetDebugLogging (as above) with the dtsCrashLog flag, like this:

server.SetDebugLogging "c:\error.txt", dtsCrashLog

After a crash occurs, the text file will contain a stack trace file identifying the state of the calling program and the dtSearch Engine when the crash occurred.

Object Leaks

The dtSearch Engine keeps a table of the number of each type of object (SearchJob, IndexJob, SearchResults, etc.) currently allocated. This table can be used to detect leaks due to failure to release an object. For example, an application that does not release SearchResults objects after each search will accumulate SearchResults objects in memory over time, resulting in a steady increase in the memory consumed by the application.

To check the table, examine the Server.AllocatedObjectList property. This property is a simple string with the names of each object type followed by the number of currently-allocated instances of each.

Visual Basic and VBScript are supposed to automatically release objects when they go out of scope, but in practice this sometimes does not happen. To minimize the risk of this type of leakage, set any object reference to Nothing when you are done with it, like this:

Dim res as dtengine.SearchResults

Dim sj as dtengine.SearchJob

Set sj = Server.NewSearchJob

... set up search job and execute it ...

Set res = sj.Results

... display search results ...

Set res = Nothing

Set sj = Nothing


dtSearch Engine Build Tips for Visual C++ Programmers

dtSearch Home

Downloads

Quick Start

dtSearch Engine build tips for Visual C++ developers

Last Reviewed: December 1, 2000

Article: DTS0105

Applies to: dtSearch Engine 5.x, dtSearch Engine 6

This article describes some common problems that Visual C++ developers encounter when building an application with the dtSearch Engine.

Compile Errors

Precompiled header errors when compiling

By default, the Visual C++ project wizard sets up a project to use stdafx.h as the precompiled header file for every file in a project. If a file does not contain "#include <stdafx.h>", this message is generated. To fix the problem, change the build settings for dtsearch.cpp, dtsviewr.cpp, and any other file that does not #include stdafx.h to "Do not use precompiled headers". This setting is located in Settings|C++ Compile|Precompiled Headers.

Compile errors in debug builds relating to DEBUG_NEW

Sometimes the Visual C++ precompiled headers database gets confused and generates lots of false errors during a debug compile relating to new, debug_new, etc. A Build|Clean will clear these up, but they may come back periodically.

Link Errors

Undefined symbols "_dtssDoInit", "_dtssDoIndexJob", etc.

The dtSearch Engine import library is not linked in. Add dtengine.lib (for version 5.2 and earlier) or dts600.dll (for version 6) to your project.

Undefined symbols "dtsInitInfo::dtsInitInfo", "dtsIndexJob::dtsIndexJob", etc.

Add <dtsearch>\examples\cpp\common\dtsearch.cpp and dtsviewr.cpp to your project. The dtSearch Engine API uses interface structures with constructors to make it easier to set them up correctly. For example, the dtsInitInfo structure has a constructor, dtsInitInfo::dtsInitInfo, which is defined in dtsearch.cpp.

Error LINK1106: "invalid file or disk full"

Visual C++ 5.0 Service Pack 3 or Visual C++ 6 is needed to link with dtengine.lib. Earlier versions of Visual C++ produce this error message. Obtain Service Pack 3 from the Microsoft Visual C++ web site, or upgrade to Visual C++ 6.

Multiple definitions of symbols new, delete, main, dllmain, etc.

Visual C++ sometimes links the MFC library nafx*.lib and the runtime library libc*.lib in the wrong order -- often not the first time you build, but later on. The solution, as detailed in Microsoft Support Article Q148652 , is to force the correct link order using a combination of "Ignore default libraries" and "Additional libraries".


How to add fields to documents during indexing

dtSearch Home

Downloads

Quick Start

How to add fields to documents during indexing

Last Reviewed: August 25, 2004

Article: DTS0150

Applies to: dtSearch Engine 6

When documents are indexed, sometimes it is useful to be able to add one or more custom fields to the index along with each document, without changing the original file. For example, a document management program may have a series of document attributes (Author, Subject, Department, etc.) that must be associated with each file checked into the document manager.

One way to add field data is to use the field capabilities of the document itself. For example, Word and Excel documents can have one or more "Summary Information" fields. Similarly, HTML files can have META tags, WordPerfect files can have Document Summary fields, and PDF files support several predefined fields. dtSearch can recognize these fields and will automatically index them along with the document text.

Alternatively, the field information can be supplied to the dtSearch Engine during indexing using one of the "data source" APIs.

VB.NET and C#

In the dtSearch Engine's .NET API (in versions 6.4 and later) you can use the DataSourceToIndex property of an IndexJob to specify an object that will supply documents to be indexed. For each document, your DataSourceToIndex would supply a DocName, DocText (plain text), DocFields (any number of tab-delimited field-value pairs), a DocModifiedDate and a DocCreatedDate. In DocBytes, you can provide a byte array with the binary contents of the document, which dtSearch will parse just as it would parse a file on disk.

For sample code demonstrating how to index meta-data in a database along with document files, see this sample application, included with the dtSearch Engine:

C:\Program Files\dtSearch Developer\examples\cs\ado_demo

Visual Basic

In Visual Basic, you can use the DataSourceToIndex property of an IndexJob to specify an object that will supply documents to be indexed. For each document, your DataSourceToIndex would supply a DocName, DocText (plain text), DocFields (any number of tab-delimited field-value pairs), a DocModifiedDate and a DocCreatedDate. Additionally, if you set the DocIsFile flag to true, dtSearch will interpret the DocName as a filename and will index the contents of the file along with the text in DocText and the fields in DocFields.

For more information, see the "Indexing ActiveX Data Sources" topic in the dtSearch Engine help file.

For sample code demonstrating how to combine binary documents with other fields during indexing, see the dsdemo sample, which is installed to:

C:\Program Files\dtSearch Developer\examples\vbasic\dsdemo.

C++

In C++, you can use the dataSourceToIndex member of the dtsIndexJob structure to specify an object that will supply documents to be indexed. For each document, your data source would supply seek() and read() callback functions providing access to the binary data in the document, along with any number of fields in a null-delimited string set.

For more information, see these topics in the dtSearch Engine help file: "dtsDataSource," "dtsInputStream".

Java

The Java API to the dtSearch Engine Java supports a dataSourceToIndex property that has the same interface as the Visual Basic interface, except that the property names are prefaced with "get":

public interface DataSource {

public boolean getNextDoc();

public boolean rewind();

public String getDocText();

public String getDocFields();

public String getDocName();

public String getDocDisplayName();

public java.util.Calendar getDocModifiedDate();

public java.util.Calendar getDocCreatedDate();

public boolean getDocIsFile();

};

An extended version of the API also supports returning BLOB data in a memory buffer (an array of byte[]). For more information, see the DataSource2 interface documentation. For sample code, see: http://support.dtsearch.com/ftp_pub/FolderDataSource.zip

Additional Information

Some or all of the fields added through the data source APIs can be designed as "stored" fields. A stored field is a field that becomes part of the document properties that are returned in search results. For example, a document manager may add a stored row ID field to each file that is indexed so that when a file is returned in search results, the row ID field will be easily accessible.

For more information on stored fields, see "How to get field data in search results."

Related Topics

"How to index databases with the dtSearch Engine"

"Database and Field Searching" in the dtSearch Engine help file.


How to display retrieved XML documents with XSL formatting

dtSearch Home

Downloads

Quick Start

How to display retrieved XML documents with XSL formatting

Last Reviewed: June 18, 2003

Article: DTS0183

Applies to: dtSearch Web 6

The default format for display of XML documents looks like this:

<Record> This is the text of the "Record" field

<Item> This is the text of the "Item" field inside "Record"

...

To display XML with XSL styles, there are three options: (1) display the XML without hit-highlighting, so that XSL formatting will be applied automatically in the browser, (2) index the content in HTML format after the XSL formatting has been applied, or (3) add hit highlight tags to the XML.

Note: For information on searching fields in XML files, see: http://support.dtsearch.com/webhelp/dtsearch/field_searching.htm.

1. Displaying XML without highlighted hits

When XML is displayed without highlighted hits, the browser will automatically apply any style sheet that is specified in the XML file.

To display XML without highlighted hits in dtSearch Web, edit the dtSearch_options.html file to add XML to the list of "Unconverted Types", like this:

<!-- $Begin UnconvertedTypes -->

PDF XML

<!-- $End -->

With this change, dtSearch Web search results will link directly to XML files rather than highlighting hits, so XSL styles will be displayed.

To display XML without highlighted hits in dtSearch Desktop,

(1) In dtSearch Desktop, click Options > Preferences > External Viewers

(2) Click New... and set up a new item named "XML", with "*.XML" as the filename filter, and check the option to "Display file in dtSearch without hits highlighted."

2. Displaying XML as HTML with highlighted hits

To highlight hits in XML with style sheet formatting, the XML documents can be indexed using the dtSearch Spider, with the server configured to apply the XML style sheets. This way the dtSearch Spider will see the HTML output from the XSL styles, rather than the original XML. After a search, the URL that generates the HTML output will be retrieved, so dtSearch will be able to highlight hits in the generated HTML.

For more information on indexing dynamically-generated content using the dtSearch Spider, see:

How to use dtSearch Web with dynamically-generated content

3. Highlighting hits in XML

dtSearch (version 6.21 and later) can also highlight hits in XML without conversion (like the HTML-to-HTML hit highlighting feature for HTML files). This provides pure XML output with caller-specified highlight markings around each hit. Example:

<Record> This is the text of the "Record" field.

<Item> This is the text of the "Item" field inside "Record". This is a <hit>highlighted hit</hit>.</Item>

</Record>

The advantage of obtaining pure XML is that you can use XSL to format the content for display. The BeforeHit and AfterHit markings can either be an XML field tag or a unique mark (like {{{ and }}}) that will be replaced with HTML formatting instructions after XSL processing.

Requirements:

(1) dtSearch version 6.21 or later is required.

(2) The XML must have been indexed with XML field names and attributes skipped.

(3) XML-to-XML conversion must be enabled when the documents are displayed.

Indexing without field names or attributes

XML-to-XML hit highlighting requires that the XML be indexed with field names and attributes made non-searchable. This is because the highlighting algorithm assumes that only field data will be highlighted. If these flags were not set when the XML was indexed, the wrong words will be highlighted because the word counts will not match the way the document was indexed.

dtSearch Desktop

Click Options > Preferences > Indexing Options, and un-check these two boxes:

"Index field names in XML files"

"Index field attributes in XML files"

Developer API

Before indexing documents, set these flags in Options.FieldFlags:

dtsoFfXmlHideFieldNames and

dtsoFfXmlSkipAttributes

Enabling XML-to-XML conversion

dtSearch Web or dtSearch Publish

To enable XML-to-XML conversion for retrieved XML documents, change this setting in the dtsearch_options.html file:

<!-- $Begin EnableXmlToXmlHighlighting -->
1
<!-- $End -->

Developer API

To set up a FileConverter or DFileConvertJob for XML-to-XML highlighting,

(1) Set the flags to dtsConvertXmlToXml, and

(2) Set the output format to it_XML (the it_XML output format cannot be used with any input data other than XML).

(3) Set the BeforeHit and AfterHit values to XML tags to be used around hits, like "<hit>" and "</hit>".

dtSearch Desktop

XML-to-XML hit-highlighting is not possible in dtSearch Desktop.

Notes

1. XML style sheet references must be absolute to work after highlighting. When a document has been hit-highlighted, it no longer has a location to serve as a base for relative references, so if your XML file references "sample.xsl", the styles will not be found. Instead, insert an absolute reference like "/folder/sample.xsl".

2. The style sheet must define a style for the tag to be used to highlight hits. dtSearch Web uses <dtSearchHit> and </dtSearchHit> around hits in XML.

3. Hit navigation (the "Next Hit" and "Prev Hit" buttons) is not currently available for data that is displayed as XML rather than HTML.

4. Client-side formatting of XML using XSL requires Internet Explorer 6 or Mozilla.

For an example of XML-to-XML highlighting, click here. To view the documents as XML, right-click in the document window and select "View Source".

See also:

How to combine field and attribute searches in XML

Field searching


How to index databases with the dtSearch Engine

dtSearch Home

Downloads

Quick Start

How to index databases with the dtSearch Engine

Last Reviewed: November 29, 2004

Article: DTS0111

Applies to: dtSearch Engine 6.x

1. Indexing SQL Data and other data sources

This dtSearch Engine has an API that can index any data you can access from .NET, Visual Basic, Java, or C++. A sample database indexing application included with the dtSearch Engine uses Active Data Objects (ADO.NET) to index any database that can be accessed through ADO. The same approach will work with other types of non-file data.Source code for the database indexing sample application is installed to these locations:

C:\Program Files\dtSearch Developer\examples\vb.net\ado_demo (VB.NET)

C:\Program Files\dtSearch Developer\examples\cs\ado.net (C#)

C:\Program Files\dtSearch Developer\examples\vbasic\ado_demo (Visual Basic 6)

The API's used to implement this feature are documented here:

.NET: DataSource class, documented in dtSearchNetApi.chm

C/C++: dtsDataSource in the dtSearch Engine help file.

COM: Indexing ActiveX Data Sources in the dtSearch Engine help file.

Java: DataSource2 and DataSource classes

The ado_demo sample application uses the database schema to determine the tables and fields that are present in the database, and then iterates over every row of every table, indexing the entire database. The source code can be modified to exclude certain tables or fields from the index.

The basic mechanism used to implement data source indexing is the same in all of the APIs:

(1) A data source object is created that iterates over the content to be indexed. The object implements a "getNextDoc" method that returns the next document to be indexed.

(2) This object is attached to the IndexJob, and when the IndexJob is executed, the dtSearch Engine calls the getNextDoc method repeatedly to get data to be added to the index.

(3) For each item returned by getNextDoc, the data source can return plain text, fielded data, and/or a BLOB containing binary document data (such as a PDF file).

Incremental Updates

An index of a database that was created using the data source API can be updated incrementally. To do this,

(1) Set up the IndexJob exactly as it was set up when the index was initially created, but set all of the Action flags (ActionCreate, ActionRemoveDeleted, etc.) to false), except the ActionAdd flag, which must be set to True. If ActionCreate is true, the index will be cleared at the start of the index update.

(2) Set up your data source so it will only iterate over database rows that are either (a) new, or (b) have been modified since they were last indexed. If the data source iterates over the whole database, dtSearch will still only index new or modified rows (it checks the modification date and size of each item to see if it has changed before reindexing). However, incremental updates will be much faster if you just pass dtSearch the rows that need to be indexed.

Binary documents in fields (BLOB data)

If data in a column consists of binary files such as Word documents or Excel spreadsheets, you can index the contents of these columns along with other fields. The data source API provides a way to combine (field, value) pairs from other columns of the same row with a binary document such as a Word file into a single logical document for indexing and searching purposes. dtSearch will index the document just as it would if found on disk, including any fields in the document itself (such as Document Summary Information fields). For more information on how this is done, see the article, "How to add fields to documents during indexing."

Selecting which tables and columns to index

The ado_demo samples all implement a SampleDataSource class that traverses the entire database, using schema information to get the list of tables and columns in the database. To modify the sample code so it indexes only certain tables or only certain fields, modify the SampleDataSource class, changing the code that gets the list of tables and columns from the schema to instead use a specific list of tables and columns.

Third-party database-indexing tools

Cybergroup's dbConnector is available from Cybergroup and can be used to index any ODBC-compliant database using a GUI. For more information, see:

http://www.dtsearch.com/CS_CybergroupdbConnec.html

http://www.cybergroup.com/

2. Indexing desktop databases with the dtSearch Engine and ODBC

dtSearch indexes, searches and displays Access databases using Microsoft ODBC drivers. If the ODBC drivers are installed, dtSearch will index a database file as a series of records. The name of each record includes the filename of the database, the name of the table, and a record identifier. If instead dtSearch indexes a database as a single text file, you probably do not have the ODBC drivers installed.

Database indexing works best when the ODBC drivers are able to use a primary key to access records in the database. A primary key is a field that is unique for each record and that is designated as a primary key in the database.

ODBC indexing of databases is implemented in an external file parser DLL, dtv_odbc.dll, that is included with dtSearch and the dtSearch Engine. The dtv_odbc.dll file must be located in a subfolder named "viewers" under the dtSearch Engine "Home" directory. For more information, see the dtSearch Engine help topic, Options and the Home and Private Directories.

dtSearch can also index some database formats directly, without the need for OBDC drivers. These include: XML, CSV (comma-separated values), and XBase (FoxPro, dBASE, and other .DBF-compatible formats).

Additional Information

For more information about indexing databases with dtSearch, please see the following topics:

Database and Field Searching in the dtSearch Engine help file.

How to get field data in search results

How to add fields to documents

Troubleshooting database indexing

How to use dtSearch Web with dynamically-generated content

Microsoft Resources

Updated Microsoft ODBC drivers are available from the Microsoft web site. These drivers enable dtSearch Desktop or the dtSearch Engine to index and search Access, Paradox, FoxPro, and dBase databases. For download information, please see this article in the Microsoft Knowledgebase:

Q239114 ACC2000: Updated Version of Microsoft Jet 4.0 Available in Download Center

http://support.microsoft.com/support/kb/articles/Q239/1/14.ASP

The Microsoft Data Access Components (MDAC) are available from the Microsoft web site at www.microsoft.com/data. The MDAC download includes Active Data Objects and OLE DB providers.

Documentation for the Microsoft OLE DB provider for Oracle is available in the MSDN Library, here:

http://msdn.microsoft.com/library/default.asp?URL=/library/psdk/dasdk/oled19id.htm

Oracle Resources

For information on Oracle object models, including the Oracle Provider for OLE DB and Oracle Objects for OLE, please see these articles on the Oracle web site:

http://technet.oracle.com/tech/nt/O8iR2_Windows.html


How to integrate the dtSearch Engine with a language analyzer

dtSearch Home

Downloads

Quick Start

How to integrate the dtSearch Engine with a language analyzer

Last Reviewed: October 8, 2003

Article: DTS0192

Applies to: dtSearch Engine 6

The dtSearch Engine versions 6.22 and later have a Language Analyzer API that can be used to integrate morphological analyzers and custom or dictionary-based word breakers into the dtSearch Engine indexing process.

For C++ sample code demonstrating use of the Language Analyzer API, see the examples\cpp\LanguageAnalyzer folder after installing the dtSearch Engine. A CSampleLanguageAnalyzer class that can be used as a starting point for development is in la_sample.cpp in this folder. For API documentation, see the "Language Analyzer API" topic in the dtSearch Engine help file, dtengine.chm.

Additional sample code demonstrating use of the Language Analyzer API to integrate with Rosette® Globalization Platform from Basis Technology Corp. is also available from tech@dtsearch.com. However, because the Basis Technology Corp. product's API is covered by a nondisclosure agreement, we can only provide this sample code to customers who have signed a nondisclosure agreement with Basis Technology Corp. To obtain the sample code, email or fax the signature page of your agreement to tech@dtsearch.com or fax it to 301-263-0781.

Copyright © 1995-2005 dtSearch Corp.


How to write dtSearch ASP applications without using Session

dtSearch Home

Downloads

Quick Start

How to write dtSearch ASP applications without using Session

Last Reviewed: February 10, 2001

Article: DTS0156

Applies to: dtSearch Text Retrieval Engine 6

Background

A web-based application built with the dtSearch Engine will generally include the following elements:

(1) The user enters a search request on a search form and click a button to search

(2) The ASP script on the server receives the search request, passes it to the dtSearch Engine in a SearchJob, and obtains a SearchResults object

(3) The script then uses the SearchResults object to write HTML search results with links to view the documents, usually with hits highlighted.

(4) When a user clicks on one of the links, ASP script on the server receives the request and uses the dtSearch Engine to highlight hits in the selected document.

Highlighting hits is done in one of two ways: (1) a dtSearch FileConverter object is used to generate an HTML version of the document text, with hits marked, or (2) in the case of PDF files, an XML stream is sent to Adobe Reader to tell Adobe Reader which words to highlight. (The XML stream for PDF files is generated by the SearchResults object's MakePdfWebHighlightFile method. For more information on this method, please see Article DTS0152, Troubleshooting PDF hit highlighting)

The SearchResults object is helpful for hit highlighting because it provides methods like MakePdfWebHighlightFile that are otherwise difficult or impossible to duplicate. However, a script will have the SearchResults object in step (3) (when writing HTML search results back to the user) but not in step (4) (when a request is received to view a document with hit highlighting, after the user has obtained the list of retrieved documents).

Disadvantages of using the Session

Because of the need to have a SearchResults object to highlight hits, developers will often attach the SearchResults object to the Session variable, making it available to scripts in the application throughout the user's session. Using the Session variable this way has a number of disadvantages:

(1) It is harder to add additional servers to the site, because each user has to be directed to the same computer he or she started with.

(2) The SearchResults object remains in memory until the server's Session timeout is reached, consuming memory during this time (about 1-4k for each search results item).

(3) When the Session timeout limit is reached, the SearchResults object disappears and the user has to search again to see additional documents from search results.

(4) The Session variable is sensitive to the interaction between the browser and the server and to the server's configuration settings, so an application that depends on the Session will be more fragile than one that does not.

Ideally, a web-based searching application should be stateless, so that the server can retain no information at all about a user after the search results are returned, but the user can click on an item in search results and view it with hits highlighted, regardless of how long it has been since the search.

Stateless SearchResults

To provide a Stateless way to access SearchResults, the dtSearch SearchResults object has UrlEncodeItem and UrlDecodeItem methods. These methods let you encode individual search results items in the links written to the search results list, and decode an item when the user clicks on it. UrlEncodeItem creates a query string that identifies the document retrieved and that includes the hit offsets. UrlDecodeItem interprets that query string to make a SearchResults item matching the item that was encoded with UrlEncodeItem.

(1) Writing search results

When writing search results, for each item included in the list, your ASP script would use UrlEncodeItem to make a link for that item, like this:

selfUrl = "http://" & Request.ServerVariables("SERVER_NAME") & _

Request.ServerVariables("SCRIPT_NAME")

results.GetNthDoc(iDoc)

encodedItem = results.UrlEncodeItem()

link = selfUrl & "?cmd=getdoc&" & encodedItem

Response.Write "<A href=" & quote & link & quote & " target=doc>"

Response.Write results.DocDetailItem("_shortName") & "</A><BR>"

(2) Reconstructing search results

When a user clicks on one of the links from the sample above, the script will receive a query string containing a "cmd=getdoc" value followed by the URL-encoded search results item that the user clicked on. To make a search results object containing only this item, the script would implement the "getdoc" request starting with something like this:

sub GetDoc

' Reconstruct search results for the item the user clicked on

Dim results

Set results = Engine.NewSearchResults

results.UrlDecodeItem(Request.ServerVariables("QUERY_STRING"))

results.getNthDoc(0)

UrlDecodeItem is called with the entire query string submitted to the script. It does not matter if this string contains additional values, as long as it also contains the query generated by UrlEncodeItem(). After calling UrlDecodeItem(), the search results object contains a single document record and can be used to highlight hits in that document.

Sample Code

For sample code to a complete ASP application that uses UrlEncodeItem and UrlDecodeItem, please see the dtsearch.asp sample included with the dtSearch Text Retrieval Engine 6. It will be in your <dtsearch>\examples\asp folder.


Implementing Document Classification

dtSearch Home

Downloads

Quick Start

Implementing document classification

Last Reviewed: July 11, 2002

Article: DTS0179

Applies to: dtSearch Engine 6

Often it is useful to supplement a keyword search with a criterion that relates to a document's classification. For example, in a collection of HTML documents about cars, an option could be provided to limit a search by brand (Ford, Honda, etc.) or vehicle type (sedan, SUV, minivan).

Building classification information into the index

The first step to implementing document classification is to associate one or more fields with each document that will be used to define the categories. With cars, there could be a "Brand" field and a "Type" field. The field can either be embedded in the document (for example, as a META tag in an HTML file, or a document property in a Word document), or it could be associated with a document when the file is indexed, using the data source API. Each HTML file in the document collection on cars could contain two meta tags, like this:

<meta name="Brand" content="Ford">

<meta name="Type" content="sedan">

For more information on ways to associate field information with documents in an index, see:

How to add fields to documents during indexing

"Adding fields to documents" in the dtSearch Engine help file, dtengine.chm

Searching using classifications (option 1 - field searches)

Once the classification information is built into the index, it can easily be added to a search request. If a user indicates that he only wants to search for documents on Ford cars, the search request would be modified by adding a field search of the Brand field on Ford, like this:

[1] (user request) and (brand contains ford)

If the user wants to search only documents on Ford cars that are sedans, the search request would be modified by adding two field searches, one specifying that Brand must contain Ford, and the other specifying that the Type must be sedan:

[2] (user request) and ((brand contains ford) and (type contains sedan))

Searching using classifications (option 2 - xfilter searches)

While field searches are effective at classifying documents, in very large document collections they are relatively inefficient. Suppose, for example, that the collection of documents about cars contained 20,000,000 HTML files. The "brand contains ford" expression would generate millions of document matches, and the "type contains sedan" would also generate millions of matches. Adding these two criteria to a search will, therefore, greatly increase the amount of work that must be done to resolve the search request. In large document collections, optimizing these types of queries is critical for good performance.

One way to do this is with a type of query expression, an "xfilter" search, that is designed to implement a simple filter for documents, without considering hit locations or checking word proximity information. Documents either match the filter and can be retrieved, or they do not and are excluded. An xfilter can be combined with a boolean query using the standard and/or/not connectors, and can be based on the document name, date, or size, or the presence in the document of a word. Examples:

[3] (user request) and xfilter(name "abc*.html")

This would match any document that contains (user request) and that is named abc*.html.

[4] (user request) and xfilter(word "sedan")

This would match any document that contains (user request) and that also contains the word "sedan". (Because xfilters are resolved without considering word locations, there is no way to specify a phrase in an xfilter. Only single words can be included in xfilter expressions.)

To add a field restriction to a word in an xfilter expression, add the field name before the word, separated by ::, like this:

[5] (user request) and (xfilter(word "Type::sedan") and xfilter(word "Brand::Ford"))

While this search request is logically equivalent to [2] above, it is much faster. Because dtSearch knows that it can ignore word locations, it can optimize the query, greatly reducing the time needed to process the two classification criteria. The :: syntax is available in dtSearch versions 6.1 and later.

Searching using classifications (option 3 - SearchFilter objects)

Another approach to classifications can take advantage of the fact that, in most cases, a relatively limited number of categories will be used repeatedly in searches. Rather than processing the classification criteria for every search, the classification criteria can be generated once each time the index is updated, and then reused for searches. The SearchFilter object provides a way to do this.

A SearchFilter is an efficient, in-memory object that identifies a set of documents from one or more indexes. When the SearchFilter is attached to a SearchJob, the SearchJob will only return documents that are part of the SearchFilter. Because of the way they are implemented, SearchFilters are faster than any other mechanism for selecting items to be included in search results. Once a SearchFilter has been constructed, it can be saved to disk and read from disk as needed. A web-based application can keep its search filters in memory for use in searches. Once constructed, a search filter can be accessed from multiple threads simultaneously.

SearchFilters do not use names to identify documents because a filter may specify thousands, or hundreds of thousands, of documents, and a table of filenames would take too much memory and would take too long to check. Instead, each document is identified by (a) the index it belongs to, and (b) the document's DocId, a unique integer that is assigned to each document in an index. The docId for a document can be obtained by searching for the document; the document properties returned in Search Results will include the docId.

A SearchFilter is implemented in the dtSearch Engine using a table of bit vectors, one for each index in the filter. Each bit vector has one bit for each document in its index (the bit for each document corresponds to its docId). For example, a SearchFilter for a single index with 1,000,000 documents would have 1,000,000 bits, or 125 kilobytes of data. When a SearchFilters is written to disk, it is stored in a compressed format that generally takes substantially less space than the in-memory representation (which is optimized for speed).

For more information on SearchFilters, see:

http://support.dtsearch.com/webhelp/dtengine/searchfilter.htm (VB, ASP)

http://support.dtsearch.com/webhelp/dtengine/dtssearchfilter.htm (C++)

http://support.dtsearch.com/webhelp/dtengine/javadoc/com/dtsearch/engine/searchfilter.html (Java)


.NET Compatibility Information

dtSearch Home

Downloads

Quick Start

.NET Information

Last Reviewed: July 20, 2004

Article: DTS0172

Applies to: dtSearch Engine 6

Native .NET Interface

The dtSearch Engine (versions 6.4 and later) includes a native .NET interface implemented in dtSearchNetApi.dll. For API documentation, see:

http://support.dtsearch.com/webhelp/dtengine/net/doc/index.html

Troubleshooting: "File or assembly name dtSearchNetAPI, or one of its dependencies, was not found."

dtSearchNetApi.dll depends on dten600.dll, so if either DLL cannot be located, this error will occur. To correct the problem, either (1) add the folder where dten600.dll is located (the dtSearch Engine BIN folder) to the system PATH, or (2) copy dten600.dll into the application folder.

C# (using COM interop)

In .NET, COM objects are used by importing the type library using the tlbimp.exe tool (included with the .NET SDK). Using tlbimp.exe, you can generate an assembly that wraps the dtSearch Engine COM interface. tlbimp.exe also has options to customize the assembly (changing the name, adding public key/private key information, etc.) The samples below include a pre-built assembly.

API documentation for the COM interface can be found in dtengine.chm, in the section on ActiveX (COM) Automation. (This API is also sometimes referred to in the documentation as the "Visual Basic" API.)

A C# implementation of a search application is available for download at: http://support.dtsearch.com/ftp_pub/csSample.zip

A C# implementation of a database indexing application using ADO.NET is available for download at http://support.dtsearch.com/ftp_pub/ado_demo_cs.zip

VB.NET (using COM interop)

Some dtSearch Engine API functions, such as NewSearchJob, return Object rather than a specific type. If you are compiling with the "Strict" option, you will need to cast these objects to the appropriate type. Example:

Visual Basic 6 code to initialize a SearchJob:

Dim mySearchJob as dtengine.SearchJob

Set mySearchJob = theEngine.NewSearchJob

VB.NET code to initalize a SearchJob

Dim mySearchJob as dtengine.SearchJob

mySearchJob = CType(TheEngine.NewSearchJob(), dtengine.ISearchJob)

The CType expression converts the Object returned by TheEngine.NewSearchJob() to a SearchJob. (Note the odd spelling of the SearchJob type name, which is needed to avoid a cast error on execution. The "I" in front of ISearchJob identifies the COM class associated with SearchJob.)

Sample code: a VB.NET-compatible version of the dtdemo sample application is available for download at http://support.dtsearch.com/ftp_pub/dtdemo_DotNet.zip

Applications that use the DataSourceToIndex in IndexJob must use a class object built with COM support as the data source. While VB 6 objects inherently support COM, .NET objects do not. The easiest way to create a .NET class with COM support is to use the "Add Class" wizard: Project > Add Class... > COM Class.

Sample code: a VB.NET version of the ado_demo sample is available for download at http://support.dtsearch.com/ftp_pub/adodemo_DotNet.zip

ASP.NET

See note under "VB.NET" for information on creating dtSearch Engine objects in the VB.NET language. A simple ASP.NET search application is available for download at: http://support.dtsearch.com/ftp_pub/asp_DotNet.zip


Optimizing search performance with the dtSearch Engine

dtSearch Home

Downloads

Quick Start

Optimizing search performance with the dtSearch Engine

Last Reviewed: July 22, 2002

Article: DTS0153

Applies to: dtSearch Engine 6

This article describes programming techniques to maximize searching performance with the dtSearch Engine. Available memory and hard disk space, and Windows configuration settings such as the use of NTFS compression, can also have an effect and are discussed in this article: What web servers does dtSearch Web 6 work with?

Execute vs. ExecuteInThread

The MaxFilesToRetrieve value

The DelayDocInfo flag

The TimeoutSeconds and AutoStopLimit values

Designing Efficient Queries

How dtSearch executes a search

For each index that dtSearch searches, there are two steps to the search: (1) word lookup, and (2) enumeration of the documents that satisfy the search request. The first step generally takes a small fraction of a second, even for very large indexes, and is usually not a significant part of the total time required for the search. The time required for the second step is proportional to the number of documents retrieved. Therefore, the second step will take ten times longer for a search that finds 1000 files than for a search that finds 100 files.

The key to fast searching is, therefore, minimizing the time required for the second step, enumerating the files that match the search request.

Execute vs. ExecuteInThread

Applications written in Visual Basic, Delphi, or other languages that use the COM interface have two ways to execute a search: Execute, or ExecuteInThread. The choice between these two depends on how the calling application will monitor the progress of the search.

The most efficient option is to use Execute with the AutoStopLimit, TimeoutSeconds, and MaxFilesToRetrieve properties set up before the search to limit the amount of resources a search will consume. This is usually the best option for use on web servers.

Desktop-based applications usually have to monitor and control a search job more closely. Usually there will be a "Cancel" button that the user can click to halt the search, and the application may also display a running count of the number of files retrieved. The need to implement a "Cancel" button and to track the progress of the search can be handled in either of two ways: (1) callback functions implemented through the StatusHandler property of a SearchJob, or (2) searching in a separate thread.

When an application uses Execute to start a SearchJob, it can implement ReceiveFound and CheckForAbort callback functions to monitor the progress of a search and cancel the search if the user hits a "Cancel" button. ExecuteInThread starts the search job in a separate thread and returns immediately, and provides IsThreadDone and AbortThread methods to enable the calling application to cancel the search.

In an application that implements a "Cancel" button feature, ExecuteInThread is much faster than Execute because it eliminates the need for the callback functions ReceiveFound and CheckForAbort. Callbacks from Visual Basic or other COM-based languages are very time-consuming because each one involves a separate IDispatch invocation. Using Execute and the callbacks generates an IDispatch invocation for each implemented callback function, each time a document is found. This can have a very significant effect on search performance. Therefore, for applications that need to implement something like a "Cancel" button, ExecuteInThread should be used instead of Execute for SearchJobs.

Note: An application that uses ExecuteInThread should not call IsThreadDone() continuously in a loop to check whether the thread is done. Instead, it should call Sleep() between IsThreadDone() calls to give the searching thread time to work.

The MaxFilesToRetrieve value

How to use MaxFilesToRetrieve
C/C++ | Set maxFilesToRetrieve in dtsSearchJob, and use a dtsSearchResults object to obtain search results.
C++ Support Classes | Call DSearchJob's SetMaxFiles() method
Java | Call SearchJob.setMaxFilesToRetrieve()
Visual Basic ASP Delphi | Set the SearchJob.MaxFilesToRetrieve and set SearchJob.ResultsInMemory to True

The MaxFilesToRetrieve value limits the number of items returned in search results to the best-matching documents. The search is fully executed, so you get the total number of files retrieved, but only the best-matching items are returned in search results. This value can be used in combination with the DelayDocInfo flag to greatly improve search performance

The DelayDocInfo flag

How to use DelayDocInfo
C/C++ | Set the dtsSearchDelayDocInfo bit in the dtsSearchJob.searchFlags
C++ Support Classes | Call DSearchJob's SetDelayDocInfo() method
Java | Call SearchJob.setSearchFlags(Constants.dtsSearchDelayDocInfo)
Visual Basic ASP Delphi | Set the SearchJob.DelayDocInfo flag to True

The DelayDocInfo flag optimizes searches by waiting until document information records are requested by SearchResults before reading them from an index. For example, consider a search with a MaxFilesToRetrieve value of 100 (so only the best-matching 100 documents will be returned) that retrieves 2000 documents. Using the DelayDocInfo flag tells dtSearch to ignore the document information for the 1900 documents that did not make it into the top 100 and only to read the information for the best-matching 100 files.

Using the DelayDocInfo flag can improve the performance of long searches by a factor of 10 or more.

Search callback notifications During a search, your program can receive callback notifications as each file is retrieved. If the DelayDocInfo flag is set, you can still receive these notifications, but the filenames will be blank.

The TimeoutSeconds and AutoStopLimit values

How to use TimeoutSeconds
C/C++ | Set the timeoutSeconds value in dtsSearchJob
C++ Support Classes | Call DSearchJob's SetTimeoutSeconds() method
Java | Call SearchJob.setTimeoutSeconds()
Visual Basic ASP Delphi | Set the SearchJob.TimeoutSeconds property

How to use AutoStopLimit
C/C++ | Set the autoStopLimit value in dtsSearchJob
C++ Support Classes | Call DSearchJob's SetStopLimit() method
Java | Call SearchJob.setAutoStopLimit()
Visual Basic ASP Delphi | Set the SearchJob.AutoStopLimit property

Using the MaxFilesToRetrieve value and the DelayDocInfo flag, the amount of time required for each item retrieved in a search can be reduced quite a bit, but search time will still be roughly proportional to the number of documents retrieved. While a search that finds 1000 documents will still be very fast, a search that retrieves a sufficiently large number of documents can still take a long time.

To prevent searches that find hundreds of thousands or millions of documents from consuming an excessive amount of resources on a server, you can use two limits on a search job that will halt the search unconditionally.

First, you can use TimeoutSeconds to halt a search that takes more than a certain number of seconds. For example, if you set TimeoutSeconds = 5, a search will always halt after five seconds.

Second, you can use the AutoStopLimit to halt a search after a specified number of documents have been retrieved. For example, if you set AutoStopLimit to 1000, the search will automatically halt after 1000 documents have been found.

AutoStopLimit vs MaxFilesToRetrieve AutoStopLimit tells dtSearch to stop searching after a certain number of files have been found, while MaxFilesToRetrieve tells dtSearch how many files to return in search results. For example, if AutoStopLimit is 500 and MaxFilesToRetrieve is 100, then the search will return the best-matching 100 files out of the first 500 retrieved.

How to tell if a limit was reached The error handler for a search job will tell you if the search job was terminated because of one of these limits. If the TimeoutSeconds value was reached, it will contain dtsErTimeout (13). If the AutoStopLimit was reached, it will contains dtsErSearchLimitReached (120).

Efficient Queries

dtSearch is designed to perform fast searches for keywords in documents. In building applications with dtSearch, developers often want to use the search indexes to perform other, more database-like functions. While this is possible with dtSearch, it is important to understand the trade-offs of various query types to make efficient use of dtSearch search functions.

(1) Minimize the number of "words" matched by your query.

Consider a document manager that uses a document "ID" consisting of a two-letter state code and a number, such as ID_NY01234 or ID_CA54312. To enable user to search by state, the document manager would add "and ID_NY*" or "and ID_CA*" to the end of the query, so that only documents matching the requested state prefix would be returned.

The problem with this approach is that it generates a separate word match for each document in the state requested. If there are 20,000 New York documents, the search involves 20,000 words in addition to the user's search request.

A better approach is to design identifiers to support the types of searches you expect to perform. In this example, adding a space before the number in the document ID would dramatically improve the efficiency of the queries needed to search by state. The improved document IDs would look like "ID_NY 01234" and "ID_CA 54312". Making the state and the number separate words makes it possible to search for "ID_NY" (one word) instead of "ID_NY*" (potentially thousands of words) when looking for New York documents.

(2) Use File Conditions when possible

If for some reason it is impossible to avoid a search condition that generates thousands of word matches, consider using a "word" File Condition in your search request instead of a text search for the word. A File Condition is a condition that documents must satisfy in a search but that does not generate hits. File Condition searches can be much faster than adding words to a search request. Most commonly, File Conditions are used for things like filename patterns or date range searches (for example, a search for documents modified after December 12, 2000, or for documents named "x*.doc"). You can also require that documents satisfying a condition must contain one or more words. Examples:

xfilter(word "abc*")

Document must contain a word beginning with abc

xfilter(word "date20020101~~date20020131")

Document must contain "date" followed by a range from 20020101 to 20020131

xfilter(word "datefield::20020101~~20020131")

Document must contain a value between 20020101 and 20020131 in the field named datefield

Please see the "File Conditions" topic in the dtSearch Engine help file for more information on the syntax for adding File Conditions to a search. Field names in file conditions are supported in dtSearch Engine versions after 6.10.

(3) Avoid using text searches that will match all or most of your documents

Another feature often added to document management products is a set of document properties that users can search on. For example, a collection of agency letter rulings might have an "IsSuperceded" flag indicating that the letter ruling is no longer regarded as valid. To search for only current documents, a document manager might add "...and (IsSuperceded contains false)" to the end of every query. A more efficient way to to implement this type of searching is to apply a filter to search results after the search: implement the IsSuperceded flag as a stored field, search without the IsSuperceded part of the query, and then filter out superceded documents by checking the IsSuperceded flag in search results. This can also be done using the SearchFilter object, which provides an efficient way to limit a search to a subset of the document collection. For information on SearchFilter objects, see the "SearchFilter" topic in dtengine.chm (Visual Basic, ASP) or the dtsSearchFilter topic (C++).

For more information on efficient ways to implement document classification in searching, see Implementing document classification.


Sorting search results

dtSearch Home

Downloads

Quick Start

Sorting search results

Last Reviewed: March 27, 2003

Article: DTS0184

Applies to: dtSearch Engine 6.1 and later

How to sort search results using the dtSearch Engine

After a search, search results are returned sorted by relevance. You can use the flags below to re-sort search results by other criteria. In Visual Basic, use the Sort() method in SearchResults. In C++, use the sort() function in dtsSearchResults or DSearchResults.

The sort function has two arguments: flags, and field. The field is a string that is used when flags includes dtsSortByField. A field used for sorting must be designated as a stored field during indexing so that the value will be available for sorting.

Another way to sort by a custom sort key is to use the SearchResults.SetSortKey() method to assign a sort key to each item in the search results list. After each key is assigned, sort using the dtsSortBySortKey flag.

How to sort search results using dtSearch Web

In dtSearch Web, the sort criterion is passed through the "sort" form variable. The pageSize variable determines how many items are returned in each page, and the maxFiles variable determines how many items are retrieved in total.

If sort is not "size", "name", "date", or "hits", then dtSearch Web will assume that the sort key is a stored field and will use the dtsSortByField search flag.

The sort form variable can be followed by colon and a numerical value that will be combined with the sort type. Example: "subject:0x210002". The numerical value is a combination of any of the sort flags described below, expressed as a hexadecimal integer.

hits

In a search that is sorted by hits, dtSearch Web will return up to maxFiles of the most relevant documents, organized into pages each with pageSize documents. If pageSize is not specified in the search form, the maxFiles value will be used as the page size.

date

Sorting by date works like sorting by hits, except that the most recent documents are returned instead of the most relevant.

size, name, and custom fields

When sorting by criteria other than hits or date, dtSearch will return up to maxFiles of the most relevant files, organized into pages each with pageSize documents, with the entire results list sorted by the specified criteria. For example, if the sort criterion is "size", pageSize is 10, and maxFiles is 100, dtSearch will find the 100 most relevant files (not the 100 largest), and will display them in pages of 10 documents, sorted by size.

Summary of Search Flags

Flag | Meaning
dtsSortByName (0x4) | Sort by filename (without path)
dtsSortByLocation (0x400) | Sort by the path of the file
dtsSortByDate (0x8) | Sort by modification date, including the time
dtsSortByTime (0x800) | Sort by modification time, ignoring the date
dtsSortByHitCount (0x400000) | Sort by number of hits
dtsSortByRelevanceScore (0x800000) | Sort by relevance score
dtsSortByHits (0x10) | Sort by hit count or score, depending on whether the automatic term weighting was used in the search
dtsSortBySize (0x20) | Sort by file size
dtsSortByField (0x40) | Sort by one of the elements in the userFields set. The second argument userField identifies the field to use for sorting
dtsSortByIndex (0x80) | Sort by the index retrieved from
dtsSortByType (0x100) | Sort by file type
dtsSortByTitle (0x200) | Sort by the title string
dtsSortBySortKey (0x1000) | Sort by caller-specified sort key (use setSortKey for each item to specify the key)
dtsSortAscending (0x2) | Add dtsSortAscending to the flag to request an ascending-order sort. Otherwise the sort will be in descending order.
dtsSortNumeric (0x20000) | Sort by the numeric value of a field instead of its string value. This would cause "20" to be considered greater than "9". The sort key will be a signed, 32-bit integer.
dtsSortFloatNumeric (0x100000) | Sort by the floating point numeric value of a field instead of its string value. The sort key will be a double-precision floating point number.
dtsSortCleanText (0x200000) | Remove leading punctuation or white space from sort value before sorting. Also removes "re:", "fw:", and "fwd:".
dtsSortCaseInsensitive (0x010000) | Make string comparisons in the sort case-insensitive
dtsSortPdfUseTitleAsName (0x040000) dtsSortHtmlUseTitleAsName (0x080000) | When sorting by filename, use the PDF or HTML Title as the filename for PDF or HTML files.

Add dtsSortAscending to specify an ascending order (the default is descending order). Add dtsSortCaseInsensitive or dtsSortNumeric to modify a sort based on a string criterion.


Troubleshooting -- ConvertPath returns a blank string

dtSearch Home

Downloads

Quick Start

Troubleshooting -- ConvertPath returns a blank string

Last Reviewed: April 28, 2003

Article: DTS0157

Applies to: dtSearch Text Retrieval Engine 6

After a search, the SearchResults object contains a list of the files that were retrieved. These files are expressed as local paths, such as c:\docs\sample.doc. For web-based applications, it is usually preferable to report document locations using virtual directory names, such as /docs/sample.doc. Also, when highlighting hits in PDF files or creating direct links to retrieved documents, a virtual directory name is required.

To provide a way to map local paths to virtual paths, the dtSearch Engine Server object has a ConvertPath method:

ConvertPath(Path as String, iSite as Long, flags as Long, reserved as String)

as String

If ConvertPath returns a blank string, it means that either (1) the folder passed in does not have a virtual directory associated with it, (2) the registry tables have not been refreshed since the virtual directory was created, or (3) there is something wrong with the arguments being passed to ConvertPath.

1. Check that the folder has a virtual directory associated with it

Open Internet Services Manager and check the list of virtual directories for the web site. Each web site has its own list of virtual directories, so if you have more than one web site, check that the one you are calling ConvertPath for has the virtual directory definition for the folder.

2. Run dtSearch Web Setup

In dtSearch Desktop, click File > dtSearch Web Setup. When dtSearch Web Setup program opens, close it without doing anything. This will refresh the registry tables that dtSearch uses to store virtual directory information. There is no need to rebuild your indexes after doing this.

3. Stop and restart the W3 service, or reboot the server

Restarting the W3 service will make the ASP application re-read the virtual directory tables from the registry. (You can also call ConvertPath with the dtsCpReadFromRegistry flag to do this.)

In some cases ASP applications are not unloaded even when the W3 service is stopped and restarted. In these cases, a reboot may be necessary to reload the ASP application.

4. Check the ConvertPath arguments

Path: Check that the path is exactly the same as the virtual directory mapping, or a subdirectory of a virtual directory mapping. Note that mapped network drives cannot be used as virtual directories because Internet Information Server (IIS) and components called from IIS may not see the mapping.

iSite: Check that this matches the web site that has the virtual directory

flags: In ASP, if you use a named constant like dtsCpMakeVirtual, and the constant is not declared in the script or is incorrectly referenced through a type library, ASP will not warn you of the mistake and will instead create a new variable with a value of zero (0). dtsCpMakeVirtual is defined as 1, so try replacing dtsCpMakeVirtual with 1 in your script.

5. Use a debug log to check how dtSearch is handling the call internally

A debug log can help you to check how dtSearch is interpreting the values passed to ConvertPath. For information on debug logging, see:

Diagnostic tools for Visual Basic and ASP Developers

Additional Information

See the ConvertPath documentation below for more information, or see the "Server" topic in the dtSearch Engine documentation (dtengine.chm).

ConvertPath

Use the Server object's ConvertPath method to convert between virtual and local paths. iSite is the id of the web site on a web server. If there is only one web site on the server, use 1. The ASP value Request.ServerVariables("INSTANCE_ID") can be used if the web server has more than one web site. (The web site id is needed because each site has its own mapping between virtual and local paths.) For the flags value, use dtsCpMakeVirtual to convert from local to virtual paths. Example:

VirtualPath = Engine.ConvertPath(LocalPath, _ &

Request.ServerVariables("INSTANCE_ID"), dtsCpMakeVirtual, "")

Internet Information Server stores its mapping between virtual and local paths in the Metabase. Reading this information from the Metabase is time-consuming and requires access rights that may not be available to a process running in the web server context. Therefore, the dtSearch Engine caches this information in the registry, under this key:

HKEY_LOCAL_MACHINE\Software\dtSearch Corp.\dtSearch\Virtual Roots

By default, ConvertPath will use the cached table values. Call ConvertPath with the dtsCpRefreshMappings + dtsCpStoreInRegistry flags to update the cached information. Running dtWebSetup.exe, the dtSearch Web Setup program, will also update this information. (You do not have to do anything in dtWebSetup -- just run it and exit to refresh the tables.)


Troubleshooting: System.IO.FileNotFoundException accessing dtSearchNetApi.dll

dtSearch Home

Downloads

Quick Start

Troubleshooting: System.IO.FileNotFoundException accessing dtSearchNetApi.dll

Last Reviewed: October 24, 2004

Article: DTS0197

Applies to: dtSearch Engine 6.40 and later

Symptom:

This error occurs running an application that uses the dtSearch Engine's .NET API:

An unhandled exception of type 'System.IO.FileNotFoundException' occurred in [program name]

Additional information: File or assembly name dtSearchNetApi, or one of its dependencies, was not found

Resolution

dtSearchNetApi.dll depends on dten600.dll. When you add a reference to dtSearchNetApi.dll in your project, Visual Studio may copy dtSearchNetApi to the project folder. However, dten600.dll is not copied, and because dtSearchNetApi.dll is not in the same folder as dten600.dll, Windows may not be able to resolve the dependency on dten600.dll.

To fix the problem:

(1) Copy dten600.dll to the folder where dtSearchNetAPI.dll is located,

(2) Copy dten600.dll to a folder that is listed on the system PATH, such as the c:\windows folder, or

(3) Add the folder where dten600.dll is located to the system PATH (see "How to change the system PATH" below for instructions).

See also:

dtSearch .NET API Information

.NET API Documentation

http://support.dtsearch.com/webhelp/dtengine/net/doc/index.html

How to change the system PATH

The PATH is a list of folder names that Windows uses to locate executables. The following steps will add the dtSearch program folder to the system PATH.

(1) Click Start > Settings > Control Panel

(2) Click System

(3) Click the Advanced tab.

(4) Click the Environment Variables button

(5) Under System variables, locate the "Path" variable and click on it

(6) Click the Edit button

(7) Add this to the end of the current value (assuming you installed dtSearch to the default location):

;c:\Program Files\dtSearch Developer\bin

The semicolon is needed to separate the dtSearch directory name from the last item listed in the existing PATH.

A reboot may be necessary for some programs to recognize the change to the PATH.

For more information on Windows environment variables, see:

How To Manage Environment Variables in Windows XP

http://support.microsoft.com/default.aspx?scid=kb;en-us;310519&sd=tech


Troubleshooting -- Using the dtSearch Engine with Access databases

dtSearch Home

Downloads

Quick Start

Troubleshooting -- Using the dtSearch Engine with Access databases

Last Reviewed: May 13, 2002

Article: DTS0177

Applies to: dtSearch Engine 6

dtSearch and the dtSearch Engine use the Microsoft ODBC database drivers to access desktop database files (Access, Fox Pro, Paradox, DBase) through ODBC. When a database file is indexed, dtSearch will treat each record of the database as a separate document, and fields in the database will be searchable by name.

Symptoms

- When an Access (*.MDB), Paradox, or XBase file is indexed in your application, it is treated as one long text file.

- Hit highlighting does not work (no text is returned from FileConverter or dtssConvertFile).

Troubleshooting Steps

(1) Check that the ODBC drivers are installed on your computer.

Try indexing the database file with dtSearch Desktop. If the file is still indexed as a text file rather than a series of database records, see Troubleshooting database indexing for steps to get ODBC database indexing working with dtSearch Desktop.

(2) Check that the dtSearch Engine is set up to use ODBC

The dtSearch Engine connects to ODBC databases through an optional external file parser, dtv_odbc.dll. When the dtSearch Engine is initialized, it checks for a "Viewers" folder under the dtSearch Engine "HomeDir" folder, and loads external file parsers from that folder. For example, if the HomeDir is set to c:\MyProgram, dtSearch will look for external file parsers in c:\MyProgram\Viewers. Therefore, to enable ODBC indexing, dtv_odbc.dll should be located in this folder.

(a) Check that your application is setting the HomeDir correctly.

In ASP, Visual Basic and Java, the HomeDir is set through the HomeDir property of the Options object. In C++, it is set through the homeDir member of dtsOptions.

(b) Use debug logging to check that dtv_odbc.dll is being found and loaded.

For information on generating a debug log in your application, see:

Diagnostic Tools for Visual Basic Developers

Diagnostic Tools for C++ Developers

Diagnostic Tools for Java Developers

After logging an index update or an attempt to highlight hits in a database record, search the debug log for dtv_odbc.dll to see if it was found in the HomeDir folder and if it was loaded successfully.

(c) Use debug logging to check for ODBC driver errors

A debug log will also log any errors reported by the ODBC driver that the dtSearch Engine invoked to open a database. To find these errors, search the log file for "Odbc".

Additional Information

For more information about indexing databases with dtSearch, please see the following topics:

Database and Field Searching in the dtSearch Engine help file.

How to index databases with the dtSearch Engine

How to get field data in search results

How to add fields to documents

Troubleshooting database indexing

How to use dtSearch Web with ODBC


Troubleshooting: java.lang.UnsatisfiedLinkError

dtSearch Home

Downloads

Quick Start

Troubleshooting -- java.lang.UnsatisfiedLinkError

Last Reviewed: October 27, 2004

Article: DTS0198

Applies to: dtSearch Engine 6.20 and later

Symptom:

This error occurs running an application that uses the dtSearch Engine's Java API:

java.lang.UnsatisfiedLinkError: no dtsjava in java.library.path

Resolution: Windows

Under Windows, the Java interface to the dtSearch Engine is implemented with three files:

dtSearchEngine.jar, which has the Java classes

dtsjava.dll, a JNI wrapper around the dtSearch Engine's C++ interface

dten600.dll, the dtSearch Engine, which performs the indexing and searching functions

dtSearchEngine.jar just be in one of the folders that is listed in the CLASSPATH environment variable. Adding the dtSearch Engine examples/java/classes folder to the CLASSPATH, or copying its contents to a folder on the CLASSPATH, will make these classes available in Java.

dtsjava.dll and dten600.dll must be in one of the folders that is listed in the PATH environment variable. For instructions on changing the PATH environment variable, see "How to change the system PATH", below. Note that Windows has two PATH variables: one "System" path which applies to all users, and one for the current user. Changes to the current user path may not be visible to some programs, such as services or programs that run in the context of a web server. Therefore, any changes should be made to the System PATH. Usually a reboot is needed for changes to the System PATH to be recognized.

Resolution: Linux

Under Linux, the Java interface to the dtSearch Engine is implemented with two files:

dtSearchEngine.jar, which has the Java classes

libdtsjava.so, which includes both the dtSearch Engine and a JNI wrapper around the dtSearch Engine's C++ interface

dtSearchEngine.jar just be in one of the folders that is listed in the CLASSPATH environment variable.

libdtsjava.so must be in a folder that is listed in the LD_LIBRARY_PATH environment variable. You can also copy the dtSearch .so files to your /usr/local/lib folder, or use ldconfig to make them accessible on your system.

See also:

dtSearch Java API

How to change the system PATH in Windows

The PATH is a list of folder names that Windows uses to locate executables. The following steps will add the dtSearch program folder to the system PATH.

(1) Click Start > Settings > Control Panel

(2) Click System

(3) Click the Advanced tab.

(4) Click the Environment Variables button

(5) Under System variables, locate the "Path" variable and click on it

(6) Click the Edit button

(7) Add this to the end of the current value (assuming you installed dtSearch to the default location):

;c:\Program Files\dtSearch Developer\bin

The semicolon is needed to separate the dtSearch directory name from the last item listed in the existing PATH.

A reboot may be necessary for some programs to recognize the change to the PATH.

For more information on Windows environment variables, see:

How To Manage Environment Variables in Windows XP

http://support.microsoft.com/default.aspx?scid=kb;en-us;310519&sd=tech


Troubleshooting PDF hit highlighting (dtSearch Engine)

dtSearch Home

Downloads

Quick Start

Troubleshooting PDF hit highlighting (dtSearch Engine)

Last Reviewed: December 15, 2000

Article: DTS0152

Applies to: dtSearch Engine 6

This article covers problems that developers may encounter displaying PDF files with hit highlighting in Adobe Reader through a browser interface.

Related articles

Troubleshooting PDF Indexing (dtSearch Desktop)

Troubleshooting PDF Indexing (dtSearch Engine)

PDF file appears as blank page in dtSearch Web

PDF Files appear without hit highlighting in dtSearch Web

How PDF hit highlighting works

When a user clicks on a link to a PDF file on a web page, the browser loads Adobe Reader as a plug-in and uses it to display the page. Adobe Reader knows how to interpret a type of URL that provides hit highlight information. The URL format looks like this:

http://www.dtsearch.com/sample.pdf#xml=http//www.dtsearch.com/hits.xml

The #xml= portion of the link points to a URL that returns an XML stream describing the location of the hits in the PDF file. The format of the XML file is described in this document, which is also included in the Acrobat SDK.

Adobe Technical Note 5172 -- Highlight File Format

http://partners.adobe.com/asn/acrobat/docs/File_Format_Specifications/HighlightFileFormat.pdf

[Note: Registration with Adobe is required to access this file.]

When the hit highlighting API works, hits will be highlighted in Adobe Reader inside the user's browser and hit navigation buttons on the Adobe Reader toolbar will let the user navigate from hit to hit. If it does not work, often the only symptom you will see is the absence of hit highlighting.

Usually the #xml portion of the URL does not point to a text file but instead requests the XML from a script or program, like this:

http://www.dtsearch.com/sample.pdf#xml=http//www.dtsearch.com/dtsearch.asp?cmd=getPdfHits&idoc=5

The dtSearch Engine provides a MakePdfWebHighlightFile method in the SearchResults object to generate this XML stream. For sample code demonstrating this, please see the dtsearch.asp sample included with the dtSearch Engine.

Troubleshooting steps

1. Check that the XML stream is being generated.

If you are generating XML using an ASP script, a script error may be preventing the XML from being generated correctly. Because of the way PDF links are created, this script error would not normally be visible.

To check the XML stream returned by your application, first generate a link that should display a PDF file with hit highlighting, and then cut and paste the #xml= portion of the link into your Internet Explorer address bar with view-source: in front, like this:

view-source:http://www.example.com/SampleScript.asp?getPdfHits&...

The XML will appear in Notepad. (If you just enter the URL of the XML stream directly, the browser will refuse to display it because Adobe's XML hit highlight stream is not correct XML.) You may see a script error message, indicating why the XML stream was not generated.

2. Check that the XML stream is correct.

To do this, save the results of the view-source URL above from Notepad to a file named test.xml in the root folder of your web server. Save the PDF file to test.pdf in the same folder. Open your browser and enter the following URL, replacing "localhost" with the address of your web site, if appropriate:

http://localhost/test.pdf#xml=http://localhost/test.xml

If the XML stream is correct, test.pdf should appear in a browser window with hits highlighted. If hits are not highlighted, check the information in test.xml against the Adobe documentation of the Highlight File Format (see link above).

If incorrect XML is being generated, use the debug logging feature in the dtSearch Engine to generate a log of your request for the XML stream to see what is going wrong. For information on debug logging with the dtSearch Engine, see: "Diagnostic Tools for Visual Basic and ASP Developers".

3. Check the way you are returning the XML stream

The hit highlighting process is very sensitive to variations in the way you return the XML stream. See the GetPdfHits function in the dtsearch.asp sample included with the dtSearch Engine for sample code. Your script should: (1) indicate that the content type of the data is "text/xml", (2) set CacheControl to "no-cache" (this prevents old XML streams from being re-used), and (3) avoid using pragma no-cache, which we have found will prevent hit highlighting from working. Example:

Response.ContentType = "text/xml"

' DO NOT USE THIS: Response.AddHeader "Pragma", "no-cache"

' INSTEAD, USE THIS:

Response.CacheControl = "no-cache"

' Reconstruct search results for the item the user clicked on

Dim res

Set res = Engine.NewSearchResults

res.UrlDecodeItem(Request.ServerVariables("QUERY_STRING"))

res.getNthDoc(0)

Dim hits

hits = res.MakePdfWebHighlightFile()

response.write(hits)

Set res = nothing

4. Avoid returning framesets

A frameset is a convenient way to display search results, because the list of items can appear in one frame with the currently-selected document in the other. When setting up framesets, we recommend that you create the frameset in your search form, as dtSearch Web does, rather than trying to generate the frameset dynamically in response to a search.

It is possible to generate framesets with dynamically-generated content in the src= links, so that when a user clicks on a link the PDF file will appear inside a frame with other frames displaying other content relating to the search. The src= link for the PDF file would contain the same hit-highlighting URL that you would use if linking to the document directly.

There are two problems with this approach:

(1) When a link asks the browser to open a PDF file in a new window or frame, as opposed to one that already exists when the link is clicked, sometimes the result is a blank page.

(2) The src= links in framesets do not seem to work as consistently as href= links when highlighting hits.

5. Check your site with different browser versions

Some problems only affect specific browser versions. For example, pragma no-cache (mentioned in step 3 above) prevents hit highlighting from working in Netscape 4.72 but not in Internet Explorer 5.5.


Troubleshooting PDF indexing (dtSearch Engine)

dtSearch Home

Downloads

Quick Start

Troubleshooting PDF indexing (dtSearch Engine)

Last Reviewed: January 7, 2001

Article: DTS0151

Applies to: dtSearch Engine 6

This article covers problems that developers may encounter indexing PDF files with the dtSearch Engine. For information on indexing PDF files with the dtSearch Desktop program, see Troubleshooting PDF Indexing.

Symptoms of PDF indexing problems

1. An error code is returned when your application attempts to index a PDF file

2. Garbage data appears in the word list in an index

3. Words that appear to be in the PDF files are not searchable

4. Hit highlighting appears random

5. Only English text appears to be indexed correctly

Potential Causes

1. The PDF files contain LZW compression, and your application is not licensed to use LZW

2. The PDF files are locked with a security password

3. The PDF files do not contain any text

4. The PDF files contain text but have no encoding information

Troubleshooting Steps

Check whether the PDF files have LZW compression

PDF files created with Adobe Acrobat 3 or earlier may contain a type of compression called "LZW." The LZW algorithm is patented by Unisys, and dtSearch Corp.'s license agreement with Unisys permits us to enable LZW decompression in the dtSearch Engine only for developers who also have a license with Unisys. The dtSearch Desktop program does not have this restriction and can index and search LZW-compressed PDF files. Please see the "Indexing and Searching Adobe Acrobat (PDF) Files" topic in the dtSearch Engine help file for information on the Unisys LZW patent.

To tell if a PDF file has LZW compression, you can open the PDF file in Notepad or any other text editor (PDF is a text-based format). In Notepad, search for the string LZWDecode. If you find LZWDecode in the PDF file, it contains LZW compression. In this case, to index the file you will either need to use the dtSearch Desktop program or get a license from Unisys to index the file with the dtSearch Engine.

Check whether the PDF file has a security password

If a PDF file has a security password, dtSearch may not be able to open it to extract the text for indexing. To see if a PDF file has a security password, open it in Adobe Reader and click File|Document Info|Security. A dialog box will appear that will tell you if the file has a password. For information on indexing PDF files that have security, please see "Security passwords on PDF files."

Check whether the PDF file contains text

Some PDF files are nothing more than a PDF wrapper around a TIFF image, with no text in the file. To see if a PDF file contains text,

1. Open the file in Adobe Reader 4

2. Select the Text Select Tool (press "V" or click on the [image] icon in the toolbar)

3. Try to select some text.

Check whether the PDF file contains valid encoding information

Some PDF files contain text but use an encoding that is meaningless outside of the PDF file. For each character, the PDF file contains embedded font information that describes how to draw the PDF file, but the characters do not correspond to an encoding that can be used to extract text from the file. As a result, the PDF file looks like a normal document but there is no meaningful text in the file. For more information, see the "PDF" section in Unicode support in dtSearch 6.

To see whether a PDF has valid encoding information,

1. Open the file in Adobe Reader 4

2. Select the Text Select Tool (press "V" or click on the [image] icon in the toolbar)

3. Select a block of text

4. Click Edit|Copy

5. Open Notepad, Microsoft Word, or another program that can accept pasted text.

6. Click Edit|Paste

If you see what looks like random letters instead of the text you copied from the PDF file, the PDF file lacks encoding information.


Troubleshooting Synonym and Stemming Searches

dtSearch Home

Downloads

Quick Start

Troubleshooting Synonym and Stemming Searches

Last Reviewed: January 13, 2003

Article: DTS0190

Applies to: dtSearch Engine 6

Synonyms

dtSearch provides three kinds of synonym expansion:

(1) "User" synonyms are based on the user thesaurus, thesaur.xml, which is an XML file with a table of synonym sets. To create a set of User synonyms, run dtSearch Desktop and click Options > Preferences > User Thesaurus. After you create some synonym groups and save them, the data will be stored in thesaur.xml in your dtSearch UserData folder.

(2) "WordNet Synonyms" are synonyms defined in the WordNet thesaurus, which is included with dtSearch.

(3) "WordNet Related Words" are related words (antonyms, sub-categories, etc.) defined in the WordNet thesaurus.

Symptom: Synonym expansion does not work

(1) Check that you are setting the synonym flags correctly. To enable synonym searching the dtsSearchSynonyms flag must be set, AND one or more of the following additional flags must be set to indicate the type of synonym expansion to perform: dtsSearchUserSynonyms, dtsSearchWordNetSynonyms, or dtsSearchWordNetRelated.

(2) Check that dtSearch is finding the thesaurus that corresponds to the type of synonym searching that you are using.

dtSearch gets the user thesaurus filename from Options object's UserThesaurusFile property. This should be the full path and filename of the thesaur.xml file with your user thesaurus list.

The WordNet thesaurus must be in a folder named WordNet that is a subfolder of either the HomeDir or the PrivateDir. A debug log of the search will tell you where dtSearch is looking for the thesaurus files and what settings are being used as the HomeDir, PrivateDir, and UserThesaurusFile (look for <dtsOptions> in the log).

Symptom: Incorrect synonym expansion

(1) User Thesaurus: Check that the synonym rules are set up correctly. A common error is to include a word in the name of a synonym group, but not in the list of synonyms. For example, this is an item from a thesaur.xml file:

<Item>

<Name>"Example"</Name>

<Synonyms>"Sample" "examplar" "model" "case in point"</Synonyms>

</Item>

For this item, "Example" would not find any synonyms, because it is not one of the words listed in the <Synonyms> section. Another common error is to omit the quotation marks around phrases.

(2) WordNet Thesaurus: If you see puzzling results from a synonym search, start dtSearch Desktop and click Search > Thesaurus, then enter the search word that produced the puzzling results. The Thesaurus dialog box will list all of the meanings of the word, along with the synonyms associated with each. For example, "find" as a synonym for "rule" seems incorrect, but in fact it is a synonym of the one of the verb definitions of "rule" (to "rule" in a judicial case, as in "The court finds the defendant guilty as charged").

(3) Combined search options. Search features such as stemming that apply to a search also apply to synonym lookup. This means, for example, that a stemming search for "ruling" will find "rule" and also "find" (see above). If you combine phonic searching with synonym searching, you will get back all synonyms of words that sound like the search term, which is likely to be an absurdly broad set of results.

Stemming

Stemming rules are stored in a file named stemming.dat, which dtSearch looks for in the HomeDir or PrivateDir. You can also specify the location of the stemming rules file in Options.StemmingRulesFile.

Symptom: Stemming does not work

(1) Check that you are setting the dtsSearchStemming flag in your SearchJob. In ASP applications, a mis-referenced constant value is sometimes replaced with zero, with no error message, so even though you have dtsSearchStemming in ASP code, if the constant is not referenced correctly, the SearchJob may not have stemming enabled. A debug log will tell you what search flags are actually being passed to the dtSearch Engine.

(2) Check that dtSearch is finding the stemming rules. To do this, generate a debug log and look in the log stemming.dat, to see where dtSearch is trying to read the stemming rules and whether the file can be accessed.


How to combine field and attribute searches in XML

dtSearch Home

Downloads

Quick Start

How to combine field and attribute searches in XML

Last Reviewed: August 30, 2004

Article: DTS0195

Applies to: dtSearch Engine 6

dtSearch indexes attributes in XML files as if they were fields. For example:

<name first="John" last="Smith" />

This is indexed as it if were:

<name>

<first>John</first>

<last>Smith</last>

</name>

In some cases, though, the intent of an attribute is to modify the field name, not to provide a value. Example:

<name type="first">John</name>

<name type="last">Smith</name>

The default treatment of XML attributes in dtSearch makes it difficult to search for a first name that was stored with this use of attributes. (A search for "(/name/type contains first) and (/name contains John)" will match any document that contains a "first" value in the "/name/type" field and also "John" in a /name field. However, these may be different /name fields in the same document.) A simple transformation can be performed on the XML that will cause the fields to be indexed in a way that permits searching on a first name. To do this, replace the attribute with a generated field that combines the attribute and named value, like this:

<name>

<TypeFirst>John</type_first>

<TypeLast>Smith</type_last>

</name>

With this field structure, a search can be limited to the first name field as follows:

/name/TypeFirst contains John

See also:

How to combine field and attribute searches in XML

Field searching


Email Conversion Tools

dtSearch Home

Downloads

Quick Start

Email Conversion Tools

Last Reviewed: December 24, 2004

Article: DTS0169

This article collects information on tools for converting message archives in proprietary email formats into standard or documented formats that dtSearch supports. This list is provided for informational purposes only. dtSearch Corp. cannot warrant or guarantee or provide support for these products. Please send suggestions for other products to add to the list (or feedback on any of the products listed) to tech@dtsearch.com.

To use message archives with dtSearch, the recommended formats are MSG, MBOX, Eudora or HTML. To ensure that MBOX archives are detected correctly, set up a file type definition in Options > Preferences > File Types to specify which files are to be indexed as MBOX archives.

dtSearch Desktop 6.5 includes a command-line program, mapitool, which can convert PST files to MSG. For mapitool documentation, see C:\Program Files\dtSearch\bin\mapitool.html.

Tools

Eudora

http://www.eudora.com

Eudora has an "Import" function that can import messages from Microsoft Outlook or Outlook Express into Eudora's message format.

Mozilla

http://www.mozilla.org

Like Eudora, Mozilla's email program can import messages from Microsoft Outlook. Mozilla stores emails in the MBOX format, which dtSearch can index and search directly.

MessageSave

http://www.techhit.com/messagesave/

Converts Outlook messages as txt or msg files.

WN MailKeeper

http://www.wickett.net/

Converts PST or Exchange Mailbox to individual MSG files, which dtSearch can index directly.

ComAxis

http://www.comaxis.com/

Conversion tools for a variety of email formats

SMT Archiver

Archives PST data in standard formats such as RTF, plain text, etc.

Transend

http://www.transend.com

Transcend Migrator for conversion of Exchange/Outlook, Lotus Notes, GroupWise, Netscape, IMAP, Eudora, Outlook Express, AOL, cc:Mail, Notwork, MHS, and others.

ol2mbox

http://sourceforge.net/projects/ol2mbox

Open-source program that converts PST archives to MBOX format

ForMorph Message Converter ($20)

http://www.fkeeps.com/fmorph/fmindex.htm

Conversion tools for CompuServe and AOL archives.

Information Sites

Information on message archive conversion tools

http://www.interguru.com/MailInformation.htm

Information on message archive and address book conversion.

http://www.emailman.com/conversion/index.html


How to get the Windows Installer

dtSearch Home

Downloads

Quick Start

How to get the Windows Installer

Last Reviewed: October 24, 2001

Article: DTS0149

Applies to: dtSearch 6

dtSearch 6 uses a new Microsoft component, the Windows Installer service, for installation. If you have Windows 2000 or Microsoft Office 2000, you already have the Windows Installer. If not, the dtSearch setup program should install it for you automatically. You can also get the Windows Installer software directly from Microsoft using the links below.

Windows Installer Downloads

Windows Installer for Windows 95, 98, or ME

http://www.microsoft.com/downloads/release.asp?ReleaseID=32831

Windows Installer for Windows NT or Windows 2000

http://www.microsoft.com/downloads/release.asp?ReleaseID=32832

The entire Windows Installer developer SDK is available here:

http://msdn.microsoft.com/downloads/default.asp?URL=/code/sample.asp?url=/msdn-files/027/001/457/msdncompositedoc.xml

Troubleshooting

Please see this Microsoft Knowledgebase article for troubleshooting suggestions if the Windows Installer setup does not work correctly.

Q260404 HOWTO: Troubleshoot Windows Installer Engine Problems

http://support.microsoft.com/support/kb/articles/Q260/4/04.asp


How to open CHM (HTML Help) files

dtSearch Home

Downloads

Quick Start

How to open CHM (HTML Help) files

Last Reviewed: March 13, 2000

Article: DTS0148

Applies to: dtSearch 6

dtSearch 6 uses a new kind of help file, HTML Help, instead of the older WinHelp format. HTML Help files have the .CHM extension. If you have Windows 2000 or Microsoft Office 2000, you probably already have the HTML Help viewer.

If you are unable to open the dtSearch 6 help files or other .CHM files, you can get the latest HTML Help viewer from the dtSearch web site or from the Microsoft web site. The HTML Help viewer is a free download and is about 700k in size.

Note: If you have Windows 2000, you should use the version of HTML Help included with Windows 2000 and should not use the link below. The hhupd.exe file is not designed to work with Windows 2000.

To get the HTML Help viewer from the Microsoft web site, click here:

http://msdn.microsoft.com/library/tools/htmlhelp/wkshp/download.htm

After downloading, open the hhupd.exe file to install the latest HTML Help viewer.


dtSearch Text Search Engine -- Support [0315]

dtSearch Home

Downloads

Quick Start

dtSearch 6 Information

Upgrade Information

How to order

dtSearch Desktop Quick Start

dtSearch Web Quick Start

dtSearch Desktop/dtSearch Network Manual (PDF)

dtSearch Desktop/dtSearch Network Help (CHM format)

dtSearch Web Manual (PDF)

dtSearch Web Help (CHM format)

All products

dtSearch system and hardware requirements

How to change the size of the "title" field

How to make blocks of text non-searchable

How to prevent filenames and document properties from appearing in documents

How to get field data in search results

How to index a web site with the dtSearch Spider

How to make '+', '#', or other characters searchable

How to segment long text, HTML, or XML files during indexing

How to use dtSearch or dtSearch Web with OCR

How to send large files to dtSearch Technical Support

Hyphenation Options

Index Size

Installation Problems

International Language Support

Security passwords on PDF files

'Too many words retrieved in index' message

Unicode support in dtSearch 6

What file formats does dtSearch support?

Articles

The Art of the Text Query (PDF article)

Distributed Searching: Evolution to XML (PDF article)

Web-Based Document Management (PDF article)

Electronic Publishing (PDF article)

Crossing the Full-Text Search / Fielded Data Divide (PDF article)

Searching Beyond the Tower of Babel: Unicode and Text Retrieval (PDF article)

dtSearch Desktop/dtSearch Network

Accessibility -- dtSearch Desktop

Document is not found after a search

Does dtSearch have any size limitations?

dtSearch 6 Quick Start

dtSearch 6 Upgrade Information

dtSearch 5 Quick Start

dtSearch system requirements

How to add a list of files, folders, or web sites to an index

How to change dtSearch file locations

How to change the hit highlighting color in PDF files

How to customize the way dtSearch prints search results

How to display PDF and HTML titles in search results

How to exclude folders from an index

How to index databases with dtSearch

How to index Outlook messages with dtSearch

How to link images and documents

How to make common words like 'the' searchable

How to run multiple copies of dtSearch on the same machine

How to search for connector words like 'and' and 'not'

Launching files from dtSearch

'ODBC Resource DLL Is a Different Version' Message

PDF files open slowly with Acrobat 6 installed

Phrase or boolean search is not working

Troubleshooting -- HTML files are indexed as text

Troubleshooting database indexing

Troubleshooting hit highlighting problems

Troubleshooting missing DLL problems

Troubleshooting missing indexes

Troubleshooting PDF hit highlighting problems in dtSearch Desktop

Troubleshooting PDF indexing

Troubleshooting scheduled index updates

'Unable to create a directory' message installing dtSearch

dtSearch Web

dtSearch Web Quick Start

dtSearch Web system requirements and performance

dtSearch Web and ASP Demos

dtSearch Web document access problems

How to add custom fields to dtSearch Web search results

How to display links to the original files in dtSearch Web

How to install dtSearch Web on Windows 2003 Server

How to use dtSearch Web with dynamically-generated content

How does dtSearch Web handle security?

How to use dtSearch Web with ODBC

How to use dtSearch Web with PDF files

PDF file appears as blank page in dtSearch Web

PDF files appear without hit highlighting in dtSearch Web

Troubleshooting hit highlighting problems

Troubleshooting -- dtSearch Web search results formatting

Troubleshooting -- dtSearch Web server error codes

Troubleshooting -- 'Encrypted paths' error message from dtSearch Web

What web servers does dtSearch Web 5.x work with?

dtSearch Publish

How to run dtSearch from a CD or DVD with a web browser interface

dtSearch Engine

.NET information

Automation errors accessing dtSearch Engine methods or properties

CreateObject errors creating dtSearchEngine.Server

dtSearch Web and ASP Demos

Diagnostic tools for .NET developers

Diagnostic tools for C++ developers

Diagnostic tools for Java developers

Diagnostic tools for Visual Basic and ASP developers

dtSearch Engine Build Tips for Visual C++ Programmers

How to add fields to documents during indexing

How to display retrieved XML documents with XSL formatting

How to generate a synopsis to display in search results

How to index databases with the dtSearch Engine

How to integrate the dtSearch Engine with a language analyzer

How to write dtSearch ASP applications without using Session

Implementing document classification

Optimizing search performance with the dtSearch Engine

Sorting search results

Troubleshooting -- ConvertPath returns a blank string

Troubleshooting -- System.IO.FileNotFoundException accessing dtSearchNetApi.dll

Troubleshooting -- Using the dtSearch Engine with Access databases

Troubleshooting PDF hit highlighting (dtSearch Engine)

Troubleshooting PDF indexing (dtSearch Engine)

Troubleshooting synonym and stemming searches

Other

dtSearch Features

Email conversion tools

How to get the Windows Installer

How to open CHM (HTML Help) files

dtSearch Desktop/Network Training -- RCC Consulting (PowerPoint)

Click here to download all dtSearch Technical Support Articles in a searchable HTMLHelp file. (Last update: January 14, 2005)

Please send any questions or comments to: tech@dtsearch.com. Thank you!

Copyright © 1995-2004 dtSearch Corp.


How to put documents and indexes on a CD

dtSearch Home

Downloads

Quick Start

How to put documents and indexes on a CD

Last Reviewed: May 21, 2001

Article: DTS0137

Applies to: dtSearch 5.x or dtSearch 6.x

[Please see "How to run dtSearch from a CD with a web browser interface" for information on how to do this with dtSearch 6 using a browser-based interface.]

Question

How can I put the dtSearch program, my documents, and my indexes on a CD so that I can search directly from the CD?

Answer

Create a CD master folder on your hard disk with the dtSearch program, documents, and indexes all under the same folder. For example, you could set up the CD master folder like this:

c:\cd_master

c:\cd_master\docs

c:\cd_master\index

c:\cd_master\dtsearch

To do this,

(1) Copy your documents to c:\cd_master\docs

(2) Install dtSearch in c:\cd_master\dtsearch

(3) Run dtSearch from this folder and use Index|Create Advanced to build an index in c:\cd_master\index

If you transfer the cd_master folder to a CD and create an autorun.inf file pointing to the dtSearch program, dtSearch will start automatically when the CD is inserted and can be used to search the index on the CD.

Licensing

If the CDs will be distributed to people who do not have their own copy of dtSearch, you will need to purchase an Electronic Publishers' Toolkit license to create distributable search-only dtSearch CDs.


Forensic Searching Features in dtSearch

dtSearch Home

Downloads

Quick Start

Forensic features in dtSearch Desktop

Last Reviewed: December 2, 2002

Article: DTS0181

Applies to: dtSearch Desktop

Quick language check.

Option to index hidden HTML content
Forensic indexer

Non-text file detection

Extext.exe text extraction utility

Search for List of Words

The Search for List of Words dialog box provides a way to search for a long list of words, and create a list of matching files, in a single step. To search for a list of words,

1. Create the word list in any of the file formats that dtSearch supports, such as Microsoft Word, Unicode text, WordPerfect, etc. For non-English text, Microsoft Word 97 or later or Unicode text are the best formats.

2. Click Search > Search for List of Words... or press F9. To search without an index, click Search > Unindexed Search for List..., or press Ctrl+F9.

3. Enter the name of the file with the list of words. To browse for the file, click the ... button.

4. Under Search type, select the option that describes the type of search request in the text file.

One word or phrase per line The text file contains a series of lines, each of which contains a single word or phrase.
Natural language search Treat the entire contents of the file as a single natural language search request.
Boolean (and, or not...) Treat the entire contents of the file as a single boolean search request.

5. Under Search features, select the search options that you want to use in the search (stemming, fuzzy searching, etc.)

6. Select the type of search results that you want from the search.

Write list of filenames to a text file Check this box, and enter a filename, if you want a plain text list of filenames listing all of the documents matching the search request. To export the list to Excel, leave the file type as "Tab separated (CSV) for Excel", which is the default.
Open search results in dtSearch Check this box if you want to see the search results list in dtSearch Desktop, just as you would after a search using the Search dialog box.

7. Click Search to start the search.

8. To open the list of files in Excel, click the Open File button. Excel will open with the list of retrieved file. In Excel, click Format > AutoFormat to have Excel set up the column widths correctly.

If the search finds a very large number of files, the list of files in the text file will be complete, and the search results in dtSearch Desktop will display the best-matching 5,000 documents.

If some of the words in the list are not English words, the word list file should be in a format that is able to store Unicode text, such as Microsoft Word 97 or later, Microsoft Excel 97 or later, or the Unicode text format.

Text Extraction Utility (Extext.exe)

Extext.exe is a tool for extracting text from large binary files, such as undeleted data recovered from a hard disk. It converts the input data to a series of small Unicode text or HTML files containing extracted text. Extext assumes that the input data consists of fragments of files rather than a single complete document and so looks for sequences of data that appear to be text, Unicode text, or UTF-8 text.

To use Extext, click Start > Programs > dtSearch > Tools > Extext. Click the Help button for documentation.

Forensic Indexer Utility (Forix.exe)

Forix.exe is a tool for indexing larger volumes of data than will fit in a single index. It creates an index collection with as many indexes are needed to index all of the data selected. After indexing is complete, the indexes will be searchable in dtSearch. Forix automatically invokes the dtSearch Indexer to create the indexes, and it can be stopped and restarted so that the whole collection does not have to be indexed at once. Each time Forix runs, it remembers the last index collection created and the amount of data indexed so far. (It is called a "Forensic Indexer" because it is intended for creating one-time indexes of very large collections of data, a situation that most often arises in forensic applications.)

To use Forix, click Start > Programs > dtSearch > Tools > Forix - Forensic Indexer. Click the Help button for documentation.

Non-text file recognition

Multimedia files such as images, sound, or video files contain no text and are normally excluded from indexing. In forensic applications, however, filename filters cannot be used because of the possibility that misleading filenames were used (such as "something.jpg" for a Word document). To address this problem, dtSearch 6.06 can recognize multimedia files by their contents rather than just relying on the extension. This makes it possible to index without excluding multimedia files, without bloating the index with random binary data.

Quick Language Check

To quickly check whether an index contains any Russian, Arabic, Hebrew, or Greek text,

1. Start dtSearch Desktop

2. In the Search dialog box, click Browse Words

3. In the space for the word to match, enter one of the following special terms: )arabic, )hebrew, )russian, )greek

4. The dialog box will list all words in the specified language that were found in the index.

The language check is done using the Unicode character ranges associated with each language.

Indexing Hidden Html Content

To set up dtSearch Desktop to index HTML comments, scripts, links, and styles, which are normally not searchable or visible,

1. Click Options > Preferences > Indexing Options

2. Check this option setting: Index HTML scripts, styles, links and comments.


Home and Private Directories

The dtSearch option settings include two directories: the "Home" and the "Private" directories.

The "Home" directory is the location of settings files and executable components such as file parsers. This directory can be a shared network directory and is usually where the dtSearch Engine DLL, dten600.dll, is located.

The "Private" directory is the location where temporary files can be written and where per-user options settings are stored.

The two directories can be the same as long as only one user at a time is using the directory.

When the full path of a settings file, such as the user thesaurus (thesaur.xml) or the stemming rules file (stemming.dat) is not provided in the Options object, dtSearch will look for these files first in the Private directory and then in the Home directory.


Placeholder

dtSearch Home

Downloads

Quick Start

This topic is created so that Robohelp's automatically-generated default page will mess up this page rather than the real default page.