The term rootkit is used to describe the mechanisms and techniques whereby malware, including viruses, spyware, and trojans, attempt to hide their presence from spyware blockers, antivirus, and system management utilities. There are several rootkit classifications depending on whether the malware survives reboot and whether it executes in user mode or kernel mode.
A persistent rootkit is one associated with malware that activates each time the system boots. Because such malware contain code that must be executed automatically each system start or when a user logs in, they must store code in a persistent store, such as the Registry or file system, and configure a method by which the code executes without user intervention.
Memory-based rootkits are malware that has no persistent code and therefore does not survive a reboot.
There are many methods by which rootkits attempt to evade detection. For example, a user-mode rootkit might intercept all calls to the Windows FindFirstFile/FindNextFile APIs, which are used by file system exploration utilities, including Explorer and the command prompt to enumerate the contents of file system directories. When an application performs a directory listing that would otherwise return results that contain entries identifying the files associated with the rootkit, the rootkit intercepts and modifies the output to remove the entries.
The Windows native API serves as the interface between user-mode clients and kernel-mode services and more sophisticated user-mode rootkits intercept file system, Registry, and process enumeration functions of the Native API. This prevents their detection by scanners that compare the results of a Windows API enumeration with that returned by a native API enumeration.
Kernel-mode rootkits can be even more powerful since, not only can they intercept the native API in kernel-mode, but they can also directly manipulate kernel-mode data structures. A common technique for hiding the presence of a malware process is to remove the process from the kernel's list of active processes. Since process management APIs rely on the contents of the list, the malware process will not display in process management tools like Task Manager or Process Explorer.
You can find more information on rootkits, including rootkit libraries and source code, at www.rootkit.com.