HIGH UTILIZATION Network performance is a key concern for any network administrator. Those concerned with performance are always looking for ways to measure and monitor their network and are constantly looking for ways to improve performance. Novell has always been very concerned about NetWare's performance. This document will discuss high utilization issues that have been reported since the shipment of NetWare 4.x and how to get the problem resolved. The current release of the NetWare 4.x product is 4.10. NetWare 4.0, 4.01 and 4.02 are all outdated products; therefore, this document will focus on 4.10 issues. NetWare 4.10 was a free upgrade to those who registered 4.0x and resolved many 4.0x utilization problems. All development and enhancements are being done to the 4.10 release. Administrators who do not understand the "Processor Utilization" statistic in the MONITOR.NLM, believe that the higher the percentage, the worse NetWare's performance will be. This is entirely false. Some processes make efficient use of the processor and as a result may cause 100% utilization. This type of utilization is entirely appropriate. Most of the time when utilization moves up to 100% it is totally normal, it means the process is using the processor efficiently. It may stay at 100% for a couple of minutes. It is NOT normal when the utilization is at 100% for a couple of minutes and connections are dropped or server performance is severely impaired. This is what will be referenced as high utilization problems in this document. A note about MONITOR utilization. The Monitor utilization is not an entirely accurate number. Some server processes call a NetWare function called CyieldWithDelay or CyieldUntilIdle. If there is a thread spinning on one of these functions, the server will appear to have high utilization. Utilization will then be inaccurate. Don't panic when utilization goes to 100 percent for a few seconds and bounces down. This is normal for all 4.10 servers. If you are experiencing a high utilization problem and your server hangs or drops connections, use the following checklist to resolve the problem: REQUIRED PATCHES AND FIXES FOR UTILIZATION Operating System - Apply the patches from 410PTx.EXE. These patches fix problems that resulted in high utilization. These patches also fix a variety of additional problems. Loading all of the patches will reduce your chances of having future problems. Verify that SET UPGRADE LOW PRIORITY THREADS is set to OFF. If it is ON it will contribute to any utilization problems the server may be having. Directory Services - Update DS to version 4.89a or later. Novell Support has seen issues with DS causing high utilization and 4.89a has solved all these issues. Efficient tree design is essential to avoid utilization problems. The size and number of partition replicas can cause utilization problems if not managed properly. DS needs to keep synchronization among all servers in the replica ring. The more replicas there are of any partition, the more traffic will be on the wire. Novell recommends having at least three replicas of each partition in the tree. This provides fault tolerance and allows for DS recovery if a database were to become corrupt. Compression - It is essential to have the OS patches loaded when using compression. Compression takes CPU cycles to compress and decompress files. The default set parameters for compression take this into consideration. Compression is designed to take place after midnight when most servers have little or no traffic. Also, the DAYS UNTOUCHED BEFORE COMPRESSION set parameter is designed to make sure frequently used files are not compressed. Any adjustments to the default compression SET parameters may severely impact the server's performance. Users with disk restrictions may try to flag their home directory to IC (immediate compress) to save disk space. Flagging directories to IC will affect server performance. Normally, compression is a low priority thread, which means it only compresses files when the server is idle. When the IC flag is set, compression is bumped up to a regular priority and will not wait for idle time. Setting DELETED FILES COMPRESSION OPTION = 2 will cause the immediate compression of files that have been deleted. This will cause high utilization because the processor is immediately compressing files upon their deletion. Compression needs to be monitored carefully to avoid utilization problems. For additional information refer to TID1005736, "Compression and High Utilization." Suballocation - Suballocation is another disk space saving technique new to NetWare 4.x. It is very important to load the patches when suballocation is enabled. Suballocation does not have any SET parameters to adjust. Everything is done automatically. It is very important to monitor the disk space to avoid suballocation problems. Novell Engineering recommends keeping 10-20 percent of the volume space free to avoid suballocation problems. It is also important to keep more than 1000 "free blocks" on each volume with suballocation enabled. Suballocation uses these free blocks to free up additional disk space. SERVMAN.NLM can be used to monitor the volume statistics and check for "free blocks". If free blocks are low, run a PURGE /ALL from the root of the volume to free the "freeable limbo blocks" and move them to the "free blocks" pool. To avoid doing this often, set the P (PURGE) attribute on directories that create a large amount of temporary files. The P attribute does not flow down the file system. This needs to be taken into consideration when setting the P attribute. Also, setting IMMEDIATE PURGE OF DELETED FILES = ON at the server console will avoid the purgeable files taking all the "free blocks". Like compression, volumes with suballocation enabled need to be monitored carefully to avoid utilization problems. For additional information refer to TID1005436, "High Utilization and Suballocation." Packet Burst - Novell support has seen high utilization problems caused by packet burst from both NetWare and Microsoft requester clients. Most of these problems have been solved with an OS patch or a new requester. Load the 410 OS patches and get the new FIO.VLM which fixes the problem with the Novell requester. The file is VLMFX3.EXE in the NSD area of CompuServe or the Internet. AppleTalk - There was also a problem with the ATXPR.NLM causing high utilization. The file 41MAC1.EXE contains the new version of ATXPR.NLM. The file 41MAC1.EXE is found in the NOVLIB area LIB 6. Printing - Having a large amount of printers on a server can affect network performance. This number depends on the type of printing that is being done. Printing CAD/CAM designs take much more processor time than text documents. Having more than 40 printers can degrade server performance. It is recommended, if you are concerned about utilization problems, that you set your print devices to do the processing instead of the server. This will slow down printer output but will relieve the utilization on the file server. NOVELL HAS FIXED EVERY REPORTED HIGH UTILIZATION PROBLEM RELATED TO THE OS OR DS. If you have completed the previous checklist and are still having problems, look at the following troubleshooting checklist to try to determine what is causing the problem. THESE ARE TROUBLESHOOTING TECHNIQUES ONLY. THEY ARE NOT NOVELL'S FINAL SOLUTIONS. TROUBLESHOOTING UTILIZATION Compression - Eliminate compression as a possible problem by setting ENABLE FILE COMPRESSION=OFF. This will cause files to be queued for compression but the files will not be compressed. However, accessing compressed files causes them to be decompressed. This will eliminate compression as the cause of high utilization. Packet Burst - Novell technical support has a module called PBRSTOFF.NLM that will disable Packet Burst from the server. The module needs to be loaded when the server comes up. Only connections that are established after the module is loaded will have packet burst disabled. This will isolate utilization problems related to packet burst. Other NLM based products - When utilization problems occur, eliminate as many variables as possible. Remove any products until the server is in "bare bones" form. This helps determine if it is an OS issue or a problem with a loaded module. Novell is only able to determine an OS problem when it is at it's "bare bones" form. Drivers - In troubleshooting Utilization problems it may be necessary to eliminate certain LAN or Disk drivers as being a possible cause. As part of normal troubleshooting procedures, you may be asked by Novell Technical Support to replace your LAN card with a different manufacturer's LAN card to eliminate the card and the driver as a possible problem. Novell Engineering is very interested and very dedicated to resolving issues with NetWare 4.1, especially high utilization problems. At this point in the troubleshooting process it will be necessary to get a core dump from the server experiencing the high utilization. For additional information on core dumps refer to Novell Application Notes, February 1995, "Resolving Critical Server Issues." NOVELL RECOMMENDATIONS 1. Load ALL the patches from 410PTx.EXE. The OS patches fix known problems with suballocation and other NetWare functions. These patches have been tested and are recommended by Novell Engineering. 2. Verify that SET UPGRADE LOW PRIORITY THREADS is set to OFF. Setting this parameter to ON allows low priority threads to get on the regular run queue. This may contribute to utilization problems. 3. Keep 10%-20% of the volume space free. Disk space management is essential to avoid problems with compression and suballocation. Make sure there is always 10%-20% available disk space on all volumes. This can be monitored my SERVMAN.NLM, NWADMIN.EXE or NDIR /VOL. If the file server is running low on disk space, either delete unneeded files or add additional disk space to the volume. 4. Maintain a minimum of 1000 free blocks on each NetWare volume. Suballocation uses free blocks to perform its function. When free blocks are low suballocation could go into "aggressive" mode, lock the volume and cause high utilization. Maintaining more than 1000 free blocks will avoid this problem in most cases. If there are not at least 1000 free blocks on the volume, run a PURGE /ALL from the root of the volume. This will free the "freeable limbo blocks" and move them back to "free blocks." 5. Set PURGE flag on all directories that have a large amount of temporary files created. Every temporary file that is created will be put on the "deleted file list." These files are kept on the disk until a PURGE is run. In order to avoid disk space and free block problems, set the PURGE flag on specific directories that create large numbers of temporary files. Also setting IMMEDIATE PURGE OF DELETED FILES=ON, after running a PURGE /All, will avoid this problem. High processor utilization alone should not be thought of as a bad thing. When changing SET parameters, make sure you know what effect it will have on processor utilization. High utilization problems may be very specific. If you have completed the above checklists and are still having utilization problems, contact your Novell Reseller or Novell Technical Support. Novell cannot fix problems they do not know about. Rich Jardine Product Support Engineer, NetWare OS Support Novell Inc.