www.delorie.com/djgpp/v2faq/faq115.html

search

| Previous | Next | Up | Top |

Q: I tried to improve DJGPP I/O throughput by defining a 64KB-large buffer for buffered I/O with a call to setvbuf, but that had no effect. Why is that?

Q: It is obvious that disk-bound programs compiled with DJGPP will run awfully slow, since FAT is such a lousy filesystem!

Doing I/O from protected-mode programs requires that low-level library functions move the data between the extended memory and low memory under the 1 MByte mark, where real-mode DOS can get at it. That area in the low memory is called the transfer buffer(Note: Here's a more detailed explanation. DOS cannot access memory above 1MB mark, where your DJGPP program lives, since real-mode addresses are 20-bit wide, which covers only the first megabyte. So, each time a DJGPP program needs to call a DOS function (or any other real-mode service, like some BIOS interrupt) and needs to pass data to or from that service, we must use some buffer in conventional memory to communicate. The transfer buffer is a block of conventional memory that the DJGPP startup code allocates for this purpose. When a real-mode service is called, the data that needs to be submitted to it is copied to the transfer buffer, and the address of the transfer buffer is passed to the real-mode service. If the service returns some data (e.g., if you want to read a portion of a file), data is copied from the transfer buffer when the service returns.

The transfer buffer primarily exists for library functions, but it can also be used by an application, if it needs to invoke real-mode services.). This data shuffling means that some I/O speed degradation is inevitable in any protected-mode program which runs on top of DOS (including, for example, Windows programs when Windows 3.X is set to 386-Enhanced mode). By default, DJGPP moves data in chunks of 16 KB, so defining a buffer larger than that won't gain anything. The size of the transfer buffer is customizable up to a maximum of 64 KB(Note: Actually, the maximum possible value is FEF0h, or 65254 in decimal, because the transfer buffer is created by the startup code by resizing the PSP memory block. Since the resized block needs to leave 256 bytes for the PSP, and needs to be aligned on a 16-byte boundary, you cannot have the entire 65535 bytes for the transfer buffer. If you invoke stubedit with a bufsize=64k parameter, what you actually get is a 2KB buffer, since the combined size of the PSP and the transfer buffer will wrap around in a 16-bit variable when the startup code computes it. The version of stubedit which will come with DJGPP v2.02 will explicitly warn you about this case and will reset any value that is too large to the maximum allowed size of FE00h (65024 decimal) bytes--this is less than FEF0h because the latter is not aligned on the 512-byte DOS sector size, which could slow down disk I/O.), so if your program really reads a lot of large files, you might be better off enlarging it (with the STUBEDIT program).

Some people think that FAT is such a lousy filesystem, that programs which do a lot of disk I/O must run terribly slow when compiled with DJGPP. This is a common misconception. The speed of disk I/O is determined primarily by how efficient is the code in the operating system kernel that handles the filesystem, and the device drivers for the I/O-related devices like the hard disk, not by the disk layout. It is true that DOS and BIOS don't implement I/O too efficiently (they use too many tight loops waiting for low-level I/O to complete), but a large disk cache can help them tremendously. In addition, Windows 9X bypasses DOS and BIOS I/O code entirely, and uses much more efficient protected-mode code instead. Experience shows that DJGPP programs on plain DOS systems with a large (8MB and up) disk cache installed run about 30% slower than a Linux system on the same machine; and Windows 9X will run the same programs at roughly the same speed as Linux. If you get much slower performance on DOS/Windows, chances are that your system is not configured optimally.

Some programs which only copy data between two files might gain significantly if you write your custom low-level I/O functions that avoid moving data to extended memory (only to move them back to the transfer buffer). However, these cases are rare.

You can help support this site by visiting the advertisers that sponsor it! (only once each, though)