|
Hylafax Mailing List Archives
|
[Date Prev][Date Next][Thread Prev][Thread Next]
[Date Index]
[Thread Index]
irqtune
Here is the README file from irqtune. Please forgive me if it's too long.
I include it whole in an attempt to provide enough information to allow
folks to decide if they are interested in it. I am not associated with
irqtune in any way - it's just a program I use.
_________________________________________________________________
IRQTUNE -- A LINUX IRQ PRIORITY OPTIMIZER
Copyright 1996, 1997 by Craig Estey.
irqtune version 0.6
Last updated: Sun Oct 19 16:35:00 PDT 1997
See the Changes section at the bottom of this document.
Table of Contents
irqtune changes the IRQ priority of devices to allow devices that
require high priority and fast service (e.g. serial ports, modems) to
have it.
With irqtune, a 3X speedup of serial/modem throughput is possible.
_________________________________________________________________
WHERE DO I GET IRQTUNE?
irqtune is free software under the terms and conditions of the GNU
Public License. See the file COPYING, included in the distribution,
for details.
* The author is Craig Estey, (cae@best.com).
* This FAQ is available online via http://www.best.com/~cae/irqtune.
This is the most authoritative reference and will always contain
the most up-to-date information.
* The distribution is a gzipped tar archive that contains all
programs, sources, and this FAQ (in both HTML and text versions).
It is available from:
+ http://www.best.com/~cae/irqtune/irqtune.tgz
+ ftp://shell5.ba.best.com/pub/cae/irqtune.tgz
_________________________________________________________________
HOW DO I KNOW IF I NEED IRQTUNE?
You are running Linux on an x86 PC, other architectures to be
implemented later--Sorry.
You probably need irqtune, if you are experiencing any of the
following:
* SLIP/PPP transfers seem slow. For example, using a 28.8 (or
better) modem, the effective throughput is approximately 700
bytes/second instead of the expected 2500 bytes/second.
* A running serial or SLIP/PPP connection is slow, drops data,
hangs, or times out.
* Netscape hangs mysteriously or stalls when trying to access a web
page.
* Equivalent serial/PPP programs under Windoze run much faster than
under Linux.
* Disk accesses seem to interfere with SLIP/PPP.
* Interrupt handlers for specialized, time critical devices don't
get control when they need to.
_________________________________________________________________
WHAT IS ACTUALLY HAPPENING TO CAUSE THESE PROBLEMS?
When the PC boots Linux, the timer is given, by default, the highest
IRQ priority in the system (it's IRQ 0 and thus, priority 0). On a
standard configuration, the serial ports are priority 11 and 12!!!
This means that 10 other devices have higher priority.
Q: SO WHAT DOES IRQ PRIORITY DO?
When multiple devices are in contention to interrupt the CPU, their
priority decides which interrupts will occur in what order.
Q: WHEN DOES THIS CONTENTION OCCUR?
After an arbitrary period of having interrupts disabled (e.g after a
cli), at the point where they're reenabled (sti). This can happen in
several places:
* In an ISR that runs with interrupts locked, it happens in the
epilog, just before attempting to execute the bottom-half.
* The bottom-half itself may do a lock and unlock.
* When a task that enters the kernel to do a system call, the system
call handler may lock and unlock interrupts briefly.
* Almost anywhere in the kernel are brief periods where it does a
cli to lock interrupts and then an sti to unlock them again.
* Under Linux, from the moment the first instruction of the ISR
prolog is executed until an sti is done (This occurs even in
slow interrupts where the prolog does an sti after just a few
instructions).
Q: IF THERE ARE MULTIPLE INTERRUPTS NOW PENDING, WHICH ONE GETS THE SERVICE,
THE SERIAL OR SOME OTHER?
In the default configuration, the serial ISR will usually lose as it's
priority 11.
_________________________________________________________________
HOW DOES IRQTUNE HELP THIS?
irqtune gives priority 0 to whatever device we specify. If we specify
a serial device, irqtune guarantees that the serial ISR gets control
whenever a contention occurs.
_________________________________________________________________
WHY DOES THE SERIAL INTERRUPT SERVICE REQUIRE THE HIGHEST PRIORITY?
Q: WHY DOES THE SERIAL DEVICE MERIT SUCH SPECIAL TREATMENT?
Serial devices are somewhat unique. Even though they have one of the
slowest data rates (relative to a disk), they are the largest consumer
of interrupts and are extremely sensitive to interrupt latency (the
time from when a device yanks the IRQ line until its ISR is executed).
Q: COULD YOU GIVE A CONCRETE EXAMPLE OF THIS?
* For a modem running at 33.6, we can have a peak maximum (with
compression) of 6700 bytes/second. The serial driver programs the
silo to interrupt after 8 bytes, leaving a latency window of 8
bytes. This means that when the serial port yanks its IRQ line, it
can still absorb 8 more characters before its buffer will overflow
and data will be dropped
* In terms of time, this means that the maximum that the serial ISR
may be delayed is (8 / 6700) seconds or 1194 microseconds. It also
means that the serial ISR will require 838 interrupts/second.
* Currently there are 52 devices that install their ISR with the
SA_INTERRUPT option. This means that they wish to run with
interrupts disabled. I have not looked at all the devices, but
just assume for the moment that all will run with interrupts
locked for an arbitrary period.
* Assume we've got 4 of them pending (remember, at higher priority).
Assume that they will lock interrupts individually for 320, 300,
190, and 500 microseconds, respectively. Assume that the serial
ISR also wants an interrupt.
* This means that the serial ISR will have to wait for (320 + 300 +
190 + 500) or 1310us for these other higher priority ISR's to run
to completion. This is greater than the maximum of 1194 us.
* With internal modems, we probably just see start/stop behavior
causing a slowdown. With an external modem, data will be dropped.
This will cause PPP to see a CRC error and request transmission.
Remember, the entire packet must be retransmitted, which means we
just wasted 296-1500 bytes (depending on the selected MRU). With
an MRU of 1500 bytes, we just wasted 23.4% of the bandwidth in a
given second--all for the loss of a single byte!
* Worse yet, consider a 115 Kbps ISDN card that is masquerading as a
serial device (Yes, I know it should be a DMA device). In this
case, the maximum delay the serial ISR can wait is 695 us. And
that's just one ISDN channel. Try adding the other and, well,
golly :-).
Q: IN THIS EXAMPLE, HOW WOULD BOOSTING SERIAL IRQ PRIORITY HELP?
* If the serial IRQ had priority 0, it would get in before the
others. It could also re-interrupt after any individual lockout
window. Thus, the maximum it would be forced to wait would be
largest individual time, not the summation of all of these times.
In this example, this means 500 us. which is now within the 1194
us. maximum for the serial ISR.
* It has been my experience that the serial device must always win
these battles for contention. When serial devices don't get what
they need, when they need it, they slow horribly or drop data
outright.
_________________________________________________________________
ISN'T THAT EXAMPLE VERY UNLIKELY UNDER LINUX?
Unlikely doesn't mean never. There are places were the contention
period must occur, no matter how we program the CPU.
Variations in CPU speed, RAM size, RAM speed, cache size, disk speed,
disk rotational position, number and type of other devices, system
workload, etc, etc etc, all contribute to variations of order and
timing of internal OS events.
Unlikely on one system may mean 1/1000th % chance. On another system,
it may mean happens 50 times/second. Beyond a certain point, it all
comes down to measurement.
Tight, accurate, repeatable measurement is the key to system tuning.
If we can't measure it, we can't tune it. Once we can measure these
things, we can then try various combinations until we achieve our
desired results. It can often be pointless to guess at performance.
In fact, the interrupt disable windows themselves are used to prevent
data corruption caused by various parts of the kernel that try to
update a data structure simultaneously and get corrupted. Many of
these windows are put there in the unlikely case that such corruption
would occur. In this context, unlikely is never considered a reason to
forego interrupt locking. Eliminating a necessary interrupt lock
window could result in kernel panics, RAM corruption, disk data
corruption, etc.
_________________________________________________________________
DOESN'T THIS HURT THE PERFORMANCE OF OTHER DEVICES?
Not really.
In actual practice, most devices don't even notice the difference.
Most other devices (e.g. disks, tape, ethernet) are DMA devices. The
DMA does most of the work, thus greatly reducing their need for
interrupts. If the device allows a request queue, it may function
autonomously on several requests, producing only one interrupt for the
entire batch.
Furthermore, serial interrupt services are, themselves, very fast.
They slam their data as quickly as possible and get out ASAP. No fancy
calculations, just the minimum, mindless data transfer. Almost
everything else is handled later, in the bottom-half with interrupts
enabled. In fact, a serial ISR may have to re-interrupt it's own
bottom-half several times.
Those devices that do experience some slight slowdown are more likely
to have long interrupt disable windows themselves. Having several
smaller cli/sti windows is much better than one large cli/sti
window--It's just harder to program.
Q: BUT SUPPOSE I WANT A BALANCED PRIORITY SYSTEM?
Well, actually a prioritized system behaves like a balanced
system--most of the time. This occurs when all devices have short
interrupt lockout windows and short ISR execution times. The priority
mechanism is like a safety valve--it only really matters when some
device or combination of devices has held interrupts locked for an
extended period of time.
Q: BUT DOESN'T THAT SEEM UNFAIR TO OTHER DEVICES?
When a disk ISR gets delayed, that's all that happens, a slight
delay--disks and tapes are not real-time devices. When a serial ISR
gets delayed, data is destroyed--serial devices are real-time devices
that dictate the cadence of the entire system.
It may sound cruel, but you just can't be fair.
And speaking of sound, if the sound card gets delayed, we hear an
annoying pop right in the middle of our favorite piece of music.
These are real-time devices that can not tolerate excessive delays. If
we must defer some less time critical devices to meet the minimum
real-time criteria of devices, then that's a bargain.
Q: BUT SUPPOSE I REALLY WANT BOTH FAST SERIAL AND FAST DISK?
Ultimately, it's a bit of a compromise. Which is better:
* Reliable serial and slightly slower disk.
* Slightly faster disk and unreliable serial/modem support.
When paying an ISP for Internet access in $$$/hour, it's an easy
decision :-).
_________________________________________________________________
ISN'T THIS IRQ PRIORITY THING A BIT OF A NEW IDEA?
No. It's actually an old idea. I've been doing device drivers since
1977 and Unix kernel work since 1981. I've personally written 8 serial
drivers have used this many times commercially. Giving the serial
device the highest priority is actually standard practice in many
systems. With a 4Mhz CPU, these problems used to occur at 1200 baud
:-)
_________________________________________________________________
HOW DO I INSTALL IRQTUNE?
Q: WHERE SHOULD I PLACE IRQTUNE FILES?
* Decide what directory you would like to contain irqtune's home
directory. Some good choices are: /usr/local, /usr/lib,
/usr/lib/hwtools, /home.
* Exactly where depends upon your local conventions.
* In this example, we'll use /usr/local.
Q: HOW DO I UNPACK THE ARCHIVE?
* Go to the containing directory.
cd /usr/local
* Unpack the tar file:
tar zxvf irqtune.tgz
* This will create a directory, irqtune in /usr/local. This is
/usr/local/irqtune. We will call this IRQTUNE_HOME.
Note: If tar's z option has problems:
* An alternate unpack command:
gzip -d < irqtune.tgz | tar xvf -
Q: HOW DO I DO A SIMPLE INSTALLATION?
* Go to the IRQTUNE_HOME directory:
cd /usr/local/irqtune
* To install irqtune, enter:
make install
* This will install the files:
/sbin/irqtune
/sbin/irqtune_mod.o
/sbin/irqtune_npr.o
Q: WHY ARE .O FILES BEING PLACED IN THE /SBIN DIRECTORY?
As a note to purists, the installation directory is completely
arbitrary. /sbin is short, sweet, and easy to type.
The standard convention would be to install irqtune as /sbin/irqtune
and the .o files as, say, /usr/lib/irqtune/*.o. irqtune uses a
slightly different convention. It wants all installed files to be in
the same directory. This works fine because argv[0] will point to the
.o files. This also allows:
* Multiple versions of irqtune to be installed simultaneously (in
different directories).
* Simplifies testing of new versions.
* Provides better assurance that the built version is correct.
* Using one directory produces less clutter.
* One directory is more portable across all the myriad kernel
revisions.
* Future versions of irqtune may imbed the .o's directly into
irqtune.
Q: WHAT IF I REALLY DON'T WANT .O FILES IN /SBIN?
If placing .o files in /sbin is deemed to be an anathema, we have two
options:
* Simply use another directory in place of /sbin:
make SBIN=/whatever install
* Install with a stub (This will install /sbin/irqtune as a shell
script and use IRQTUNE_HOME/sbin/irqtune_*.o):
make INSTALL=sh install
Q: ARE THERE ANY SPECIAL CONSIDERATIONS FOR THIS ALTERNATE INSTALLATION
METHOD?
Yes. If the shell/stub installation method is used:
* IRQTUNE_HOME must not be removed.
* If using irqtune from an /etc/rc.d/rc.* file and irqtune will be
invoked before secondary mounts are done, IRQTUNE_HOME should be
placed on the root device.
_________________________________________________________________
HOW DO I USE IRQTUNE? DON'T I HAVE TO REBUILD MY KERNEL?
No, we do not have to rebuild the kernel. irqtune uses insmod and
rmmod to dynamically load and unload a kernel module. But it is
correct to sense that irqtune is a kernel patch.
Q: OK, IF IT'S A KERNEL PATCH, WHY NOT JUST ISSUE A KERNEL PATCH LIKE
EVERYBODY ELSE DOES (E.G. DIFF -U OUTPUT)?
irqtune will work even if we don't have the kernel source loaded. It
uses insmod to load the patch, invoke it, and then unload it. The IRQ
priority changes will last so long as the kernel is booted.
Q: HOW DO WE INVOKE IT?
irqtune takes two arguments optional arguments:
* irqtune master slave
The default is 3 14 which will work for many standard configurations.
See What about my non-standard hardware configuration? for details.
* Here is the usage output from irqtune:
usage: irqtune [options] [master] [slave]
version: 0.6
arguments:
master -- high priority IRQ on PIC master (DEFAULT: 3)
slave -- high priority IRQ on PIC slave (DEFAULT: 14)
general options:
-h -- display help
-v -- display irqtune version
-o -- reset to original values (0/8)
priority table options:
-q -- suppress priority table printing
-s -- sort table by priority
-x -- show inactive devices in table
error control options:
-e -- show full errors
-f -- force loading even if probe detects errors
-u -- force module unload
-V -- insmod verbose mode
-w -- treat warnings as errors
documentation options:
-F<file> -- use <file> instead of /proc/interrupts
-i -- install program
-L<directory> -- directory to search for insmod
-n -- nogo mode (just show what would happen)
Q: COULD WE DO THIS FROM MY /ETC/RC.D/RC.LOCAL FILE?
Yes. Just add a /sbin/irqtune line to this file and we're in business.
We may also issue another irqtune command at any time.
Q: WHAT IF IRQTUNE FAILS TO LOAD?
See the What about irqtune load failures or incompatibilities with
kernel revisions? section.
Q: AFTER IRQTUNE SETS A PRIORITY, HOW CAN WE QUERY THE RESULTS LATER?
We can't.
Due to a limitation of the PC interrupt hardware, it is not possible
to read back values set previously.
irqtune will attempt to place a message in the system log
(/usr/spool/syslog/syslog) when it changes the configuration.
Examining this log file, or simply invoking irqtune again, may be the
best ways to work around the hardware limitations.
Note: Some users have tried to use the "-n" option, thinking it will
act as a "query" mode. This option is used, primarily, to generate
examples in this document and will not have the desired effect.
_________________________________________________________________
WHAT ABOUT MY NON-STANDARD HARDWARE CONFIGURATION?
irqtune defaults for a standard IRQ configuration. It assumes that the
highest priority device should be on IRQ 3. This is normally the first
serial port on standard configurations, which is what you want.
Q: HOW DO I DETERMINE WHAT MY IRQ CONFIGURATION IS?
NOTE: For brevity, we've combined the non-sorted and sorted output in
these examples.
* Just type /sbin/irqtune -n -o and we'll get something like:
SORTED BY IRQ: SORTED BY PRIORITY:
I00/P00: 8578913 timer I00/P00: 8578913 timer
I01/P01: 109547 keyboard I01/P01: 109547 keybo
ard
I02/P02: 0 + cascade I02/P02: 0 + casca
de
I03/P10: 86470 + serial I11/P05: 197648 + sermu
x
I04/P11: 197648 + serial I12/P06: 17968 + eth
I11/P05: 197648 + sermux I13/P07: 1 math
error
I12/P06: 17968 + eth I14/P08: 93123 + Ux4F
I13/P07: 1 math error I03/P10: 86470 + seria
l
I14/P08: 93123 + Ux4F I04/P11: 197648 + seria
l
NOTE: /proc/interrupts, and therefore irqtune, only reports on active
devices. So to scope out the serial IRQ's, ideally, you'd have X
Windows up with your serial mouse and be connected via PPP to the net.
Q: OK, WE'VE GOT THE OUTPUT FROM /SBIN/IRQTUNE -N -O, WHAT DO WE DO WITH IT?
The leftmost number is the IRQ number. The next number is the
priority. The rightmost column is the internal device name (not to be
confused with /dev names). In the above case, the two serial ports are
on IRQ 3 and IRQ 4. Just use the lower number, in this case 3:
* /sbin/irqtune 3
This sets IRQ 3 to the highest priority.
* After this command, the IRQ priorities are now:
SORTED BY IRQ: SORTED BY PRIORITY:
I00/P05: 8578913 timer I03/P00: 86470 + seria
l
I01/P06: 109547 keyboard I04/P01: 197648 + seria
l
I02/P07: 0 + cascade I00/P05: 8578913 timer
I03/P00: 86470 + serial I01/P06: 109547 keybo
ard
I04/P01: 197648 + serial I02/P07: 0 + casca
de
I11/P12: 197648 + sermux I14/P07: 93123 + Ux4F
I12/P13: 17968 + eth I11/P12: 197648 + sermu
x
I13/P14: 1 math error I12/P13: 17968 + eth
I14/P07: 93123 + Ux4F I13/P14: 1 math
error
Q: BTW, WHAT'S THE CASCADE DEVICE I SAW IN THE OUTPUT OF IRQTUNE?
Glad you asked. There are actually two interrupt controllers, a master
and a slave. The slave is cascaded to the master via its IRQ 2. The
master controls IRQ's 0-7 and the slave controls IRQ's 8-15.
You actually may select two high IRQ priorities, one for the master
and one for the slave. irqtune defaults the slave to IRQ 14, which is
normally the disk controller.
In fact, cascade is sort of a "zero width" device as it does not
contribute to interrupt latency. Setting the cascade to top priority
on the master has an interesting effect which we'll see shortly.
Q: BUT WE'VE ALSO GOT AN ETHERNET CONTROLLER ON IRQ 12. WHAT ABOUT THAT?
In this case, we might want to use:
* /sbin/irqtune 3 12
because we want our ethernet card to have a higher priority than the
disk controller. Actually if we did have this configuration, setting 3
14 (the default) would make the ethernet card, the lowest priority
device in the system.
* The resulting priority would be:
SORTED BY IRQ: SORTED BY PRIORITY:
I00/P05: 8578913 timer I03/P00: 86470 + seria
l
I01/P06: 109547 keyboard I04/P01: 197648 + seria
l
I02/P07: 0 + cascade I00/P05: 8578913 timer
I03/P00: 86470 + serial I01/P06: 109547 keybo
ard
I04/P01: 197648 + serial I02/P07: 0 + casca
de
I11/P14: 197648 + sermux I12/P07: 17968 + eth
I12/P07: 17968 + eth I13/P08: 1 math
error
I13/P08: 1 math error I14/P09: 93123 + Ux4F
I14/P09: 93123 + Ux4F I11/P14: 197648 + sermu
x
Q: WHAT ABOUT THE SERIAL MULTIPLEXER CARD ON IRQ 11?
This is a bit tricky because now we've got a serial device on the
slave controller. It would be much better to put all serial cards on
the master controller. Things would stay much simpler.
In this case we would want to use:
* /sbin/irqtune 11
* The resulting priorities would be more complex and would result in
something like:
SORTED BY IRQ: SORTED BY PRIORITY:
I00/P13: 8578913 timer I02/P00: 0 + casca
de
I01/P14: 109547 keyboard I11/P00: 197648 + sermu
x
I02/P00: 0 + cascade I12/P01: 17968 + eth
I03/P08: 86470 + serial I13/P02: 1 math
error
I04/P09: 197648 + serial I14/P03: 93123 + Ux4F
I11/P00: 197648 + sermux I03/P08: 86470 + seria
l
I12/P01: 17968 + eth I04/P09: 197648 + seria
l
I13/P02: 1 math error I00/P13: 8578913 timer
I14/P03: 93123 + Ux4F I01/P14: 109547 keybo
ard
Q: WAIT A MINUTE. DIDN'T WE JUST SPECIFY A SLAVE IRQ NUMBER AS THE MASTER TO
IRQTUNE?
Yes, this is shorthand way of saying 2 11. You can make a slave device
top priority, but we get no options for the master IRQ. It will always
be 2, the cascade device. Remember, the cascade device contributes no
latency delay by itself.
Q: SO WHY IS THIS CONFIGURATION SO BAD?
Well, we boosted the priority of the serial multiplexer at the expense
of the regular serial ports. The only way to allow all serial ports
equally high priority is to group them on consecutive IRQ's and set
the high priority for the lowest of those IRQ's.
Q: HOW CAN WE FIX THIS WITH SOFTWARE?
We can't.
We're limited by the architecture of the PC and its interrupt
controllers. We must change the IRQ of a device by physical
restrapping--we can't do it by reprogramming the priority alone.
We'll go back to the earlier example. We'll wave a magic wand and
Poof!--assume we just restrapped the serial multiplexer to IRQ 5:
* /sbin/irqtune 3
* The resulting priorities would be:
SORTED BY IRQ: SORTED BY PRIORITY:
I00/P05: 8578913 timer I03/P00: 86470 + seria
l
I01/P06: 109547 keyboard I04/P01: 197648 + seria
l
I02/P07: 0 + cascade I05/P02: 197648 + sermu
x
I03/P00: 86470 + serial I00/P05: 8578913 timer
I04/P01: 197648 + serial I01/P06: 109547 keybo
ard
I05/P02: 197648 + sermux I02/P07: 0 + casca
de
I12/P13: 17968 + eth I14/P07: 93123 + Ux4F
I13/P14: 1 math error I12/P13: 17968 + eth
I14/P07: 93123 + Ux4F I13/P14: 1 math
error
Q: COULD WE DO MORE OF THIS RESTRAPPING, SAY, WITH THE ETHERNET CONTROLLER?
Sure. Waving the wand again, we restrap the ethernet card to IRQ 6.
* /sbin/irqtune 3
* The resulting priorities would be:
SORTED BY IRQ: SORTED BY PRIORITY:
I00/P05: 8578913 timer I03/P00: 86470 + seria
l
I01/P06: 109547 keyboard I04/P01: 197648 + seria
l
I02/P07: 0 + cascade I05/P02: 197648 + sermu
x
I03/P00: 86470 + serial I06/P03: 17968 + eth
I04/P01: 197648 + serial I00/P05: 8578913 timer
I05/P02: 197648 + sermux I01/P06: 109547 keybo
ard
I06/P03: 17968 + eth I02/P07: 0 + casca
de
I13/P14: 1 math error I14/P07: 93123 + Ux4F
I14/P07: 93123 + Ux4F I13/P14: 1 math
error
SO IN ORDER TO GET THE BEST OVERALL SYSTEM, WE MAY NEED TO CHANGE IRQ
PRIORITY AND PHYSICALLY CHANGE THE HARDWARE IRQ CONFIGURATION?
Exactly.
Different systems may have highly different criteria for what is
optimum. It is, ultimately, a choice that each system administrator
must make based upon the specific requirements for the particular
system in question. We can only provide tools to do the job, but the
final choice is ultimately decided on a case-by-case basis. There is
no one size fits all solution.
_________________________________________________________________
WHAT ABOUT IRQTUNE LOAD FAILURES OR INCOMPATIBILITIES WITH KERNEL REVISIONS?
Q: WHAT'S THE BOTTOM LINE?
irqtune makes every attempt to load its kernel module. irqtune probes
the kernel, /proc/ksyms, and insmod. It attempts to detect, correct,
or work around any difficulties. If the problems are truly severe,
irqtune will report this also. Normally, irqtune probes silently,
only reporting the results of the local system configuration if
there's an non-recoverable error.
* To show what irqtune actually checks for each time it's invoked,
here is the full error output (-e) from irqtune:
irqtune: version is 0.6
irqtune: kernel version 2.0.30
probe: irqtune must be invoked via the full path -- OK
probe: /sbin in $PATH -- YES
probe: insmod found in $PATH (/sbin) -- OK
probe: insmod simple execution -- OK
probe: insmod has version (2.0.0) -- YES
probe: rmmod found in insmod directory -- OK
probe: insmod version supports command line options -- OK
probe: insmod version (2.0.0) compatible with kernel version (2.0.
30) -- OK
probe: insmod version should be 2.1.34 (or better) -- WARNING
probe: insmod and kernel compatible with CONFIG_MODVERSIONS -- OK
probe: irqtune_mod loading will be tried -- OK
probe: kernel version irqtune built under (1.0.0) matches current
system -- NO
probe: kernel IRQ handling is compatible -- OK
probe: kernel has module support (CONFIG_MODULES) -- OK
probe: kernel has symbols -- OK
probe: kernel is using versions (CONFIG_MODVERSIONS) -- NO
probe: kernel symbols are checksummed (CONFIG_MODVERSIONS) -- NO
probe: kernel has /proc/interrupts -- OK
irqtune: simulating IRQ priority of 3/14
I00/P05: 8578913 timer
I01/P06: 109547 keyboard
I02/P07: 0 + cascade
I03/P00: 86470 + serial
I04/P01: 197648 + serial
I11/P12: 197648 + sermux
I12/P13: 17968 + eth
I13/P14: 1 math error
I14/P07: 93123 + Ux4F
irqtune: complete
* irqtune probes report YES/NO for informative messages. These may
be ignored. irqtune will automatically compensate for any
irregularities found.
* irqtune probes report OK/WARNING for problems that may prevent
proper operation. These may require changes to the local
configuration. Generally, these may be ignored unless irqtune
fails to load.
* irqtune probes report OK/ERROR for problems that will prevent
proper operation. These will require changes or upgrades to the
local configuration.
* If it is believed that the probes are reporting errors when none
actually exist, the -f option may be used to force irqtune to
continue anyway.
No matter what happens with probes or loads, irqtune will report the
final completion status as its last line: complete or error.
Q: WHAT SHOULD BE CHECKED FIRST?
* If we did a simple installation, we must specify the full
pathname, even if irqtune is placed in a directory that is in
$PATH. This is required because irqtune uses argv[0] to locate its
irqtune_mod.o file.
* /sbin should be in $PATH--this is the probable place for insmod.
As a convenience, irqtune will add /sbin to $PATH automatically.
* Be sure that the kernel was built with modules support
(CONFIG_MODULES) enabled. irqtune will probe for and report this
condition.
Q: WHAT IF THE CURRENT KERNEL REVISION IS DIFFERENT FROM THE KERNEL REVISION
IN THE PREBUILT MODULES?
This does not matter. irqtune is 99.44% kernel revision independent.
It is almost never necessary to rebuild the prebuilt modules. If
irqtune fails to load the modules, consider everything in this
section carefully before rebuilding irqtune.
Notes to programmers:
* The "kernel_version[]" does not need to match the kernel revision
for a module to load successfully.
* A module does not need MODVERSIONS to load in a MODVERSIONS
kernel.
Rebuilding the binaries before exploring the other options is a lot
like Vonnegut's OFF switch--it's comforting but not connected to
anything :-)
Q: ARE THERE ANY KERNEL REVISIONS THAT IRQTUNE WON'T WORK WITH?
When irqtune was first released, some experimental changes were made
to the kernel to solve the IRQ priority problem by use of a
round-robin, balanced priority system that was incompatible with
irqtune. These changes were ultimately removed but irqtune will not
work with kernel revisions 2.0.15 to 2.0.18. irqtune will detect this
condition and report an error. See the Where can I find additional
documentation or downloads? section.
Q: ARE THERE ANY INSMOD REVISIONS THAT ARE INCOMPATIBLE?
irqtune first trys to load irqtune_mod.o and falls back to
irqtune_npr.o if it detects a load error in insmod.
Some versions of insmod have severe difficulty loading modules when
the kernel is using MODVERSIONS. There is a known bug in insmod:
* 2.1.X insmod prior to 2.1.34 (e.g. 2.1.23)
* 2.0.X kernel
* kernel built with CONFIG_MODVERSIONS
Loading irqtune_mod.o will crash insmod, partially lock up irqtune's
module. irqtune will detect this and skip the irqtune_mod.o loading
entirely.
Generally, we should not use an older insmod with a newer kernel (e.g.
using a 2.0.X insmod on a 2.1.X kernel). irqtune will detect and
report this.
If insmod still has difficulties, we may want to upgrade it to 2.1.34
(or better). Newer versions of insmod are guaranteed to be backward
compatible to older kernels. This will increase the probability that
the irqtune_mod.o will load and irqtune will not have to fallback to
irqtune_npr.o. Note: insmod is actually part of the modutils
package. modutils under 2.0.X is called modules. See the Where can I
find additional documentation or downloads? section.
Q: WHAT IS THE DIFFERENCE BETWEEN IRQTUNE_MOD.O AND IRQTUNE_NPR.O?
The only kernel symbol that irqtune's kernel module (irqtune_mod.o)
uses is printk (to print a confirmation message to syslog). The
irqtune_npr.o module is exactly the same as irqtune_mod.o except
that it does not use printk. Since irqtune pre-checks all parameters
before attempting to load the kernel module, the confirmation message
is a nicety but not a necessity.
Q: WHAT IF WE DON'T HAVE ELF BINARY SUPPORT?
Well, we should upgrade the kernel as ELF binaries are cool :-). But
if that's not possible, we'll just have to recompile irqtune to create
a.out binaries. This is, perhaps, the only justification for
rebuilding irqtune. Just be sure that /usr/src/linux/include is
installed. The exact procedure for building a.out binaries can vary
with compiler revision, so it's important to check the documentation
on this (a parameter or two may need to be added).
* Then, just type:
make warp9
* Here is the makefile's own documentation:
INSTALLATION:
install install prebuilt binaries in /sbin
custom create custom installation -- Normally _not_ required
-- Try
install first (EQUIVALENT: kvers sbin install)
uninstall remove prebuilt binaries from /sbin
INSTALLATION OVERRIDES:
SBIN SBIN=/whatever specify installation directory (DEFAUL
T: /sbin)
INSTALL INSTALL=simple simple installation (DEFAULT: simple)
INSTALL=sh install as shell stub
INSTALLER installation program (bin/install) (DEFAULT: sbin/irq
tune)
REBUILDING FROM SOURCE:
kvers create kernel_version
sbin rebuild binaries from source (full or partial)
FORCE FULL REBUILD (requires /usr/src/linux/include):
clean remove .o files to force full recompile)
warp9 rebuild all (EQUIVALENT: clean kvers sbin install)
Q: WHAT ABOUT IRQ SHARING OF SERIAL PORTS?
Under the 2.0.X (and later) kernels, use of IRQ sharing will defeat
IRQ priority because the serial port ISR's are installed as slow
rather than fast interrupts (e.g. they don't use the SA_INTERRUPT
flag).
* Change the serial port IRQ configuration so that ports don't share
IRQ's
* The other option is a kernel source patch to
drivers/char/serial.c:
Change the following line:
#define IRQ_T(info) ((info->flags & ASYNC_SHARE_IRQ) ? SA_SHIRQ :
SA_INTERRUPT)
and replace it with:
#define IRQ_T(info) (((info->flags & ASYNC_SHARE_IRQ) ? SA_SHIRQ :
0) | SA_INTERRUPT)
Under earlier kernels this is not a problem because the serial ISR was
always installed with SA_INTERRUPT.
_________________________________________________________________
WHAT ABOUT HARDWARE/CONFIG PROBLEMS?
Q: WHAT IF THE SERIAL PORT DOESN'T WORK?
* The First Law of Computing: Make sure it's plugged in.
* Double check the strapping for port and IRQ assignments. Verify
that the port/IRQ desired is actually the one that is strapped.
Some serial ports will appear to work even if the IRQ is
wrong--they will just be extremely slow. Some Plug-N-Play (PnP)
ports may need to be explicitly strapped.
* Check for port/IRQ conflicts with other devices. Serial ports may
share IRQ assignments but sharing an IRQ between a serial port and
something else (e.g. an ethernet card) is fraught with peril. See
the previous section about serial IRQ sharing under 2.0.X kernels.
* Verify that the IRQ that the kernel reports as a result of
auto-detection is the IRQ that was strapped. Some serial ports or
hardware configurations can fool the auto-detect. If necessary,
use the setserial program (in /etc/rc.d/rc.local) to force the
kernel port/IRQ settings to match the hardware strapping.
* Verify that irqtune got the correct IRQ numbers for the specific
configuration. For example, suppose our primary serial port is on
IRQ 3, but we gave irqtune the value 4. We just made the serial
device on IRQ 3 into priority 14, not priority 0.
* Verify that the serial port baud rate is set correctly. That is,
it is the same as what the modem expects. The serial port baud
rate also has performance implications (See Q: What about
increasing the serial port baud rate? for details).
Q: WHAT IF PPP DOESN'T WORK?
See the PPP man page and PPP-Howto for best information, but some
recommended options:
* asyncmap 0
* crtscts
* defaultroute
* modem
* mru 296 mtu 296
* passive
AT-like modems have a special character to escape from data mode to
command mode. To avoid confusion here, we'll call this modem escape
character a guard character.
The default guard character is '+' (decimal 43, hex 2B). Normally, 3
such characters are required within a special timing sequence.
Although it is unlikely, it is still possible that some PPP packets
could generate the guard sequence inadvertantly. To prevent this, we
may want to inhibit the generation of the guard character in a data
sequence. To do this, we would add the additional PPP option:
* escape 2B
Since '+' is a common ASCII character (PPP escaped characters generate
two characters), we may wish to use a less common value for the guard
character. For example, a less common value might be (decimal 200, hex
C8). We would add an ATS2=200 command to our modem dialer script and
change the PPP escape option to C8.
Q: WHY DOES PPP CONSISTENTLY DROP EVERY SECOND PACKET SENT FROM LINUX,
RESULTING IN A 50% PACKET LOSS?
Some braindamaged PPP implementations do not handle PPP flag
optimization!
The PPP protocol uses a flag byte to separate packets. Each packet
begins with a flag and ends with a second flag.
* Loosely, each packet looks something this:
<FLAG> <PACKET (e.g. header, data, CRC)> <FLAG>
* Thus, a sequence of packets might look like this:
<FLAG><PACKET1><FLAG> <FLAG><PACKET2><FLAG> <FLAG><PACKET3><FLAG>
...
* However, as an optimization, the PPP protocol permits trailing and
leading flag bytes to be combined within a sequence of packets:
<FLAG> <PACKET1> <FLAG> <PACKET2> <FLAG> <PACKET3> <FLAG> ...
Although the PPP protocol requires implementations receiving packets
to handle flag optimization, some broken PPP implementations do not
understand it!
These implementations see the trailing flag, process the packet, then
look for a fresh flag. They don't realize that the trailing flag of
PACKET1 may perform double duty as the leading flag of PACKET2. They
will ignore all data until they see a new flag (which, in this
example, is the flag between PACKET2 and PACKET3). Thus, PACKET2 will
be seen as noise data and be ignored. These implementations will only
see only the odd number packets (e.g. PACKET1, PACKET3, PACKET5,
etc.), resulting in a 50% packet loss!
Linux PPP implements flag optimization correctly and enables it by
default. As charity to others, Linux does allow flag optimization to
be turned off, but currently, this this requires the kernel to be
rebuilt.
In Linux, to turn off flag optimization on transmit, do the following:
* Edit the file drivers/net/ppp.c
* Change the line:
static int flag_time = OPTIMIZE_FLAG_TIME;
* Replace this with:
static int flag_time = 0;
* Rebuild the kernel.
Note: A better solution is to return the defective PPP implementation
to the vendor and demand a refund or replacement!
Q: HOW CAN WE BE CERTAIN THAT THE PPP FLAG OPTIMZATION LOSS IS OCCURING?
By lowering the baud rate to something that is guaranteed not to drop
data due to speed problems (e.g. 300 baud). If we get a consistent 50%
loss at this low rate, this is almost certain proof of the flag
optimization problem.
_________________________________________________________________
WHAT OTHER PERFORMANCE SOFTWARE REMEDIES MUST BE DONE?
Q: WHAT ABOUT USING HDPARM -U TO SET THE INTERRUPT-UNMASK FLAG IN THE HARD
DISK DRIVER?
This is only necessary for the IDE driver. The SCSI driver has short
disable windows by default. This will shorten the IDE interrupt
disable windows.
Beware: Without this option, IDE disk activity will almost certainly
cause serial data dropouts. If we have an IDE disk, this is mandatory.
Q: WHAT ABOUT DISABLING VAN JACOBSEN HEADER COMPRESSION IN PPP?
This reduces the amount of bottom-half processing the system has to do
at the expense of larger packets being sent. This may be helpful on
slower CPU's or heavily loaded configurations.
Q: WHAT ABOUT ADJUSTING THE MRU/MTU NUMBERS IN PPP?
Reducing the MRU/MTU to a minimum (296) reduces the bottom-half
processing and flip-buffer latency at the expense of adding extra
overhead bytes due to the reduced packet size. The optimal value will
vary from configuration to configuration.
Beware: Start with 296 as the optimal may not be 1500.
The flip-buffer is a double buffer mechanism in the serial/tty drivers
through which all data must pass. It has a fixed size of only 512
bytes. MRU/MTU greater than the flip-buffer size may create an
internal race condition that may cause dropouts on slower CPU's or
heavily loaded configurations.
Q: WHAT ABOUT GOING TO NEWER KERNEL REVISIONS?
Although irqtune will work surprisingly well with just about any
kernel revision, the low level IRQ handlers and device drivers have
been vastly improved in the 2.0.X kernels. This will only improve
irqtune's effect.
Q: WHAT ABOUT INCREASING THE SERIAL PORT BAUD RATE?
The serial port baud rate should be high enough to support the maximum
expected transfer rate--but no higher. Higher speed settings place
extra strain on the CPU, increasing the likelihood of overruns.
For a 33.6 modem, the minimum baud rate would be 38400. However, with
compression, the expected transfer rate can be as high as 6 KB/second.
This would require a baud rate of 57600. This may strain the CPU, and
since the transfer rate is nominally about 4 KB/second, a lower baud
rate may be a good compromise.
The best way is to try several rates, then benchmark them to see which
provides the best overall performance.
Note: Because of backward compatibility to older systems, we can't
just set 57600 directly with stty, kermit, pppd, etc. Specify 38400 to
these programs, and use the setserial program with an option of
spd_hi. For ISDN speeds, use spd_vhi. Other options are possible so
be sure to consult the manpage.
_________________________________________________________________
HOW CAN I TELL IF IRQTUNE ACTUALLY DID ANYTHING FOR ME?
Well, first off, if PPP/SLIP was dying mysteriously, it will probably
be more reliable.
Q: HOW CAN WE BENCHMARK IRQTUNE?
Run without it and get a feel for the transfer rate:
* Hit many favorite web sites and note the transfer rates in
bytes/second. Make life easy. Netscape is at least one browser
that reports transfer rates in bytes/second in the status line.
* FTP reports the transfer time of a file in bytes/second. Download
(or upload) a few files (300K or greater to smooth out the
benchmark) and note the transfer rates. Use a mix of ascii and
binary files that to see the effect of modem compression. Binary
won't compress too well so the numbers will reflect the real
transfer rate. Ascii files which typically compress 2:1 will up
the effective rate by 2X, making it more stressful.
* Try several things of varying duration, different times of day,
different sites to accomodate variations in network loading. Don't
stop until there is an average set of numbers that are more or
less repeatable.
Repeat this using irqtune and note the transfer times again.
NOTE: IRQTUNE just won't quit--if you want to test in the original
mode again, reboot the system first.
Q: WHAT IF WE STILL DON'T SEE ANY REAL IMPROVEMENT?
It's a matter of probability. Performance measurement is as much art
as science.
* We're much more likely to see improvement on a DX2/66 than a
Pentium/166. With the 166, we may be overpowering the problem.
We'll still have a problem, we just won't notice it as much
because the Pentium has the extra speed to burn. A badly tuned
Ferrari may still outperform a well-tuned VW :-)
* What problem was occurring before? Which of the symptoms listed
earlier is happening? If performance was 2500 bytes/second before
using irqtune, we're less likely to notice the smaller jump to,
say, 2800.
* System loading is very light. The problems that irqtune will fix
are more likely to happen when more devices and more work are
added. On a certain configuration, irqtune had little effect.
However, when a brand new SCSI DAT drive was added, serial
performance nosedived. Using irqtune brought performance back to
the correct level.
* We may have a rogue interrupt service that disables interrupts for
something outrageous, say 2 ms. This is singularly longer than the
1194 us. in the earlier example. irqtune will still help, but the
real solution here is to reduce interrupt lockout times in the
other device below the 1194 us. threshold.
_________________________________________________________________
WHERE CAN I FIND ADDITIONAL DOCUMENTATION OR DOWNLOADS?
* man pages for: insmod, hdparm, pppd, setserial
* The Linux Documentation Project
+ Homepage: http://sunsite.unc.edu/LDP
+ HOWTO's: ftp://sunsite.unc.edu/pub/Linux/doc/HOWTO
* The Linux Software Map (Hypertext)
+ http://www.boutell.com/lsm
* Linux v2 Information HQ
+ http:/www.linuxhq.com
* ftp.kernel.org
+ http://ftp.kernel.org
* 2.0.X kernel and compatible modutils downloads
+ ftp://ftp.funet.fi/pub/Linux/kernel/linux/v2.0,
ftp://ftp.funet.fi/pub/Linux/kernel/linux/v2.0/modules-2.0.
0.tar.gz
+ ftp://ftp.kernel.org/pub/linux/kernel/v2.0,
ftp://ftp.kernel.org/pub/linux/kernel/v2.0/modules-2.0.0.ta
r.gz
* 2.1.X kernel and compatible modutils downloads
+ ftp://ftp.funet.fi/pub/Linux/kernel/linux/v2.1,
ftp://ftp.kernel.org/pub/linux/kernel/v2.1/modutils-2.1.34.
tar.gz
+ ftp://ftp.kernel.org/pub/linux/kernel/v2.1,
ftp://ftp.kernel.org/pub/linux/kernel/v2.1/modutils-2.1.34.
tar.gz
_________________________________________________________________
CHANGES
REVISION 0.2 CHANGES:
* No code changes
* Major rewrite and expansion of the problem explanation section
* More thorough explanation of how and why irqtune works
* Explanation of why serial devices must be highest priority
* Impact on other devices
* Cleaner and better installation instructions
* Better benchmarking section
* Problem resolution section
* Explanation of my prior misread on the EOI thing
REVISION 0.3 CHANGES:
* irqtune automatically prints interrupt priority table
* Significantly improved error detection and reporting
* Improved loading under different kernel revisions
* insmod now invoked with -x to improve loading
* Added module without printk's -- irqtune_npr.o
* Special makefile options for difficult or custom rebuilds
* Corrected error in hdparm -u note.
* Improved the non-standard configuration section
* Added note and patch about IRQ sharing under 2.0.X kernels
REVISION 0.4 CHANGES:
* No code changes
* Updated links in the FAQ to reflect configuration changes at my
ISP, best.com. Specifically, the FTP address changed from
ftp://www.best.com/pub/cae/irqtune.tgz to
ftp://ftp.best.com/pub/cae/irqtune.tgz
* irqtune distribution now available via HTTP as well as FTP.
* Consolidated and improved trouble resolution sections.
* Added explanation of irqtune_mod.o vs. irqtune_npr.o.
* Added explanation of /sbin usage.
* Corrected some typos.
REVISION 0.5 CHANGES:
* Added alternate installation procedure with "INSTALL=" option.
* /sbin is added to $PATH automatically.
* Added "-u" option.
* Consolidated and improved installation documentation.
* Additional trouble resolution documentation.
* Again, updated links in the FAQ to reflect configuration changes
at my ISP, best.com. Specifically, the FTP address changed from
ftp://ftp.best.com/pub/cae/irqtune.tgz to
ftp://shell5.ba.best.com/pub/cae/irqtune.tgz
REVISION 0.6 CHANGES:
* Added an "inb" to force PIC to a determinate state before setting
priority.
* Added include of errno.h to allow compilation with newer libc
distributions.
* Added code to probe ksyms, kernel, insmod for incompatibilities.
Specifically, versions of insmod that have bugs (fixed in insmod
2.1.34) are detected and handled correctly.
* Added "-f" option.
* Added documentation about insmod incompatibilities and upgrades.
* Added documentation and workaround for broken PPP implementations
of PPP flag optimization.
* Added documentation about recommended serial port baud rates.
* Added section with hyperlinks to other documentation and
downloads.
* Added irqtune to LSM (finally :-).
* Install script now obsolete. irqtune now self installs. Some
moronic systems still don't have csh installed.
_________________________________________________________________
TABLE OF CONTENTS
IRQTUNE -- A Linux IRQ Priority Optimizer
* Where do I get irqtune?
* How do I know if I need irqtune?
* What is actually happening to cause these problems?
* How does irqtune help this?
* Why does the serial interrupt service require the highest
priority?
* Isn't that example very unlikely under Linux?
* Doesn't this hurt the performance of other devices?
* Isn't this IRQ priority thing a bit of a new idea?
* How do I install irqtune?
* How do I use irqtune? Don't I have to rebuild my kernel?
* What about my non-standard hardware configuration?
* What about irqtune load failures or incompatibilities with kernel
revisions?
* What about hardware/config problems?
* What other performance software remedies must be done?
* How can I tell if irqtune actually did anything for me?
* Where can I find additional documentation or downloads?
* Changes
* Table of Contents
=============================================================================
Andrew Martin|andrew@housemartin.apana.org.au|http://housemartin.apana.org.au
----------------------------------------------------------------------------
Looking beyond the embers of the bridges glowing bright behind us
To a glimpse of how green it was on the other side. [Gilmour/Samson]
=============================================================================
