Re: AW: flexfax: Problems submitting very large Number of jobs sequen tial ly

[Mike Bristow <michaelb@bessie.mojo.org>]

Hi,

On Mon, Jul 27, 1998 at 02:42:19AM +0000, Didier Gautheron wrote:
> Hi
> 
> Christoph Stotz wrote:
> > The theory of too many files donīt seem to me to be the solution.
> Yes but a 1500-5000 files in a directory is a problem on all OS. and

[only those that use a linked list for directories; those that use btrees
 are much happier with big directories.]

> here you have 4 to 5 dirs :
> info
> sendq
> doneq
> docq

I've got a patch that puts the info files in subdirectories under info,
although I don't think that will help here as (AFAIU) as the numbers 
being called are in a small subset of all possible numbers.   (as opposed to
the tpc cell I see which reached a totally stupid info directory size with
~30,000 entries)

> > Is it possible to tell hylafax to call a batch-file if the number
> > of jobs in queue is smaller than the number of free modemlines
> > avaible ?
> I don't think you could do this out of the box.
> But you could use some trick on xferlog and or faxwatch to get a tick
> each time a fax is sent.

This is probably the approach I'd take; prep the queue with the number 
of modems + 1 or two; run sendfax everytime a line gets written which 
matches /""$/ or something.

> The text below is over simplified but I hope there's no big mistake.
> 
> The delay is coming from the faxq's scheduler : it's a multi threads prg
> using fork and exec to achieve its goals.
> The main thread (faxq) is waiting in a loop for events:
> events are coming from:
> fifo : 
> - hfaxq submiting job
> - faxgetty
> - faxsend (not sendfax , faxsend is the prg which actually sends the
> fax)
> child's dead:
> - mainly faxsend
> timeout
> - mainly from jobs waiting queue
> 
> Of course when faxq is not in its waiting loop it doesn't see other
> events coming, they are waiting in system's buffers.
> 
> faxq is shooting itself in the foot and it's pushing your computer in
> the red zone with big queue :
> runScheduler in faxd/faxQueueApp.c++ is processing all the jobs in the
> run gueue even if no modems are in READY state. So in your case the main
> thread is reading 1500 and up files (jobs status file) for nothing - if
> the main goal of hylafax is sending faxes, not trashing out of
> delay,tries or whatever faxes, of course - And runSheduler is called
> many times: each time you add a job, each time a job finished, and so
> on.
> 
> It's easy to change this ie : change assignModem behavior so
> runScheduler will return when there's no more modems available . I tried
> and yes now with a 10000 run queue, sendfax takes the same time. 
> But side effects are unknow, after all it's just faxq's main routines
> :).

Perhaps it would be better to change:

while (getnextentryinqueue() != NULL) {
...

to:

while (getnextentryinqueue() != NULL && ++counter < QUEUE_TOO_BIG) {
...

[or whatever]

That will help cut faxq down to size when it's deciding which fax to send
as well; when the queue has 9,000,000 entries it's probably less important 
to weight each entry and ponder on the relative merits of each one.  
QUEUE_TOO_BIG 	should probably be at least a compile time, probably better
as run time, tweakable.  It should be different on a 386 with a shonky modem
compared to a dual P-II 400 with a bank of 256 modems. :-)

-- 
"Virgin Trains: It's quicker by balloon."
	-- Peter Corlett