core dump under solaris2.5.1

[Michael Douglass <mikedoug@texas.net>]

Okay, I have compiled and installed the latest hylafax v4.0p1 on a
solaris 2.5.1 (sparc) box.  The daemons run fine, but when you try to
connect to it (say with faxstat) it (hfaxd) forks a child which then
aborts.  Below is a gdb 'bt', the function fxStr::~fxStr(), a truss
output, and some commentary.

gdb 'bt':
#0  0xef66452c in kill ()
#1  0xef62a620 in abort ()
#2  0x546ec in __eprintf ()
#3  0x41080 in fxStr::~fxStr ()
#4  0x32c30 in InetFaxServer::InetFaxServer ()
#5  0x32b4c in InetSuperServer::newChild ()
#6  0x32874 in SuperServer::inputReady ()
#7  0x43d88 in Dispatcher::notify ()
#8  0x43ac4 in Dispatcher::dispatch ()
#9  0x438d0 in Dispatcher::dispatch ()
#10 0x3bc2c in main ()

Here is fxStr::~fxStr():

fxStr::~fxStr()
{
    assert(data);
    if (data != &emptyString) free(data);
}

Here is the relevant truss output:

13328:  brk(0x0007FE18)                                 = 0
13328:  brk(0xEF7B7E18)                                 Err#12 ENOMEM
13328:  brk(0x0007FE18)                                 = 0
13328:  brk(0xEF7B7E18)                                 Err#12 ENOMEM
13328:  write(2, " S t r . c + +", 7)                   = 7
13328:  write(2, " : 1 3 4", 4)                         = 4
13328:  write(2, " :   f a i l e d   a s s".., 20)      = 20
13328:  write(2, " d a t a", 4)                         = 4
13328:  write(2, " '\n", 2)                             = 2

And the commentary:

It appears that somewhere in the handling of an fxStr in
InetFaxServer's constructor, for some reason fxStr is trying to
allocate an abnormally great amount of memory and failing, setting data
to point to NULL; which, in turn, causes the assert in ~fxStr() to
fail.  Now, by looking through Str.c++ there is only one route that I
found that this -should- be possible (I think I must have missed one).
It is when the function setMaxLength() is called as a member
function--all other functions that change where data points to then
manipulates what is stored *at* data which would cause a segmentation
violation.  (These two functions setMaxLength() and the resizeInternal()
are provided below.)

*phew* Well, I'm no expert--but can anyone point out the problem here?
I will be sending another email shortly with another potential problem
that I see with code in the fxStr function.

void fxStr::resizeInternal(u_int chars)
{
    if (slength > 1) {
        if (chars > 0) {
            if (chars >= slength)
                data = (char*) realloc(data,chars+1);
        } else {
            assert(data != &emptyString);
            free(data);
            data = &emptyString;
        }
    } else {
        assert(data == &emptyString);
        if (chars)
            data = (char*) malloc(chars+1);
    }
}

void fxStr::setMaxLength(u_int len)
{
    if (slength>1) resizeInternal(fxmax(len,slength-1));
}


-- 
Michael Douglass
Texas Networking, Inc.

   "I broke perl again"
  -- Anonymous TexasNet sysadmin