Leaks could occur under some circumstances if the file stream handler
errored out during the open call. Among others, this could cause very
big memory leaks if there were a problem with unreadable disk queue
files. In regard to the memory leak, this
closes: http://bugzilla.adiscon.com/show_bug.cgi?id=256
The original code had quite some issues, which are fixed by this
commit. Also we do more error checking now.
Signed-off-by: Rainer Gerhards <rgerhards@adiscon.com>
Under some circumstances an invalid truncation was detected. This
code has now been removed, a file change (and thus resent) is only
detected if the inode number changes.
This can happen when 0 bytes are read from the input file, and some
writer appends data to the file BEFORE we check if a rollover happens.
The check for rollover uses the inode and size as a criterion. So far,
we checked for equality of sizes, which is not given in this scenario,
but that does not indicate a rollover. From the source code comments:
Note that when we check the size, we MUST NOT check for equality.
The reason is that the file may have been written right after we
did try to read (so the file size has increased). That is NOT in
indicator of a rollover (this is an actual bug scenario we
experienced). So we need to check if the new size is smaller than
what we already have seen!
- bugfix: a couple of problems that imfile had on some platforms, namely
Ubuntu (not their fault, but occured there)
- bugfix: imfile utilizes 32 bit to track offset. Most importantly,
this problem can not experienced on Fedora 64 bit OS (which has
64 bit long's!)
by implementing some code that was missing so far ;) as well as
finding some real bugs. I also did some general cleanup, removing
debug strings and such. This code should be fairly OK to use, except
when "exec only when previous action was suspended" is used -- this is
NOT yet re-implemented in the tuned engine.
The previous fix fixed an issue with on/off bying used in the exact wrong
semantic. It corrected the situation, but failed to fix one spot where the
wrong semantics were used. This is done with this commit.
Note that this is NOT a bug seen in any released version.
currently being accessed buffer could be overwritten with new data.
While this probably did not cause access violations, it could case loss
and/or duplication of some data (definitely a race with no deterministic
outcome)
note that a buffer size calculation was done wrong, but this was cosmetic
because our buffers currently all use byte size, so even though the
formula was wrong, the result was correct.
- improved testbench
- bugfix: potential data loss during file stream shutdown
- bugfix: potential problems during file stream shutdown
The shutdown/close sequence was not clean, what potentially (but
unlikely) could lead to some issues. We have not been able to describe
any fatal cases, but there was some bug potential. Sequence has now
been straighted out.
When a write error occured in stream.c, variable iWritten had the error
code but this was handled as if it were the actual number of bytes
written. That was used in pointer arithmetic later on, and thus could
lead to all sorts of problems. However, this could only happen if the
error was EINTR or the file in question was a tty. All other cases were
handled properly. Now, iWritten is reset to zero in such cases, resulting
in proper retries.
support for enhancing probability of memory addressing failure by
using non-NULL default value for malloced memory (optional, only if
requested by configure option). This helps to track down some
otherwise undetected issues within the testbench and is expected
to be very useful in the future.
we needed to release ALL resources (including file handles!) only
after the the async writer thread has terminated (else it may access
them). In this case, we had a file handle leak, because the handle was
sometimes only opened in the async writer, but the close was attempted
before the writer even started (in some cases).
...after we seem to have identified the root cause of the segfault.
I leave them commented out so that we can re-activate it if need
arises (aka "get some practice drill first").
Most severely affected omfile. The problem was that some buffers were
freed before the asynchronous writer thread was shut down. So the
writer thread accessed invalid data, which may even already be
overwritten. Symptoms (with omfile) were segfaults, grabled data
and files with random names placed around the file system (most
prominently into the root directory). Special thanks to Aaron for
helping to track this down.
Problems could lead to abort and/or memory leak. The module is now hardened in a very
conservative way, which is sub-optimal from a performance point of view. This should
be improved if it has proven reliable in practice.
I found out that the previous segfault fix did not correct the root
cause of the problem. Thus, I can re-instantiate the more performance-
optimal logic. In the next step, I'll merge in the real fix, so do NOT
use this commit as code you actually run!
