mirror of
https://github.com/rsyslog/rsyslog.git
synced 2025-12-22 23:30:46 +01:00
165 lines
11 KiB
HTML
165 lines
11 KiB
HTML
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
|
|
<html><head><title>troubleshooting rsyslog</title></head>
|
|
<body>
|
|
<h2>troubleshooting rsyslog</h2>
|
|
<p><b>Having trouble with <a href="http://www.rsyslog.com">rsyslog</a>?</b>
|
|
This page provides some tips on where to look for help and what to do
|
|
if you need to ask for assistance. This page is continously being expanded.
|
|
<p>Useful troubleshooting ressources are:
|
|
<ul>
|
|
<li>The <a href="http://www.rsyslog.com/doc">rsyslog documentation</a> - note that the online version always covers
|
|
the most recent development version. However, there is a version-specific
|
|
doc set in each tarball. If you installed rsyslog from a package, there usually
|
|
is a rsyslog-doc package, that often needs to be installed separately.
|
|
<li>The <a href="http://wiki.rsyslog.com">rsyslog wiki</a> provides user tips and experiences.
|
|
<li>Check <a href="http://bugzilla.adiscon.com">the bugzilla</a> to see if your problem is a known
|
|
(and even fixed ;)) bug.
|
|
</ul>
|
|
<p><b>Malformed Messages and Message Properties</b>
|
|
<p>A common trouble source are <a href="syslog_parsing.html">ill-formed syslog messages</a>, which
|
|
lead to to all sorts of interesting problems, including malformed hostnames and dates.
|
|
Read the quoted guide to find relief. A common symptom is that the %HOSTNAME% property is
|
|
used for generating dynafile names, but some glibberish shows up. This is caused by the
|
|
malformed syslog messages, so be sure to read the
|
|
<a href="syslog_parsing.html">guide</a> if you face that problem. Just let me add that the
|
|
common work-around is to use %FROMHOST% or %FROMHOST-IP% instead. These do not take the
|
|
hostname from the message, but rather use the host that sent the message (taken from
|
|
the socket layer). Of course, this does not work over NAT or relay chains, where the
|
|
only cure is to make sure senders emit well-formed messages.
|
|
<p><b>Configuration Problems</b>
|
|
<p>Rsyslog 3.21.1 and above has been enhanced to support extended configuration checking.
|
|
It offers a special command line switch (-N1) that puts it into "config verfication mode".
|
|
In that mode, it interprets and check the configuration file, but does not startup. This
|
|
mode can be used in parallel to a running instance of rsyslogd.
|
|
<p>To enable it, run rsyslog interactively as follows:
|
|
<p><b><i>/path/to/rsyslogd -f/path/to/config-file -N1</i></b>
|
|
<p>You should also specify other options you usually give (like -c3 and whatever else).
|
|
Any problems experienced are reported to stderr [aka "your screen" (if not redirected)].
|
|
<p><b>Configuration Graphs</b>
|
|
<p>Starting with rsyslog 4.3.1, the
|
|
"<a href="rsconf1_generateconfiggraph.html">$GenerateConfigGraph</a>"
|
|
command is supported, a very valuable troubleshooting tool. It permits to
|
|
generate a graph of how rsyslogd understood its configuration file. It is assumed that
|
|
many configuration issues can easily be detected just by looking at the configuration graph.
|
|
Full details of how to generate the graphs, and what to look for can be found in the
|
|
"<a href="rsconf1_generateconfiggraph.html">$GenerateConfigGraph</a>"
|
|
manual page.
|
|
<p><b>Asking for Help</b>
|
|
<p>If you can't find the answer yourself, you should look at these places for
|
|
community help.
|
|
<ul>
|
|
<li>The <a href="http://kb.monitorware.com/rsyslog-f40.html">rsyslog forum</a>. This is
|
|
the preferred method of obtaining support.
|
|
<li>The <a href="http://lists.adiscon.net/mailman/listinfo/rsyslog">rsyslog mailing list</a>.
|
|
This is a low-volume list which occasional gets traffic spikes.
|
|
The mailing list is probably a good place for complex questions.
|
|
</ul>
|
|
<p><b>Debug Log</b>
|
|
<p>If you ask for help, there are chances that we need to ask for an rsyslog debug log.
|
|
The debug log is a detailled report of what rsyslog does during processing. As such, it may
|
|
even be useful for your very own troubleshooting. People have seen things inside their debug
|
|
log that enabled them to find problems they did not see before. So having a look at the
|
|
debug log, even before asking for help, may be useful.
|
|
<p>Note that the debug log contains most of those things we consider useful. This is a lot
|
|
of information, but may still be too few. So it sometimes may happen that you will be asked
|
|
to run a specific version which has additional debug output. Also, we revise from time to
|
|
time what is worth putting into the standard debug log. As such, log content may change
|
|
from version to version. We do not guarantee any specific debug log contents, so do not
|
|
rely on that. The amount of debug logging can also be controlled via some environment
|
|
options. Please see <a href="debug.html">debugging support</a> for further details.
|
|
<p>In general, it is advisable to run rsyslogd in the foreground to obtain the log.
|
|
To do so, make sure you know which options are usually used when you start rsyslogd
|
|
as a background daemon. Let's assume "-c3" is the only option used. Then, do the following:
|
|
<ul>
|
|
<li>make sure rsyslogd as a daemon is stopped (verify with ps -ef|grep rsyslogd)
|
|
<li>make sure you have a console session with root permissions
|
|
<li>run rsyslogd interactively: /sbin/rsyslogd ..your options.. -dn > logfile
|
|
<br>where "your options" is what you usually use. /sbin/rsyslogd is the full path
|
|
to the rsyslogd binary (location different depending on distro).
|
|
In our case, the command would be
|
|
<br>/sbin/rsyslogd -c3 -dn > logfile
|
|
<li>press ctrl-C when you have sufficient data (e.g. a device logged a record)
|
|
<br><b>NOTE: rsyslogd will NOT stop automatically - you need to ctrl-c out of it!</b>
|
|
<li>Once you have done all that, you can review logfile. It contains the debug output.
|
|
<li>When you are done, make sure you re-enable (and start) the background daemon!
|
|
</ul>
|
|
<p>If you need to submit the logfile, you may want to check if it contains any
|
|
passwords or other sensitive data. If it does, you can change it to some <b>consistent</b>
|
|
meaningless value. <b>Do not delete the lines</b>, as this renders the debug log
|
|
unusable (and makes Rainer quite angry for wasted time, aka significantly reduces the chance
|
|
he will remain motivated to look at your problem ;)). For the same reason, make sure
|
|
whatever you change is change consistently. Really!
|
|
<p>Debug log file can get quite large. Before submitting them, it is a good idea to zip them.
|
|
Rainer has handled files of around 1 to 2 GB. If your's is larger ask before submitting. Often,
|
|
it is sufficient to submit the first 2,000 lines of the log file and around another 1,000 around
|
|
the area where you see a problem. Also,
|
|
ask you can submit a file via private mail. Private mail is usually a good way to go for large files
|
|
or files with sensitive content. However, do NOT send anything sensitive that you do not want
|
|
the outside to be known. While Rainer so far made effort no to leak any sensitive information,
|
|
there is no guarantee that doesn't happen. If you need a guarantee, you are probably a
|
|
candidate for a <a href="professional_support.html">commercial support contract</a>. Free support
|
|
comes without any guarantees, include no guarantee on confidentiality
|
|
[aka "we don't want to be sued for work were are not even paid for ;)].
|
|
<b>So if you submit debug logs, do so at your sole risk</b>. By submitting them, you accept
|
|
this policy.
|
|
<p><b>Segmentation Faults</b>
|
|
<p>Rsyslog has a very rapid development process, complex capabilities and now gradually gets
|
|
more and more exposure. While we are happy about this, it also has some bad effects: some
|
|
deployment scenarios have probably never been tested and it may be impossible to test
|
|
them for the development team because of resources needed. So while we try to avoid this,
|
|
you may see a serious problem during deployments in demanding, non-standard, environments
|
|
(hopefully not with a stable version, but chances are good you'll run into troubles with
|
|
the development versions).
|
|
<p>Active support from the user base is very important to help us track down those things.
|
|
Most often, serious problems are the result of some memory misadressing. During development,
|
|
we routinely use valgrind, a very well and capable memory debugger. This helps us to create
|
|
pretty clean code. But valgrind can not detect everything, most importantly not code pathes
|
|
that are never executed. So of most use for us is information about aborts and abort locations.
|
|
<p>Unforutnately, faults rooted in adressing errors typically show up only later, so the
|
|
actual abort location is in an unrelated spot. To help track down the original spot,
|
|
<a href="http://www.gnu.org/software/hello/manual/libc/Heap-Consistency-Checking.html">libc
|
|
later than 5.4.23 offers support</a> for finding, and possible temporary relief from it,
|
|
by means of the MALLOC_CHECK_ environment variable. Setting it to 2 is a useful troubleshooting
|
|
aid for us. It will make the program abort as soon as the check routines detect anything
|
|
suspicious (unfortunately, this may still not be the root cause, but hopefully closer to it).
|
|
Setting it to 0 may even make some problems disappear (but it will NOT fix them!).
|
|
With functionality comes cost, and so exporting MALLOC_CHECK_ without need comes at
|
|
a performance penalty. However, we strongly recommend adding this instrumentation to your
|
|
test environment should you see any serious problems. Chances are good it will help us
|
|
interpret a dump better, and thus be able to quicker craft a fix.
|
|
<p>In order to get useful information, we need some backtrace of the abort. First, you need
|
|
to make sure that a core file is created. Under Fedora, for example, that means you need
|
|
to have an "ulimit -c unlimited" in place.
|
|
<p>Now let's assume you got a core file (e.g. in /core.1234). So what to do next? Sending a
|
|
core file to us is most often pointless - we need to have the exact same system configuration in
|
|
order to interpret it correctly. Obviously, chances are extremely slim for this to be. So we would
|
|
appreciate if you could extract the most important information. This is done as follows:
|
|
<ul>
|
|
<li>$gdb /path/to/rsyslogd
|
|
<li>$info thread
|
|
<li>you'll see a number of threads (in the range 0 to n with n being the highest number). For
|
|
<b>each</b> of them, do the following (let's assume that i is the thread number):
|
|
<ul>
|
|
<li>$ thread i (e.g. thread 0, thread 1, ...)
|
|
<li>$bt
|
|
</ul>
|
|
<li>then you can quit gdb with "$q"
|
|
</ul>
|
|
<p>Then please send all information that gdb spit out to the development team. It is best to first
|
|
ask on the forum or mailing list on how to do that. The developers will keep in contact with you
|
|
and, I fear, will probably ask for other things as well ;)
|
|
<p>Note that we strive for highest reliability of the engine even in unusual deployment scenarios.
|
|
Unfortunately, this is hard to achieve, especially with limited resources. So we are depending on
|
|
cooperation from users. This is your chance to make a big contribution to the project without the
|
|
need to program or do anything else except get a problem solved ;)
|
|
<p>[<a href="manual.html">manual index</a>]
|
|
[<a href="http://www.rsyslog.com/">rsyslog site</a>]</p>
|
|
<p><font size="2">This documentation is part of the
|
|
<a href="http://www.rsyslog.com/">rsyslog</a> project.<br>
|
|
Copyright © 2008-2010 by <a href="http://www.gerhards.net/rainer">Rainer Gerhards</a> and
|
|
<a href="http://www.adiscon.com/">Adiscon</a>. Released under the GNU GPL
|
|
version 3 or higher.</font></p>
|
|
</body>
|
|
</html>
|
|
|