2503 Commits

Author SHA1 Message Date
8f2be648cd imfile: Fixed wildcard detection issue on path wildcards
Wildcards mode was not properly detected when wildcards
were only used in a directory name on startup.
This caused imfile not to create a propper dynamic filewatch.

Closes: https://github.com/rsyslog/rsyslog/issues/1672
2017-07-25 15:42:06 +02:00
Rainer Gerhards
80148e0bcd Merge pull request #1679 from rgerhards/imptcp-robustness
Imptcp: robustness fix and new stats counters
2017-07-20 12:26:23 +02:00
Rainer Gerhards
9375e03d53 imptcp regression: do not acces session object after delete
bug in non-released code, but too much of a mess to fix in git history,
so doing a fix commit instead
2017-07-19 15:15:10 +02:00
Rainer Gerhards
058961d786 Merge pull request #1677 from PascalWithopf/i-1664
imptcp: framing/size errmsgs contain remote peer
2017-07-19 15:07:30 +02:00
Rainer Gerhards
b1d9cb4452 Merge pull request #1674 from PascalWithopf/i-477
imklog: Fix permitnonkernelfacility not working
2017-07-19 14:54:45 +02:00
Rainer Gerhards
960aa3cd9d Merge pull request #1667 from rgerhards/i-1661
impstats: add new ressoure counter "openfiles"
2017-07-19 14:51:41 +02:00
Rainer Gerhards
6770cffe0e Merge branch 'imptcp-robustness-v8.25.0' into imptcp-robustness 2017-07-19 14:24:07 +02:00
Rainer Gerhards
3f484f2808 imptcp: add session statistics conunter
- session.opened
- session.openfailed
- session.closed

These are useful for monitoring, capacity planning and troubleshooting. They
are reported on a per-listener basis.
2017-07-19 14:14:42 +02:00
Rainer Gerhards
79b4e1eae4 imptcp: cosmetic fix 2017-07-19 14:03:17 +02:00
Rainer Gerhards
8350374ee9 imptcp: fix potential socket leak on session shutdown
imptcp first tries to remove a to-be-shut-down socket from the
epoll set, and errors out if that does not work. In that case, the
underlying socket will be leaked.

This patch refactors the code; most importantly, it is not necessary
to remove the socket from the epoll set, as this happens automatically
on close. As such, we simply remove that part of the code, which
also removes the root cause of the socket leak.
2017-07-19 13:50:15 +02:00
PascalWithopf
792746a035 imptcp: framing/size errmsgs contain remote peer
closes https://github.com/rsyslog/rsyslog/issues/1664
2017-07-19 11:24:38 +02:00
Rainer Gerhards
231ab1d1cb Merge pull request #1669 from jgerhards/newdelimiter
imptcp: add parameter "multiline"
2017-07-18 14:45:04 +02:00
Rainer Gerhards
84cb9eb066 Merge pull request #1657 from SirWumpus/imfile_count
Fix behaviour of PersistStateInterval.
2017-07-18 14:38:45 +02:00
Trent Piepho
5c35619385 imklog: Fix permitnonkernelfacility not working
permitnonkernelfacility doesn't work when the new configuration syntax
is used, e.g. 'module(load="imklog" permitnonkernelfacility="on")'.
It does work with the old syntax, e.g. '$KLogPermitNonKernelFacility
on'

This is because the old style config is stored in a static global
struct "cs", while the new style config is passed in as a pointer.
Code in imklog will put old style config entries into the new config
struct, and almost all the code in imklog uses the new config struct
like it should.  Except for a check for bPermitNonKernel in Syslog()
that continued to use the static global that only has old style
configs.

Fix this by passing pModConf down into Syslog() and using that in
place of the static global.

closes https://github.com/rsyslog/rsyslog/issues/477
2017-07-18 14:31:29 +02:00
Jan Gerhards
250957d120 imptcp: add parameter "multiline"
experimental new functionality: modify octet-stuffed framing to
heuristically detect embedded newlines
2017-07-16 13:00:00 +02:00
Rainer Gerhards
8de60bf9c1 impstats: add new ressoure counter "openfiles"
closes https://github.com/rsyslog/rsyslog/issues/1661
2017-07-13 18:15:43 +02:00
Rainer Gerhards
2401385a48 Merge branch 'i-1665-v8.25.0' into i-1655 2017-07-12 14:03:01 +02:00
Rainer Gerhards
79032d68ca imudp: add "udp.sendbuf" parameter 2017-07-12 13:30:46 +02:00
Anthony Howe
0b514add1c Fix behaviour of PersistStateInterval. If PersistStateInterval=1, then
each log line read should cause the state file to be updated, but this
was not happening because nRecords was being post-increment.
2017-07-10 14:32:47 -04:00
PascalWithopf
2f40554acb codestyle: correct too long lines 2017-06-29 09:28:21 +02:00
Rainer Gerhards
48b80d3cc8 Merge pull request #1618 from infracaninophile/empty-argv
Empty argv
2017-06-27 15:37:55 +02:00
Rainer Gerhards
2ccdf61f24 Merge pull request #1637 from rgerhards/add-kafkafix
omkafka: Adding messages to failed list while doAction is in retry mode
2017-06-27 15:36:57 +02:00
c148f300cf omkafka: Adding messages to failed list while doAction is in retry mode
Bug was found using new tests.
2017-06-27 14:47:25 +02:00
PascalWithopf
5d0bcd69a2 gtls: improve error if certificate file can't be opened
closes https://github.com/rsyslog/rsyslog/issues/1121
2017-06-27 11:04:15 +02:00
Rainer Gerhards
978ddf536c Merge pull request #1620 from jgerhards/i1102
multile modules: add better error messages when regcomp is failing
2017-06-26 15:42:25 +02:00
Jan Gerhards
779fce20b3 multiple modules: add better error messages when regcomp is failing
see also: https://github.com/rsyslog/rsyslog/issues/1102
2017-06-26 11:03:00 +02:00
Rainer Gerhards
99361579fe Reported by: Chun-Tien Chang) <tcs@kitty.2y.idv.tw>
Instead of calling malloc(0) just declare an array of char* and
initialize it to a NULL terminated list with zero items.
2017-06-25 12:23:56 +01:00
f19ead2551 imfile: Added test for dynamic renamed files (wildcard)
Added check for IN_MOVED_TO in in_handleDirEvent. Statefilename and
event cookie are saved in IN_MOVE_FROM event in a linkedlist now,
so the statefile can be renamed to a new name in IN_MOVED_TO event.

Closes https://github.com/rsyslog/rsyslog/issues/1417
2017-06-23 11:21:28 +02:00
Rainer Gerhards
d060be7920 Merge pull request #1609 from alorbach/kafka-extension-omkafka
Kafka extension omkafka
2017-06-21 08:16:53 +02:00
8ecf0aeb34 kafka: Added option enable static linking of librdkafa (disabled by default).
Also saving failed msgs now when kafka produce fails. No message loss should
be possible anymore.

Also removed old kafka tests and readded new kafka tests.
If kafka instances cannot be started (happens from time to time when
kafka/zookepper sync fails), tests will be skipped instead of failing.

Closes https://github.com/rsyslog/rsyslog/issues/1559
Closes https://github.com/rsyslog/rsyslog/issues/1584
Closes https://github.com/rsyslog/rsyslog/issues/1515
May fix https://github.com/rsyslog/rsyslog/issues/1230
Closes https://github.com/rsyslog/rsyslog/issues/1052
2017-06-19 17:14:35 +02:00
PascalWithopf
261d66ad95 mmnormalize: delete dbgprintf's left from testing 2017-06-12 16:00:10 +02:00
Rainer Gerhards
202760c1de Merge pull request #1591 from radosroka/pid-fallback
imjournal: _PID fallback
2017-06-09 14:44:09 +02:00
b71c1871d0 omkafka: Added new options to save and load failed messages.
When enabled, failed messages are kept in memory queue and saved on
shutdown and loaded on startup. This prevents message loss of failed
messages.

The failed items are saved in a simple Linked List and automatically
being reprocessed on resume or doaction.

Load/Save logik implemented.

This should also remove the last possible message loss from issue:
https://github.com/rsyslog/rsyslog/issues/1052
2017-06-08 15:35:45 +02:00
e284ab2a20 omkafka: Added optional option "resubmitOnFailure" to resend on msg failure.
When a delivery callback returns with an error, omkafka can put the
message back into the rdkafka queue for transmission.

This should also remove some of the message loss from issue:
https://github.com/rsyslog/rsyslog/issues/1052
2017-06-08 15:35:45 +02:00
702df5eed6 omkafka: Implemented newer delivery callback from librdkafka.
This somehow elimitated a lot of failing messages which were not
processed on shutdown (flush) before. The old callback API was
depreceated.

This should also elimitate most of the message loss when
kafka queue is full from issue:
https://github.com/rsyslog/rsyslog/issues/1052

The new code was tested with
queue.buffering.max.messages=100 and 2000 injected messages.

Testbench: Try starting kafka server second time if failed on
first try. Sometimes the broker is still registered in zookeeper.
2017-06-08 15:35:45 +02:00
732d0e21a7 omkafka: Enhanced support to detect kafka broker problems.
Also fixed tryresume and doAction handling when action is
in suspend state.

Calling for Callbacks more often.

Adjusted kafka fail test settings.

diag.sh script detects of kafka server comes up proberly

Added liblz4 to dependencies for omkafka/imkafka needed
for static linking.

This addresses the omkafka queue problem from issue:
https://github.com/rsyslog/rsyslog/issues/1052
Yet it does not fully solve the message problem yet,
but I am working on this.
2017-06-08 15:35:45 +02:00
fa775e7590 omkafka: Added detection for broker failure and suspending/resume the action
imkafka, omkafka: switched from dynamic to static linking

Also added another test to test a typical kafka cluster failure environment.
2017-06-08 15:35:45 +02:00
43c782dc1c omkafka: Added support to set kafka timestamp
We are using rd_kafka_producev now.
Only works with librdkafka higher than 0.9.4. When using older vresion of
librdkafka, old api is being used (rd_kafka_produce).

closes https://github.com/rsyslog/rsyslog/issues/1559
2017-06-08 15:35:45 +02:00
PascalWithopf
7982f50675 omelasticsearch: replace cJSON with libfastjson 2017-06-06 08:22:35 +02:00
Rainer Gerhards
51576d1900 Merge pull request #1598 from rgerhards/i-1445
imfile: multiline timeout does not work after rsyslog restart
2017-06-01 14:30:00 +02:00
Rainer Gerhards
97f1c3f67f Merge pull request #1589 from rgerhards/mmdb-enh
mmdblookup: enhancements and fixes
2017-06-01 14:26:28 +02:00
Rainer Gerhards
82bfcf8453 imfile: multiline timeout does not work after rsyslog restart
The timeout feature for multiline reads does not correctly work
for files for which a state file existed. This is usually the
case for files that had been processed by a previous run and
that still exist on the new start. For all other files,
especially those monitored by a wildcard and newly created after
the rsyslog start, timeout worked as expected.

closes https://github.com/rsyslog/rsyslog/issues/1445
2017-06-01 10:12:33 +02:00
Rainer Gerhards
5ccb9f69a4 Merge pull request #1525 from nbrownus/imrelp-max-line
Expose configuration for max data size in librelp
2017-06-01 08:47:54 +02:00
Radovan Sroka
89ce7cb535 _PID fallback
- added fallback for _PID proprety when SYSLOG_PID is not available
- introduced new option "usepid" which sets which property should
  rsyslog use, it has 3 states system|syslog|both, default is both
- deprecated "usepidfromsystem" option, still can be used
  and override the "usepid"
- it is possible to revert previous default with usepid="syslog"
2017-05-31 15:24:45 +02:00
Rainer Gerhards
aecdbdf360 mmdblookup: upgrade from contrib module to fully supported one 2017-05-31 13:43:05 +02:00
Nate Brown
270d922961 Expose configuration for max data size in librelp 2017-05-30 20:44:21 -07:00
zacharyzhao
1bce2ebbe7 Handle inotify event IN_MOVED_TO as IN_CREATE 2017-05-30 18:22:28 +08:00
Rainer Gerhards
bd7175cf09 Merge pull request #1576 from rgerhards/travis-librdkafka
Travis librdkafka
2017-05-20 17:21:31 +02:00
Rainer Gerhards
4266ca066d Merge pull request #1570 from PascalWithopf/i-1552
imfile: add posibility to truncate msg and process the truncated part…
2017-05-20 10:23:13 +02:00
Rainer Gerhards
62074fcc23 imkafka: replace deprecated function call
Most importantly, the deprecated call had rate-limited inbound
messages using the default rate limiter (which is wrong for
these types of modules).
2017-05-19 17:55:12 +02:00