2543 Commits

Author SHA1 Message Date
frikilax
f4ea54731a MMDBLOOKUP:: ALLOW RELOAD OF MMDB FILE ON SIGHUP (#8) 2021-04-09 19:45:39 +02:00
Rainer Gerhards
2381890467
Merge pull request #4551 from julthomas/dev/jth/fmunflatten
fmunflatten: rscript function to unflatten keys in a JSON object
2021-04-06 19:14:23 +02:00
Rainer Gerhards
599c388ef7
Merge pull request #4557 from rgerhards/i4271
clarify some debug output in regard to config include load order
2021-03-29 13:08:59 +02:00
Rainer Gerhards
14aabc691e
testbench: add check for proper config include file order
This commit adds a check that include files are processed in the
proper order.

It also slightly changes some text that seemed to cause the wrong
impression that include files were processed in the wrong order.
Right the contrary is the case, as config files are being put on
a stack before they are processed.

closes https://github.com/rsyslog/rsyslog/issues/4271
2021-03-29 10:24:53 +02:00
Rainer Gerhards
b5d6451a30
Merge pull request #4554 from yifan19/busybox-logger
testbench: Skip tests that need logger -d for busybox
2021-03-29 08:21:46 +02:00
Julien Thomas
9b779a319e tests: add test case for rscript function unflatten
You may want to review some of these tests.

The "conflict" and "key truncated" test cases are checking for a debug
message (rsyslog debug), this may be overkill. Also some tests require
jq in $PATH, to get deterministic JSON output (sorted keys). If jq is
not found, the tests are skipped.
2021-03-28 13:10:02 +02:00
Yi Fan Yu
3b4f8c1894 testbench: Skip tests that need logger -d for busybox
Busybox logger doesn't have the -d flag.

Skip instead of failing the tests that need -d.
Also fix typo s/-p/-d.

busybox reference:
https://busybox.net/downloads/BusyBox.html#logger

Signed-off-by: Yi Fan Yu <yifan.yu@windriver.com>
2021-03-10 22:33:59 -05:00
Yi Fan Yu
0aac143f77 testbench: Remove python shebang in tests/testsuites
py files should be run as ${PYTHON} *.py

reference issue: #3853

Signed-off-by: Yi Fan Yu <yifan.yu@windriver.com>
2021-03-10 14:09:58 -05:00
b0b6320c2a testbench: Enhanced error detection in imrelp-tls-cfgcmd.sh
This solves test failures on some platforms like solaris.

Also added _attribute__((unused)) into SetTlsVerifyDepth in
nsd_ptcp.c to fix compiler errors with gcc9.

closes: https://github.com/rsyslog/rsyslog/issues/4544
2021-03-09 09:18:08 +01:00
b93a3dbe63 testbench: changed tlscommands for librelp tls tests.
tlscommands for the following tests have been changed:
- imrelp-tls-cfgcmd.sh
- sndrcv_relp_tls-cfgcmd.sh

closes: https://github.com/rsyslog/rsyslog/issues/4534
2021-02-16 12:27:54 +01:00
Rainer Gerhards
591e6b1f0f
omfwd: add stats counter for sent bytes
Thanks to John Chivian for suggesting this feature.
2021-02-15 10:54:09 +01:00
Rainer Gerhards
f2710f59b0
CI: switch some tests from Ubuntu 18 to 20
So we can use the more modern tools that comes with it.

We also change some plumbing as newer version of TSAN require
that.

Note: we disable some kafka TSAN/ASAN tests as librdkafka
throws errors which we currently cannot suppress. This issue
will be addressed in separate patch.
2021-01-29 13:19:26 +01:00
Rainer Gerhards
4a0c28a10d
testbench modernization
Bump dependency versions, use newer distro versions for some tests.
Make kafka distcheck separate to help diagnose flaky kafka tests.
2021-01-18 08:58:07 +01:00
Rainer Gerhards
17d66bebfb testbench: fix invalid sequence of kafka tests runs
kafka tests can not run well in parallel (mostly due to ressource
constraints on CI machines). Accidentally, this was not enforced for
one of the tests. That could lead to random failures and false positives.
2021-01-15 12:23:31 +01:00
Rainer Gerhards
e64edeca2b
testbench: fix kafkacat issues
The kafkacat tool has an upper limit of how many messages it can send
at once. Going over that limit causes messages loss. The exact limit
seems to depend on the environment. This cases testbench false positives.

This commit fixes two related issues:
- errors during kafkacat run were not detected - this has been added
- we now have a "max messages at once" setting, after which kafkacat
  is restarted for the next batch of messages. It currently is set
  to 25,000 msgs per incarnation. All tests loop now to send the
  required number of messages. This has been fixed at the testbench
  framework level, so no need to adjust individual tests.
2021-01-14 11:43:34 +01:00
Rainer Gerhards
0d5088f127
testbench: fix year-dependendt clickhouse test
A test had the year value hardcoded and as such failed whenever the
year changed. This patch corrects that.
2021-01-14 10:38:16 +01:00
Rainer Gerhards
78b4428270
Merge pull request #4476 from rgerhards/i4446
testbench: fix missing test file
2020-12-07 13:52:39 +01:00
Rainer Gerhards
2ed8a931eb
testbench bugfix: some tests did not work in make distcheck
- certificate file missing in dist tarball
- some test cases did not properly specify path to cert file
- also bumping some zookeeper component versions

Thanks to Michael Biebl for alerting us and providing part of
the fix.

closes https://github.com/rsyslog/rsyslog/issues/4446
2020-12-07 12:34:59 +01:00
Rainer Gerhards
3238729d96
immark: rewrite with many improvements
- mark message text can now be specified
- support for rulesets
- support for using syslog API vs. regular internal interface
- support for output template system
- ability to specify is mark message flag can be set
- minor changes and improvements
2020-12-07 09:45:24 +01:00
Rainer Gerhards
1c85aa8a59
add new system property $now-unixtimestamp
Among others, this may be used as a monotonic counter
for doing load-balancing and other things.

Thanks to Nicholas Brown for suggesting this feature.
2020-11-04 13:02:22 +01:00
b5356f8db7 tls: Allow calling PermitExpiredCerts with NULL parameter
In order to set the default PermitExpiredCerts handling (Denied),
we need to call PermitExpiredCerts with NULL parameter.

testbench: Add test to check expired handling in omfwd

debug: Fix dbgprintf calls with possible NULL character parameters

closes: https://github.com/rsyslog/rsyslog/issues/4425
2020-10-28 09:18:25 +01:00
Rainer Gerhards
d247c31888
add 'exists()' script function to check if variable exists
This implements a way to check if rsyslog variables (e.g. '$!path!var') is
currently set of not.

Sample: if exists($!somevar) then ...

closes https://github.com/rsyslog/rsyslog/issues/4385
2020-10-05 12:38:58 +02:00
Rainer Gerhards
57a15d81d8
Merge pull request #4420 from aaronmaxlevy/issue_4408_fix
Replaced eCmdHdlrPositiveInt with eCmdHdlrNonNegInt where default is 0
2020-09-14 11:24:17 +02:00
Rainer Gerhards
500713dbde
Merge pull request #4301 from n2yen/imhttp
http input module
2020-09-14 08:22:33 +02:00
Aaron Levy
2e3b767d03 Replaced eCmdHdlrPositiveInt with eCmdHdlrNonNegInt where default is 0 2020-09-13 15:56:07 -07:00
Rainer Gerhards
6e810d3f5e
CI: run testbench on ubuntu 20 with github action 2020-09-07 11:21:30 +02:00
Rainer Gerhards
e0b8634010
testbench fix: python 3 incompatibility 2020-09-03 12:58:57 +02:00
Rainer Gerhards
d23e73c782
Merge pull request #4394 from rgerhards/i4282
Queue: add test for segfault issue 4282
2020-09-02 10:15:06 +02:00
Rainer Gerhards
6763185783
core bugfix: segfault if disk-queue file cannot be created
When using Disk Queue and a queue.filename that can not be created
by rsyslog, the service does not switch to another queue type as
supposed to and crashes at a later step.

closes: https://github.com/rsyslog/rsyslog/issues/4282
2020-09-02 09:20:10 +02:00
Rainer Gerhards
72e91f6109
Merge pull request #4402 from rgerhards/TB-simplify
testbench: simplify test ID generation a bit
2020-09-01 15:05:13 +02:00
Rainer Gerhards
19e80e9e5c testbench: simplify test ID generation a bit 2020-09-01 14:33:41 +02:00
9513b7aa40 testbench: Add helpertool to check for minimum librelp version
- add testcases for imrelp / omrelp chained certificates
  for new librelp version 1.7.0
- add chained certificate based on existing certs

closes: https://github.com/rsyslog/rsyslog/issues/4388
2020-08-31 11:39:25 +02:00
Rainer Gerhards
89990c2a36
Merge pull request #4370 from splbio/imptcp_max_sessions
Add max sessions for imptcp.c similar to imtcp.c to prevent OOM
2020-08-25 10:30:16 +02:00
Nelson Yen
6102f6083c initial implemenation http input module
- uses http library to provide http input.
user would need to configure an 'endpoint' as input, along
with a ruleset, defining how the input should be routed in
rsyslog.

bugfix, free dynamic buf if created, don't echo to client data.

do data framing using newline character.
more input options support:
  - input name
  - flowcontrol

add support for gzip content support, and other options

- gzip content
- parse linefeeds by default, option to ignore linefeeds

add support for gzip content support, and other options

- gzip content
- parse linefeeds by default, option to ignore linefeeds

WIP - misc edits enable port, documentroot module parameter

fix overloaded stack issue

update tests to use available port instead of hard-coded one.

Add 'octet counted framing' support

- option is "SupportOctetCountedFraming", currently 'off' by
default.

update imhttp-getrequest-file.sh test to use $srcdir

imhttp - support multi-threaded connection contexts.

- tests for large data posts

- Add header data as metadata option

move mg_start into activatecnf instead of in runinput
2020-08-23 07:24:48 -07:00
Rainer Gerhards
5d78a8a5dd
Merge pull request #4380 from rgerhards/i3698
Increase timeouts in imfile-basic-2GB-file and imfile-truncate-2GB-file
2020-08-21 09:32:42 +02:00
Michael Biebl
effe40d4b4
Increase timeouts in imfile-basic-2GB-file, imfile-truncate-2GB-file
Those tests can take a long time, especially on slow architectures like
armhf, so bump the test timeouts considerably.

closes https://github.com/rsyslog/rsyslog/issues/3698
2020-08-20 15:19:02 +02:00
Rainer Gerhards
e366276aca
testbench: add additional debug support for hard to find issue
Some imfile tests have a very hard to find sporadic failure, which
is irreproducible on a regular workstation. This enhances the
testbench's debug support in the hope to provide better diagnostics
when the issue occurs in regular CI runs.
2020-08-14 15:20:54 +02:00
Alfred Perlstein
6c50426dbf Add max sessions for imptcp.c similar to imtcp.c
The max is per-instance, not global across all instances.

There is also a bugfix where if epoll failed I think we could leave a
session linked in the list of sessions, this code unlinks it.
2020-08-11 19:06:37 -07:00
Rainer Gerhards
99d950203b
Merge pull request #4343 from VultureProject/impcap-merge-upstream
IMPCAP::Fixes: segfault, memory and build corrections (resolves #4332)
2020-08-06 14:52:07 +02:00
4e5741807e testbench: set msg size to 64kb for sndrcv_omudpspoof-bigmsg.sh test 2020-07-20 08:35:57 +02:00
Rainer Gerhards
94c4a871d0
imklog: implement ruleset support
see also: https://github.com/rsyslog/rsyslog/issues/4344#issuecomment-658001854
see also: https://github.com/rsyslog/rsyslog/issues/106
2020-07-14 10:38:42 +02:00
frikilax
62dd906245 IMPCAP::Fixes: segfault, memory and build corrections (resolves #4332)
- fix bug in ethernet packets parsing
  - fix removes build error with gcc10: 'multiple definition of...'
- resolve memory leak during interface init failure (device not freed after post-create error)
- add test 'impcap_bug_ether' to prove ethernet parser fix is working
2020-07-06 10:49:47 +02:00
Rainer Gerhards
8ff2179304
Merge pull request #4321 from alorbach/pr-issue-4319
openssl: Fixed memory leak when tls handshake failed.
2020-06-22 10:45:07 +02:00
6d0fea9b69 openssl: Fixed memory leak when tls handshake failed.
Added testcase for ossl memory leak on failed handshake.

closes: https://github.com/rsyslog/rsyslog/issues/4319
2020-06-09 10:01:53 +02:00
frikilax
76318ad926 NEW INPUT MODULE:: impcap, network packets input parser + TESTS 2020-06-05 17:41:01 +02:00
Rainer Gerhards
d3f3cf54b4
testbench: add test case for cat non-existing-file
see also https://github.com/rsyslog/rsyslog/issues/4290
2020-05-22 10:13:30 +02:00
82621cdd14 imtcp: fix octet framing/stuffing problem with discardTruncatedMsg on
When "discardTruncatedMsg" is enabled in imtcp, messages were incorrectly
skipped if the last character before the truncation was the LFdelimiter.

Add two tests for octet stuffing and framing.

closes: https://github.com/rsyslog/rsyslog/issues/4281
2020-05-15 09:56:44 +02:00
Rainer Gerhards
0ae9a7a3d0
Merge pull request #4193 from jmarckel/bugfix/bug_2133
imjournal: remove strcat call
2020-05-12 12:51:10 +02:00
Rainer Gerhards
da7ca37c10 CI: fix problems with ElasticSearch download
ElasticSearch can no longer be reliably downloaded, we receive error
451 - prohibited for legal reasons. This causes false positive test
failures.

We have solved this by moving the required tarballs to rsyslog.com and
download them now from there.
2020-05-12 11:09:42 +02:00
Rainer Gerhards
6aed1b4975
Merge pull request #4219 from callmegar/master
omhttp: Adding multiple http headers capability
2020-05-11 12:12:55 +02:00