2718 Commits

Author SHA1 Message Date
Rainer Gerhards
6080121ab7
config: implement script-equavalent for $PrivDrop* statements
closes https://github.com/rsyslog/rsyslog/issues/891
2021-08-23 14:10:16 +02:00
Rainer Gerhards
1fc8a08c6f
testbench: fix privdrop tests under root user
When running under root, the privdrop tests did not properly work. This
patch fixes the issue and skips test where necessary.

This also includes some modernization of the related tests.

closes https://github.com/rsyslog/rsyslog/issues/4619
2021-08-23 09:41:43 +02:00
Rainer Gerhards
0c9b169a64
Merge pull request #4612 from n2yen/dev2-percentile
Percentile module to track percentile metrics via impstats
2021-08-16 08:31:21 +02:00
Rainer Gerhards
0461bacd4f
Merge pull request #4626 from n2yen/dev-ignore-older
imfile add `ignoreolderthanoption`
2021-08-12 10:18:25 +02:00
Nelson Yen
db44a20a25
imfile add ignoreolderthanoption
Submit on behalf of the primary author @yanjunli76

instructs imfile not to ingest a file that has not been modified in the
specified number of seconds.
2021-08-11 08:11:55 -07:00
Rainer Gerhards
a25c01c3d2
Merge pull request #4633 from pearseimperva/pearse/fix-deleteStateOnFileDelete
Fixing the deleteStateOnFileDelete option
2021-08-09 18:52:42 +02:00
Rainer Gerhards
5f6fc91417
Merge pull request #4656 from rgerhards/i4619
testbench: guard tests against unexpected warning messages
2021-08-09 15:05:15 +02:00
Rainer Gerhards
6bba5a7577
testbench: guard tests against unexpected warning messages
Practice has shown that environment-induced error messages may
appear during test runs. For example, incorrect resolver settings
may cause name resolution warnings or errors. These are unrelated
to the test itself.

This patch enables tests that are otherwise unaffected by the faillures
to continue and provide "pass" test result.

see also https://github.com/rsyslog/rsyslog/issues/4619
2021-08-09 10:41:09 +02:00
Rainer Gerhards
38856e6989
Merge pull request #4628 from rgerhards/ci-add-test
CI: add test for imtcp not correctly starting up
2021-08-09 09:29:34 +02:00
pearse
fca8e14d65 fixing the deleteStateOnFileDelete option 2021-08-06 22:25:27 +00:00
Rainer Gerhards
be22ec54ad
CI: add test for imtcp not correctly starting up and a Solaris fix
Test should have been added to commit 3d23c7ac8aea but was forgotten.

This test also showed an issue on Solaris, where NULL values cannot
be used in printf functions. This has also been addressed.

see also: https://github.com/rsyslog/rsyslog/pull/4627
2021-08-06 12:42:58 +02:00
Rainer Gerhards
39e81e133b
omfwd: add capability for action-specific TLS certificate settings
This permits to override the global definitions for TLS certificates
at the action() level.
2021-08-06 09:00:24 +02:00
Rainer Gerhards
13f5dfe2e2
imtcp: permit to use different certificate files per input/action 2021-08-04 10:23:24 +02:00
Nelson Yen
80e4ce1769 mmkubernetes fix for apiserver error handling
submit on behalf of @abwaheed
- Added graceful handling of apiserver errors with unexpected responses,
  i.e., anything other than 200, 404, or 429. Idea is that apiserver
  transient error state will recover. We don't want mmkubernetes to miss
  metadata resolution for containers that don't have cached metadata.
  During these transient error states, mmkubernetes will provide basic
  container file path based resolution of namespace and pod metadata for
  new pods whose metadata is not yet cached. After this error state
  recovers, mmkubernetes is expected to resume its metadata resolution as
  expected.
- Added a unit test case for apiserver return 500 with changes to mock server
-  Fixed existing unit test that was failing due to missing expected results file
-  Added mmkubernetes unit tests to testbench
2021-06-29 18:14:25 -07:00
Rainer Gerhards
cb686e96b8
testbench: simplify test
make easier to understand and less verbose
2021-06-24 14:46:11 +02:00
Rainer Gerhards
b78fdde38a
Merge pull request #4407 from alorbach/pr-issue-4406
testbench: Add testcase for librelp EINPROGRESS issue
2021-06-23 09:36:50 +02:00
Nelson Yen
91a2049877
percentile module to track percentile metrics via impstats
Brief overview:
TO configure tracking percentile metrics in rainerscript:
User would need to define:
  - which percentile to track, such as [p50, p99, etc.]
  - window size - note, this correlates directly with memory usage to
  track the percentiles.

To track a value, user would call built-in function `percentile_observe()` in their configurations to
record an integer value, and percentile metrics would be emitted every
impstats interval.
2021-06-23 00:14:39 -07:00
bda68f999d omhttp: Fix dynrestpath param in batch mode
When batchmode was used, the templates could not be used to
expand dynrestpath. We are now storing the restpath param
within the batch data if we are in batch mode.

- testbench: Added tests for omhttp dynrestpath param
  Testing setting the restpath by template - one with batch mode
  and one without batch mode
- When we are in batch mode, and the restpath value changes, the
  batch is submitted and reinitialized

closes: https://github.com/rsyslog/rsyslog/issues/4567
2021-06-16 11:43:08 +02:00
6e2a79376f testbench: Add testcase for librelp EINPROGRESS issue
closes: https://github.com/rsyslog/rsyslog/issues/4406
See also: https://github.com/rsyslog/librelp/issues/207
2021-06-16 09:40:29 +02:00
Rainer Gerhards
8d48ecf10a
imtcp: add more override config params to input() 2021-05-10 12:30:17 +02:00
Rainer Gerhards
a08591be5d
imtcp: add stream driver parameter to input() configuration
This permits to have different inputs use different stream drivers
and stream driver parameters.

closes https://github.com/rsyslog/rsyslog/issues/3727
2021-05-05 12:57:09 +02:00
Rainer Gerhards
3470728d44
further refactoring of tcpsrv parameter passing
Some further stream-lining and cleanup of paramter passing. This
levels ground for more substantial changes to the imtcp/tcpsrv
interaction.

see also https://github.com/rsyslog/rsyslog/issues/3727#issuecomment-525705318
2021-04-26 09:05:01 +02:00
Rainer Gerhards
39896ca622
config script: add re_match_i() and re_extract_i() functions
This provides case-insensitive regex functionality.

closes https://github.com/rsyslog/rsyslog/issues/4429
2021-04-21 12:23:46 +02:00
Rainer Gerhards
1bf65fdb52
Merge pull request #4542 from VultureProject/mmdblookup_reload
MMDBLOOKUP::Improvement:: allow mmdb to be reloaded on HUP signal
2021-04-19 10:38:26 +02:00
Nelson Yen
af8085fc83
Add built-in function get_property() to access property vars
Provides ability to evaluate a rsyslog variable using dynamically
evaluated parameters.
  1st param is the rsyslog param, 2nd param is a key, can be an array
  index or key string.

Useful for accessing json sub-objects, where a key
needs to be evaluated at runtime. Can be used to access arrays as well.

see tests for examples

remove unnecessary escape char
2021-04-15 11:52:15 -07:00
frikilax
f4ea54731a MMDBLOOKUP:: ALLOW RELOAD OF MMDB FILE ON SIGHUP (#8) 2021-04-09 19:45:39 +02:00
Rainer Gerhards
2381890467
Merge pull request #4551 from julthomas/dev/jth/fmunflatten
fmunflatten: rscript function to unflatten keys in a JSON object
2021-04-06 19:14:23 +02:00
Rainer Gerhards
599c388ef7
Merge pull request #4557 from rgerhards/i4271
clarify some debug output in regard to config include load order
2021-03-29 13:08:59 +02:00
Rainer Gerhards
14aabc691e
testbench: add check for proper config include file order
This commit adds a check that include files are processed in the
proper order.

It also slightly changes some text that seemed to cause the wrong
impression that include files were processed in the wrong order.
Right the contrary is the case, as config files are being put on
a stack before they are processed.

closes https://github.com/rsyslog/rsyslog/issues/4271
2021-03-29 10:24:53 +02:00
Rainer Gerhards
b5d6451a30
Merge pull request #4554 from yifan19/busybox-logger
testbench: Skip tests that need logger -d for busybox
2021-03-29 08:21:46 +02:00
Julien Thomas
9b779a319e tests: add test case for rscript function unflatten
You may want to review some of these tests.

The "conflict" and "key truncated" test cases are checking for a debug
message (rsyslog debug), this may be overkill. Also some tests require
jq in $PATH, to get deterministic JSON output (sorted keys). If jq is
not found, the tests are skipped.
2021-03-28 13:10:02 +02:00
Yi Fan Yu
3b4f8c1894 testbench: Skip tests that need logger -d for busybox
Busybox logger doesn't have the -d flag.

Skip instead of failing the tests that need -d.
Also fix typo s/-p/-d.

busybox reference:
https://busybox.net/downloads/BusyBox.html#logger

Signed-off-by: Yi Fan Yu <yifan.yu@windriver.com>
2021-03-10 22:33:59 -05:00
Yi Fan Yu
0aac143f77 testbench: Remove python shebang in tests/testsuites
py files should be run as ${PYTHON} *.py

reference issue: #3853

Signed-off-by: Yi Fan Yu <yifan.yu@windriver.com>
2021-03-10 14:09:58 -05:00
b0b6320c2a testbench: Enhanced error detection in imrelp-tls-cfgcmd.sh
This solves test failures on some platforms like solaris.

Also added _attribute__((unused)) into SetTlsVerifyDepth in
nsd_ptcp.c to fix compiler errors with gcc9.

closes: https://github.com/rsyslog/rsyslog/issues/4544
2021-03-09 09:18:08 +01:00
b93a3dbe63 testbench: changed tlscommands for librelp tls tests.
tlscommands for the following tests have been changed:
- imrelp-tls-cfgcmd.sh
- sndrcv_relp_tls-cfgcmd.sh

closes: https://github.com/rsyslog/rsyslog/issues/4534
2021-02-16 12:27:54 +01:00
Rainer Gerhards
591e6b1f0f
omfwd: add stats counter for sent bytes
Thanks to John Chivian for suggesting this feature.
2021-02-15 10:54:09 +01:00
Rainer Gerhards
f2710f59b0
CI: switch some tests from Ubuntu 18 to 20
So we can use the more modern tools that comes with it.

We also change some plumbing as newer version of TSAN require
that.

Note: we disable some kafka TSAN/ASAN tests as librdkafka
throws errors which we currently cannot suppress. This issue
will be addressed in separate patch.
2021-01-29 13:19:26 +01:00
Rainer Gerhards
4a0c28a10d
testbench modernization
Bump dependency versions, use newer distro versions for some tests.
Make kafka distcheck separate to help diagnose flaky kafka tests.
2021-01-18 08:58:07 +01:00
Rainer Gerhards
17d66bebfb testbench: fix invalid sequence of kafka tests runs
kafka tests can not run well in parallel (mostly due to ressource
constraints on CI machines). Accidentally, this was not enforced for
one of the tests. That could lead to random failures and false positives.
2021-01-15 12:23:31 +01:00
Rainer Gerhards
e64edeca2b
testbench: fix kafkacat issues
The kafkacat tool has an upper limit of how many messages it can send
at once. Going over that limit causes messages loss. The exact limit
seems to depend on the environment. This cases testbench false positives.

This commit fixes two related issues:
- errors during kafkacat run were not detected - this has been added
- we now have a "max messages at once" setting, after which kafkacat
  is restarted for the next batch of messages. It currently is set
  to 25,000 msgs per incarnation. All tests loop now to send the
  required number of messages. This has been fixed at the testbench
  framework level, so no need to adjust individual tests.
2021-01-14 11:43:34 +01:00
Rainer Gerhards
0d5088f127
testbench: fix year-dependendt clickhouse test
A test had the year value hardcoded and as such failed whenever the
year changed. This patch corrects that.
2021-01-14 10:38:16 +01:00
Rainer Gerhards
78b4428270
Merge pull request #4476 from rgerhards/i4446
testbench: fix missing test file
2020-12-07 13:52:39 +01:00
Rainer Gerhards
2ed8a931eb
testbench bugfix: some tests did not work in make distcheck
- certificate file missing in dist tarball
- some test cases did not properly specify path to cert file
- also bumping some zookeeper component versions

Thanks to Michael Biebl for alerting us and providing part of
the fix.

closes https://github.com/rsyslog/rsyslog/issues/4446
2020-12-07 12:34:59 +01:00
Rainer Gerhards
3238729d96
immark: rewrite with many improvements
- mark message text can now be specified
- support for rulesets
- support for using syslog API vs. regular internal interface
- support for output template system
- ability to specify is mark message flag can be set
- minor changes and improvements
2020-12-07 09:45:24 +01:00
Rainer Gerhards
1c85aa8a59
add new system property $now-unixtimestamp
Among others, this may be used as a monotonic counter
for doing load-balancing and other things.

Thanks to Nicholas Brown for suggesting this feature.
2020-11-04 13:02:22 +01:00
b5356f8db7 tls: Allow calling PermitExpiredCerts with NULL parameter
In order to set the default PermitExpiredCerts handling (Denied),
we need to call PermitExpiredCerts with NULL parameter.

testbench: Add test to check expired handling in omfwd

debug: Fix dbgprintf calls with possible NULL character parameters

closes: https://github.com/rsyslog/rsyslog/issues/4425
2020-10-28 09:18:25 +01:00
Rainer Gerhards
d247c31888
add 'exists()' script function to check if variable exists
This implements a way to check if rsyslog variables (e.g. '$!path!var') is
currently set of not.

Sample: if exists($!somevar) then ...

closes https://github.com/rsyslog/rsyslog/issues/4385
2020-10-05 12:38:58 +02:00
Rainer Gerhards
57a15d81d8
Merge pull request #4420 from aaronmaxlevy/issue_4408_fix
Replaced eCmdHdlrPositiveInt with eCmdHdlrNonNegInt where default is 0
2020-09-14 11:24:17 +02:00
Rainer Gerhards
500713dbde
Merge pull request #4301 from n2yen/imhttp
http input module
2020-09-14 08:22:33 +02:00
Aaron Levy
2e3b767d03 Replaced eCmdHdlrPositiveInt with eCmdHdlrNonNegInt where default is 0 2020-09-13 15:56:07 -07:00