479 Commits

Author SHA1 Message Date
Rainer Gerhards
e73c0bd136
testbench: make tests more robust against slow machines 2020-01-12 11:16:54 +01:00
Rainer Gerhards
52b609869d
Merge pull request #4093 from rgerhards/tb-q-shutdown-to
testbench: increase main msg queue default shutdown timeout
2020-01-08 15:44:33 +01:00
Rainer Gerhards
2377bfa0e7
imtcp: add legacy directive $inputtcpserverlistenportfile
This is added for testbench use - we need it to harden tests that
need to test legacy syntax.

The new directive is NOT intended for users. Thus it is intentionally
NOT DOCUMENTED.
2020-01-08 11:38:20 +01:00
Rainer Gerhards
73941eaad2
Merge pull request #4094 from rgerhards/tb-timeout
testbench: do not create hanging tool instances
2020-01-08 09:20:52 +01:00
Rainer Gerhards
77ee19d9b0
testbench: increase main msg queue default shutdown timeout
this is necessary for very slow test environments
2020-01-08 08:53:36 +01:00
Rainer Gerhards
bc71ab0738
testbench: use local kafka download cache if present 2020-01-07 11:13:17 +01:00
Rainer Gerhards
c8461990f6
testbench: do not create hanging tool instances
the omhttp test server hangs indefinitely if tests requiring
it fail. This is relevant if the environment ist not destructed
after CI run (the buildbot CI system does not do this on VMs).

To guard against this, we have added a timeout of 30 minutes to
ensure the tool is terminated.
2020-01-06 17:05:12 +01:00
Rainer Gerhards
3860122f41
testbench: improve speed of elasticsearch tests
we avoid unnecessary ElasticSearch restarts now. This also helps with flakes
on slow CI machines.
2020-01-01 12:23:50 +01:00
Josh Soref
af19128573 fix spelling errors in testbench 2019-12-27 13:57:25 +01:00
Rainer Gerhards
233fde2ec9
Merge pull request #3962 from alorbach/pr-issue-3959
relp: Add support setting openssl configuration commands.
2019-12-18 19:05:18 +01:00
Rainer Gerhards
c967622fdc
testbench: cleanup no longer needed debug instrumentation 2019-12-18 16:17:22 +01:00
fd6c32c2f2 testbench: Add two real tests for omsnmp
Adds a python based SNMP Trap Receiver which outputs received
SNMP Traps by UDP onyl at the moment. Can be extended later.
Tests results are verified by content_check.

The tests have to be enabed with ./configure --enable-snmp-tests

Test sndrcv_omsnmpv1_udp_dynsource.sh checks the new snmpv1dynsource
option.

In order to work, the following python packages needs to be installed:
	pip install pysnmp

On Ubuntu 18, we need these packages to be installed:
	apt install snmp libsnmp-dev snmp-mibs-downloader

closes: https://github.com/rsyslog/rsyslog/issues/3985
2019-12-03 13:26:36 +01:00
2bd6c99cbe relp: Add support setting openssl configuration commands.
Add new configuration parameter tls.tlscfgcmd to omrelp and imrelp.
(Using relpSrvSetTlsConfigCmd and relpCltSetTlsConfigCmd)

OpenSSL Version 1.0.2 or higher is required for this feature.
A list of possible commands and their valid values can be found in the
documentation: https://www.openssl.org/docs/man1.0.2/man3/SSL_CONF_cmd.html

The setting can be single or multiline, each configuration command is
separated by linefeed (n). Command and value are separated by
equal sign (=). Here are a few samples:

tls.tlscfgcmd="Protocol=ALL,-SSLv2,-SSLv3,-TLSv1,-TLSv1.2"

tls.tlscfgcmd="Protocol=ALL,-SSLv2,-SSLv3,-TLSv1
MinProtocol=TLSv1.2"

Add to new testcases for librelp and tlscfgcmd.

closes https://github.com/rsyslog/rsyslog/issues/3959
2019-11-29 16:00:27 +01:00
Rainer Gerhards
0bb33d153e
testbench: fix some issue finding correct python interpreter
see also https://github.com/rsyslog/rsyslog/issues/3853
2019-11-29 14:06:41 +01:00
Rainer Gerhards
0307a0cc0a
testbench: improve mysql test framework plumbing
This is an initial try to make mysql tests able to run in parallel,
which also hardens them against some timing problems.
2019-11-26 10:56:33 +01:00
Rainer Gerhards
771fc80112
Merge pull request #3982 from dmolik/omhttp-loki
omhttp: add basic support for Loki Rest
2019-11-22 16:41:24 +01:00
Dan Molik
8793a88987
omhttp: add basic support for Loki Rest
Loki is a new message indexer and querier from Grafana Labs. See
https://github.com/grafana/loki for details on Loki.

This change provides the initial message structure to send bulk message
payloads to the Loki Rest endpoint. omhttp, recieved a new bulk message
format called lokirest. Additionally, the plugin relies on the user to
provide the correct "stream" read message format.

A loki template must be json compatible and include a "stream" key of
key value tags, and a values key of an array of 2 element arrays, where
each 2 element array is the unix epoch in nanoseconds followed by an
unstrectured message.

An example:

    template(name="array_loki" type="string" string="{\"stream\":{\"host\":\"%HOSTNAME%\",\"facility\":\"%syslogfacility-text%\",\"priority\":\"%syslogpriority-text%\",\"syslogtag\":\"%syslogtag%\"},\"values\": [[ \"%timegenerated:::date-unixtimestamp%000000000\", \"%msg%\" ]]}")
2019-11-22 09:03:10 -05:00
Rainer Gerhards
ff4b3558cf
testbench: obtain python binary path via AM_PATH_PYTHON
see also https://github.com/rsyslog/rsyslog/issues/3853
2019-11-22 14:45:10 +01:00
Rainer Gerhards
cf6b9c5931 testbench: add some debugging instrumentation
We have a long-standing issue with the mysql tests. We can't find the
root cause with special test runs. This commit now adds some debug
info. The idea is to merge this and run it in all upcoming tests, so
that we can hopefully get a sufficiently large sample that we can
address the problem.
2019-11-21 17:05:14 +01:00
Rainer Gerhards
3b35c98578
testbench: fix issuesn regard to ElasticSearch, modernize tests
Fix some minor issue that were detected as part of other work.
2019-10-30 10:04:22 +01:00
Rainer Gerhards
e88b956e9d
core/queue: provide ability to run diskqueue on multiple threads
see also https://github.com/rsyslog/rsyslog/issues/3543
closes https://github.com/rsyslog/rsyslog/issues/3833
2019-10-30 10:04:09 +01:00
Rainer Gerhards
a1c92c1c9c
Merge pull request #3900 from rgerhards/diskq-multi-file-del
core queue bugfix: handle multi-queue-file delete correctly
2019-10-16 17:15:42 +02:00
Rainer Gerhards
103a44263d
core queue bugfix: handle multi-queue-file delete correctly
Rsyslog may leave some dangling disk queue files under the following
conditions:

- batch sizes and/or messages are large
- queue files are comparatively small
- a batch spans more than two queue files (from n to n+m with m>1)

In this case, queue files n+1 to (n+m-1) are not deleted. This can
lead to problems when the queue is re-opened again. In extreme cases
this can also lead to stalled processing when the max disk space is
used up by such left-over queue files.

Using defaults this scenario is very unlikely, but it can happen,
especially when large messages are being processed.
2019-10-15 10:51:05 +02:00
Rainer Gerhards
73dc50ad0c
testbench: fix unreliable gzipwrite test
The test was timing-sensitive as we did not properly check all data
was output to the output file - we just relied on sleep periods.

This has been changed. Also, we made some changes to the testing
framework to fully support sequence checking of multiple ZIP files.
2019-10-15 10:28:03 +02:00
Rainer Gerhards
4a4a09c5a0
testbench: improve dynstats tests and make them more portable 2019-10-09 12:21:14 +02:00
Rainer Gerhards
4638699bce
testbench: modernize and stabilize some tests 2019-08-17 18:24:54 +02:00
Rainer Gerhards
42a8051ad9
testbench: make most tests use a port file and assign listen port 0
This makes the test much more robust against heavily loaded test
systems.
2019-08-16 17:31:52 +02:00
Rainer Gerhards
e7f3649da1
Merge pull request #3822 from rgerhards/i3817
imtcp bugfix: multiple listnerPortFile parameter did not work
2019-08-15 18:27:26 +02:00
Rainer Gerhards
185b388fb0
testbench: add test for imtcp multiple listener port files
This also enhances the testbench plumbing a bit and updates other tests
to support that.
2019-08-15 10:22:48 +02:00
Rainer Gerhards
f958e1556b
testbench: modernize tests and make them more reliable 2019-08-15 09:59:52 +02:00
Rainer Gerhards
4816262912
Merge pull request #3821 from rgerhards/tb-gzipwr_large
testbench: make gzipwr_large test more reliable and improve plumbing
2019-08-14 14:44:42 +02:00
Rainer Gerhards
bc0efab9eb
testbench: improve testbench plumbing for gzip and fail cases
We have added new capabilities to the testbench plumbing to automatically
deal with gzip-compressed files. This also permits to use the wait_seq_check
function to work for gzip tests as well. The known-timing-sensitive
gzipwr_large test now makes use of the new capabilities. This enables us
to more reliably detect when we can savely shutdown the tested instance.

This commit also adds an ability to "abort" the full testbench run on
first test failure. This is especially useful during CI.
2019-08-14 12:57:49 +02:00
Rainer Gerhards
afdce50957
Merge pull request #3814 from rgerhards/tb-nitfixes
testbench: remove debug output; lessen kafaka tests output
2019-08-14 10:57:59 +02:00
Rainer Gerhards
e8b9476d0c
testbench: update sndrcv test to newest standard
this makes the test more reliable in regard to port use. This is also
a model-commit which can be used to update other tests and base new
tests on it.
2019-08-13 09:46:20 +02:00
Rainer Gerhards
63d4726f7c
testbench: remove debug output; lessen kafaka tests output
the kafka tests generate a wealth of usually-unneeded output which
clutters the CI logs and makes them almost unusable.
2019-08-13 08:27:56 +02:00
Rainer Gerhards
26047bff3e
testbench: add multithreading tests with omfile async io/zip writing
see also https://github.com/rsyslog/rsyslog/issues/3772
2019-08-02 10:19:59 +02:00
Rainer Gerhards
799f3a111e
testbench: add some more tests for HUP handling 2019-07-31 10:41:05 +02:00
Rainer Gerhards
7cb4cdfab7
imtcp: enable listenPortFileName parameter
this parameter was added, but it had no effect as it was not
passed down to the driver layer. This has been fixed. That also
now enables us to use dynamically-assigned port, which are
very useful for further testbench stabilization. Quite some
false positives occurred because the pre-selected port was
already in use again when rsyslog started.
2019-07-18 10:05:00 +02:00
Rainer Gerhards
b4fea96907
Merge pull request #3672 from rgerhards/i3667-test
testbench: test for leading space issue in RFC 5424 parser
2019-07-01 09:13:25 +02:00
Rainer Gerhards
3836859af0
testbench framework: use ip tool instead of outdated ifconfig
The framework now first checks if "ip" is available and falls back
to "ifconfig" only if this is not the case.

Thanks to Michael Biebl for the suggestion.

closes https://github.com/rsyslog/rsyslog/issues/3682
2019-06-24 15:43:37 +02:00
Rainer Gerhards
d433ee83e8
testbench: add omfile tests based on real-world issue
see also https://github.com/rsyslog/rsyslog/issues/3686
2019-06-24 09:57:54 +02:00
Rainer Gerhards
12a7a4809e
testbench: test for leading space issue in RFC 5424 parser
see also https://github.com/rsyslog/rsyslog/pull/3671
see also https://github.com/rsyslog/rsyslog/pull/3667
2019-06-18 14:43:19 +02:00
Rainer Gerhards
6c946af43c
testbench: re-enable some test on FreeBSD
They were actually failing because of a platform bug (HUP not
propperly being processed on FreeBSD). This has been fixed in
e698e32d999e39 and so we now re-enable these tests.
2019-06-14 17:07:32 +02:00
Rainer Gerhards
b8ccf168cd
testbench bugfix: potentially invalid grep call
This can lead to false positives at least on some platforms.
2019-06-13 14:11:22 +02:00
de6151b292 testsuite: Fixed wrong parameter check in diag.sh (tcpflood())
When first parameter is check_only, the tcpflood funtion shall not
abort the test itself (The fail is intended if this option is set).

closes issue #3625
2019-05-10 09:28:41 +02:00
Rainer Gerhards
036b192be8
testbench cleanup: remove no longer needed code
These files are no longer created due to the move to support
parallel test runs.
2019-05-06 14:01:00 +02:00
Rainer Gerhards
37187427bf
dnscache: fix looking issue detected by Coverity Scan
This leads to a refactoring of the looking code; issue was caused
by new TTL cache expiration code which placed not semantics on the
cache. These were not properly handled under all circumstances.
2019-04-16 15:41:32 +02:00
Rainer Gerhards
850a8d8dae
Merge pull request #3600 from Whissi/issue_3599
tests: omhttp: add Python 3 compatibility
2019-04-15 09:48:29 +02:00
Rainer Gerhards
4863db5062
Merge pull request #3393 from rgerhards/i2924
core/action: implement capability to resume/suspend via external file
2019-04-15 09:42:09 +02:00
598eec659e tcpflood: Added support setting OpenSSL command parameters.
Also added a test based on imtcp-tls-ossl-basic.sh which tests
the new -k parameter with tcpflood called:
imtcp-tls-ossl-basic-tlscommands.sh

Fixed OpenSSL error reporting in tcpflood which was running into
a loop when OpenSSL error stack was printed out.
2019-04-12 12:45:20 +02:00