18 Commits

Author SHA1 Message Date
Rainer Gerhards
69ef6e329b fix bad bash coding style and disable shellcheck false positives
Also now permit interactivly running tests without explicitly setting
$srcdir. This now works if we are inside ./tests and fails, as before,
when we are in a different directory.

Detected by shellcheck via CodeFactor.io
2018-10-23 13:27:37 +02:00
Rainer Gerhards
9378945b9d
fix useless use of cat
more or less cosmetic here, but anyhow...

detected by shellcheck
2018-10-10 15:20:15 +02:00
b08a8e2b51 testbench: Kafka plumbing
- Removed all sleeps where possible.
- Moved all kafka start/stop/download logic into functions.
- Moved kafka/zookeeper stop into error_exit and exit_test.
- Kafka/Zookeeper cleanup only done on success now.
- Kafka/Zookeeper logfiles automatically dumped on error_exit only now.
- Added cleanup for Kafka/Zookeeper instances into CI/buildbot_cleanup.sh
2018-09-19 09:14:42 +02:00
Rainer Gerhards
74207603ad
testbench: remove imtcp in kafka test where not strictly needed
This method currently has some race associated with it, also it is
really not required and the injectmsg method is superior.

Also fix imdiag's new $imdiagInjectDelayMode config directive. While
it was added, it was forgotten to actually apply the value.
We are not doing a separate commit as this is thightly coupled into
our testing here (which also as a side-effect tests the new imdiag
functionality).
2018-09-13 14:25:17 +02:00
847d4223ef testbench: Incremented input timeout to 60000 because kafka tests may fail on load otherwise.
Also fixed minor configuration and syntax issues.
Moved test init in kafka tests as far as possible to the top.
2018-09-07 11:52:57 +02:00
9d0933bccf testbench: Added new kafka tests using kafkacat for better debugging.
Fixed some issues with sndrcv kafka tests.
Generating kafka topics dynamically now it kafka tests.

Limited messagecount in some tests to 50000 for now.
2018-09-05 11:55:33 +02:00
Rainer Gerhards
566788ecac * remove unnecessary calls to "wait_startup"
This is automatically called by "startup" and "startup_vg". Tests
  looked like a left-over from old testbench plumbing.
2018-09-04 10:50:57 +02:00
Rainer Gerhards
20696a754b
testbench: streamline plumbing
no functional changes, but modernization and cleanup
2018-09-01 13:18:50 +02:00
95ed2c573b testbench: Switched to newer kafka/zookeeper versions.
Fixed test configurations (socket timeouts with newer kafka).
Reactivated some kafka tests that work now.

omkafka: Fixed "closeTimeout" setting in action shutdown
2018-08-21 15:00:44 +02:00
Rainer Gerhards
1ed6859dfd
testbench: replace hardcoded ports and improve cleanup handling
Support tools (like tcpflood) are also upgraded to support the
necessary dynamic port.s

This is part of the effort to make parallel testing possible.

We move parts of the cleanup to the buildbot cleanup, as we cannot
clean out instances on each test when we run parallel tests.
2018-08-18 20:28:31 +02:00
Rainer Gerhards
dbe61eb06b
testbench: make some test files dynamic
This is work torward the ability to run the testbench in parallel.
Some small issues in tests have also been detected, which were not
previously seen. They have been fixed. We did not do separate
commits for this as it would clutter the commit log with not
really relevant info.

Also move some cleanup to "make clean" target

To support parallel testbench runs, we need to have dynamic work files.
So deleting work files on each test does not work, especially as we can
no longer assume they are "left-overs" from a failed test - they may
actually (and quite likely) belong to tests that run in parallel.

So the proper solution is to do via "make clean".

closes https://github.com/rsyslog/rsyslog/pull/2916
2018-08-13 15:04:47 +02:00
Florian Riedl
2d22742f32 Test refactoring part 8 2018-08-13 12:58:30 +02:00
Rainer Gerhards
d8b6dc52af
testbench: more modernization of testbench plumbing 2018-08-02 14:47:19 +02:00
Rainer Gerhards
2ba3c8ddde
testbench: modernize testbench plumbing
changes some of the test commands to use bash functions
includes some small bug fixes to tests where bugs were
previously not seen due to different plumbing.
2018-07-23 17:26:34 +02:00
8ecf0aeb34 kafka: Added option enable static linking of librdkafa (disabled by default).
Also saving failed msgs now when kafka produce fails. No message loss should
be possible anymore.

Also removed old kafka tests and readded new kafka tests.
If kafka instances cannot be started (happens from time to time when
kafka/zookepper sync fails), tests will be skipped instead of failing.

Closes https://github.com/rsyslog/rsyslog/issues/1559
Closes https://github.com/rsyslog/rsyslog/issues/1584
Closes https://github.com/rsyslog/rsyslog/issues/1515
May fix https://github.com/rsyslog/rsyslog/issues/1230
Closes https://github.com/rsyslog/rsyslog/issues/1052
2017-06-19 17:14:35 +02:00
b71c1871d0 omkafka: Added new options to save and load failed messages.
When enabled, failed messages are kept in memory queue and saved on
shutdown and loaded on startup. This prevents message loss of failed
messages.

The failed items are saved in a simple Linked List and automatically
being reprocessed on resume or doaction.

Load/Save logik implemented.

This should also remove the last possible message loss from issue:
https://github.com/rsyslog/rsyslog/issues/1052
2017-06-08 15:35:45 +02:00
732d0e21a7 omkafka: Enhanced support to detect kafka broker problems.
Also fixed tryresume and doAction handling when action is
in suspend state.

Calling for Callbacks more often.

Adjusted kafka fail test settings.

diag.sh script detects of kafka server comes up proberly

Added liblz4 to dependencies for omkafka/imkafka needed
for static linking.

This addresses the omkafka queue problem from issue:
https://github.com/rsyslog/rsyslog/issues/1052
Yet it does not fully solve the message problem yet,
but I am working on this.
2017-06-08 15:35:45 +02:00
fa775e7590 omkafka: Added detection for broker failure and suspending/resume the action
imkafka, omkafka: switched from dynamic to static linking

Also added another test to test a typical kafka cluster failure environment.
2017-06-08 15:35:45 +02:00