Topic creation was racy: it can happen that the brokers have not
been synced. This causes follow-on errors and ultimately causes
the test to hang in rsyslog shutdown.
This PR introduces a method to ensure the brokers are acutally
up and running before the rest of the test is carried out. I
assume this fixes the problems we have seen. However, only
practice will show if this really is the case because the
problem happened relatively infrequently (so a couple of good
tests do not necessarily indicate "problem solved").
As such, we close the issue with this commit, but may need
to re-open it if the issue is seen again.
closes https://github.com/rsyslog/rsyslog/issues/3045
These suppressions are always used, as they can never be caused by
actual program code. So it doesn't hurt to always use them.
Also add some support for CodeCov tool.
Python3 writes to stderr immediately, and this caused the
captured output to differ with respect to Python2. Simplified
the test to do a single write to stderr. Also a cast to int
was needed when calculating 'numRepeats'.
closes#3030
- Removed all sleeps where possible.
- Moved all kafka start/stop/download logic into functions.
- Moved kafka/zookeeper stop into error_exit and exit_test.
- Kafka/Zookeeper cleanup only done on success now.
- Kafka/Zookeeper logfiles automatically dumped on error_exit only now.
- Added cleanup for Kafka/Zookeeper instances into CI/buildbot_cleanup.sh
The plugin was not handling 404 Not Found correctly when looking
up pods and namespaces. In this case, we assume the pod/namespace
was deleted, annotate the record with whatever metadata we have,
and cache the fact that the pod/namespace is missing so we don't
attempt to look it up again.
In addition, the plugin was not handling error 429 Busy correctly.
In this case, it should also annotate the record with whatever
metadata it has, and _not_ cache anything. By default the plugin
will retry every 5 seconds to connect to Kubernetes. This
behavior is controlled by the new config param `busyretryinterval`.
This commit also adds impstats counters so that admins can
view the state of the plugin to see if the lookups are working
or are returning errors. The stats are reported per-instance
or per-action to facilitate using multiple different actions
for different Kubernetes servers.
This commit also adds support for client cert auth to
Kubernetes via the two new config params `tls.mycert` and
`tls.myprivkey`.
omrelp for some time limited authentication modes to those
that were known. While this was OK, it prevented the easy
introduction of new auth modes into librel.
This has now been changed; omrelp now checks the validity of
the authmode directly via librelp by doing some librelp calls
upon processing the configuration.
Also, some tests have been updated to check this feature and
also ensure that the new librelp mode "certvalid" works
(if it is available).
* imkafka: implement multithreading support for kafka consumers.
Each consumer runs in it's own consumer thread now. New tests have also been
added for this.
- major memory leak which occurred once per message processed
So this could lead to OOM. Caused by improper free of json
structure
- another two major leaks of similar magnitued could occur if
"fromhost-ip" and/or "fromhost" properties were set
- minor leaks upon termination. these were unproblematic as
static and only occured immediately before shutdown.
But they triggered memory debugger errors.
- fixed test which did not check for mem leaks albeit it should
- core invalid handling of the "fromhost" property, if set via
the MsgSetPropsViaJSON() call. This was primarily of concern
for pmnormalize and mmexternal, and only if these properties
were used by either the rulebase or the external program
response.
Actually, most of the leaks go back to rsyslog core, but that
core functionality was not used by other modules in the same
way. But if some other would have used it, the effects would
have been the same (so be aware if you wrote custom modules).
This method currently has some race associated with it, also it is
really not required and the injectmsg method is superior.
Also fix imdiag's new $imdiagInjectDelayMode config directive. While
it was added, it was forgotten to actually apply the value.
We are not doing a separate commit as this is thightly coupled into
our testing here (which also as a side-effect tests the new imdiag
functionality).
This adds support for endmsg.regex. It is similar to
startmsg.regex except that it matches the line that denotes
the end of the message, rather than the start of the next message.
This is primarily for container log file use cases such as this:
date stdout P start of message
date stdout P middle of message
date stdout F end of message
The `F` means this is the line which contains the final part of
the message. The fully assembled message should be
`start of message middle of message end of message`.
`startmsg.regex="^[^ ]+ stdout F "` will match.
Setting timestamp to 0 now lets kafka handle this.
Also added "RD_KAFKA_V_KEY(NULL,0) if no key is configured.
testbench: Changed kafka server configuration "log.retention.hours"
property to 5000 which avoids that the log cleaner is deleting our
records before they can be processed.
Fixed some issues with sndrcv kafka tests.
Generating kafka topics dynamically now it kafka tests.
Limited messagecount in some tests to 50000 for now.
this also requires changes to some tooling.
Also, when assigning a dynamic port in tcpserver, the same port
number is used for IPv4 and IPv6.
Also removing some left-over debug output.
closes https://github.com/rsyslog/rsyslog/issues/2987
