4 Commits

Author SHA1 Message Date
8ecf0aeb34 kafka: Added option enable static linking of librdkafa (disabled by default).
Also saving failed msgs now when kafka produce fails. No message loss should
be possible anymore.

Also removed old kafka tests and readded new kafka tests.
If kafka instances cannot be started (happens from time to time when
kafka/zookepper sync fails), tests will be skipped instead of failing.

Closes https://github.com/rsyslog/rsyslog/issues/1559
Closes https://github.com/rsyslog/rsyslog/issues/1584
Closes https://github.com/rsyslog/rsyslog/issues/1515
May fix https://github.com/rsyslog/rsyslog/issues/1230
Closes https://github.com/rsyslog/rsyslog/issues/1052
2017-06-19 17:14:35 +02:00
b71c1871d0 omkafka: Added new options to save and load failed messages.
When enabled, failed messages are kept in memory queue and saved on
shutdown and loaded on startup. This prevents message loss of failed
messages.

The failed items are saved in a simple Linked List and automatically
being reprocessed on resume or doaction.

Load/Save logik implemented.

This should also remove the last possible message loss from issue:
https://github.com/rsyslog/rsyslog/issues/1052
2017-06-08 15:35:45 +02:00
732d0e21a7 omkafka: Enhanced support to detect kafka broker problems.
Also fixed tryresume and doAction handling when action is
in suspend state.

Calling for Callbacks more often.

Adjusted kafka fail test settings.

diag.sh script detects of kafka server comes up proberly

Added liblz4 to dependencies for omkafka/imkafka needed
for static linking.

This addresses the omkafka queue problem from issue:
https://github.com/rsyslog/rsyslog/issues/1052
Yet it does not fully solve the message problem yet,
but I am working on this.
2017-06-08 15:35:45 +02:00
fa775e7590 omkafka: Added detection for broker failure and suspending/resume the action
imkafka, omkafka: switched from dynamic to static linking

Also added another test to test a typical kafka cluster failure environment.
2017-06-08 15:35:45 +02:00