46 Commits

Author SHA1 Message Date
Rainer Gerhards
697f7d8378
CI: many kafkatests were not executed b/c kafkacat was renamed
kafkacat has been renamed to kcat, as it looks for trademark issues.
Kafka module tests depend on that utility and are skipped if command
is not available. This was now always the case for newer development
containers.

Name was now adjusted in all tests.
2025-09-04 17:18:42 +02:00
Nelson Yen
6102f6083c initial implemenation http input module
- uses http library to provide http input.
user would need to configure an 'endpoint' as input, along
with a ruleset, defining how the input should be routed in
rsyslog.

bugfix, free dynamic buf if created, don't echo to client data.

do data framing using newline character.
more input options support:
  - input name
  - flowcontrol

add support for gzip content support, and other options

- gzip content
- parse linefeeds by default, option to ignore linefeeds

add support for gzip content support, and other options

- gzip content
- parse linefeeds by default, option to ignore linefeeds

WIP - misc edits enable port, documentroot module parameter

fix overloaded stack issue

update tests to use available port instead of hard-coded one.

Add 'octet counted framing' support

- option is "SupportOctetCountedFraming", currently 'off' by
default.

update imhttp-getrequest-file.sh test to use $srcdir

imhttp - support multi-threaded connection contexts.

- tests for large data posts

- Add header data as metadata option

move mg_start into activatecnf instead of in runinput
2020-08-23 07:24:48 -07:00
Rainer Gerhards
1b2769f5ec
travis: remove no longer needed Trusty install scripts
This also removes all remaining references to LLVM repos, as such
closes https://github.com/rsyslog/rsyslog/issues/2380
2019-11-14 16:59:29 +01:00
Rainer Gerhards
cfebaed413
travis: libmongoc-dev does not install all needed
there seem to be packaging changes in ubuntu or some other repos
we use on travis. This should fix the issue.
2019-08-09 12:33:20 +02:00
Nelson Yen
82142c7c73 imdocker input plugin - enables pulling container logs from docker
- plugin will get (docker) container logs from a host as well as filling out some
    basic container metadata as id, name, image, labels.
  - requirements: curl-7.40.0+, for unix_domain_socket option (Docker API listens on a
    local socket). Additionally, plugin will handle the issue of log lines larger than
    16KB begin split by Docker.
  - include imdocker-unittests in testbench
  - enable imdocker tests in osx travis tests
  - use curl master branch for imdocker-tests
  - fix cflags for pthread
  - enforce minimum curl version during configuration - to 7.40.0 for imdocker
  - container polls after the initial one are automatically filtered utilizing the 'since' option as per docker api
  - imdocker tests refactored.
  - add escapeLF config option

Test bench test added:
  - imdocker-basic: checks for completeness.
  - imdocker-basic-vg: same as basic with valgrind enabled.
  - imdocker-long-logline: checks imdocker can handle 16K+ log lines.
  - imdocker-long-logline-vg: same as long-logline test with valgrind enabled.
  - add option 'retrieveNewLogsFromStart' which will cause imdocker to ignore the 'tail'
  option when retrieving container logs for newly activated containers. Containers that
  were already active when imdocker first starts still uses the tail option.
  - imdocker-new-logs-from-start: checks that new containers will ignore the tail option
  - imdocker-new-logs-from-start-vg: valgrind enabled
  - multi-line support via docker label: imdocker.startregex
  added multi-line tests for testbench

Travis test bench related history and notes:
 - Travis tests, enable imdocker
 - imdocker travis tests not enabled in precise or trusty ubuntu versions of travis-test.
   Assuming travis tests will run in ubuntu 16
 - tests: harden container name against strange file system pathes
   RSYSLOG_DYNNAME contains the file system path, which is NOT guarnateed
   to be suitable for things other than file names.
 - tests: bugfix and guard against false negative
 - content_check_with count was done before rsyslog was terminated, so
   it was uncertain how many messages were written at that spot
   (especially on slow machines)
 - so far, only the number of lines was checked, not their content. So
   any problems with the actual messages would not have been detected.
 - improved support for listcontainer options, fix racy imdocker tests
2019-04-02 08:29:05 +02:00
Rainer Gerhards
614a03550e
testbench: fix issues with journal testing
The configure/Makefile checks were not correct, leading to the
build of journal components when not necessary, even if not
supported by the platform. Thus lead to invald build and test
failures. This commit fixes that.
2019-01-18 09:09:37 +01:00
Rainer Gerhards
c4003294dc
travis: do no longer run 0mq tests
This often causes trouble when the packages are rebuild by the 0mq project
(which happens frequently). We already do intensive testing of the 0mq
components in the buildbot infrastructure, where we use dedicated containers.
This is reliable, as the containers already contain everything needed and so
do not need to reach out to the 0mq package archives. In the light of this,
let's save us the trouble of Travis failures. The only downside is that
users cannot pre-test with their local Travis when modifying 0mq modules,
which is quite acceptable.
2019-01-18 09:09:13 +01:00
Rainer Gerhards
c37ec8292e
travis: more error message to diagnose travis issue 2018-11-14 14:17:38 +01:00
Rainer Gerhards
e551642316 testbench: add missing shebangs
detected by shellcheck
2018-10-01 12:00:31 +02:00
9d0933bccf testbench: Added new kafka tests using kafkacat for better debugging.
Fixed some issues with sndrcv kafka tests.
Generating kafka topics dynamically now it kafka tests.

Limited messagecount in some tests to 50000 for now.
2018-09-05 11:55:33 +02:00
Rainer Gerhards
0bac6654c0 Travis: temporarily disable kafka tests
We already know they are currently unstable, so there is not point
in messing up CI result each time. The test will still be carried out
in nightly checks, so the problems won't be forgotten.
2018-07-29 14:38:11 +02:00
Rainer Gerhards
b06a48d6b5 Travis: further collapse build matrix 2018-01-16 13:04:47 +01:00
Rainer Gerhards
89318b2dea travis: combine some build checks
... and do them via containers. This also reduces resource usage and
runtime on Travis a bit.
2018-01-15 18:07:06 +01:00
Rainer Gerhards
b89466e105 Travis: run static analyzer via standard dev container
also some improvements to somewhat optimize the build matrix so
that we avoid unnecessary work.
2018-01-14 13:56:27 +01:00
Rainer Gerhards
f356d9f399 Travis: check build on Alpine Linux and Ubunut rolling (via docker container) 2018-01-12 17:29:02 +01:00
Rainer Gerhards
0fcc9dc609 travis: reenable static analyzer
LLVM has changed packaging structure, this fix adapts to it. So we
now have scan-build-5.0 again.
2017-12-30 12:46:45 +01:00
Rainer Gerhards
8d8abf2d9b travis: enable gcc-7 with high warning levels
see also https://github.com/rsyslog/rsyslog/pull/2121
2017-11-29 12:39:07 +01:00
Rainer Gerhards
42fd7c7b78 travis: temporarily disable new autoconf-archive so that we can merge soon
the very new version introduced new warning compiler flags, which require
some work to fix build issues. We do not want to hold the rest of the
useful work in this commit set just for that. So we temporarily disable
the new package and will work in parallel to make things build cleanly
again.
2017-11-25 18:46:00 +01:00
Rainer Gerhards
0953b7c942 Travis: temporarily disable packages that make build fail 2017-11-25 18:46:00 +01:00
Rainer Gerhards
aeeecea3f2 Travis: prepare for use of newer autoconf-archive
thanks to whissi for the package!

see also https://github.com/rsyslog/rsyslog/issues/2102
2017-11-25 18:45:40 +01:00
Rainer Gerhards
4761cbdd68 travis: add gcc-7 based compile tests 2017-11-25 15:49:05 +01:00
Hugo SOSZYNSKI
1ab4bfb0e7 Corrected some compilation warning and errors.
No longer installing libmongo-client in Travis script.
2017-11-17 15:37:26 +01:00
Hugo SOSZYNSKI
f1ec0993c4 Merge from upstream/master 2017-11-17 09:45:22 +01:00
Rainer Gerhards
0d124d164a Travis: let build requirements not interfere with line length check 2017-11-16 13:47:47 +01:00
Rainer Gerhards
6c05f41269 Travis CI: add support for cron jobs and Coverity scan
We execute different code pathes under cron jobs. It doesn't make sense
to redo what we already did before a merge. Right now, cron builds a
Coverity scan submission and uploads it. This should be set to run
once a day.

closes https://github.com/rsyslog/rsyslog/issues/2037
2017-11-16 11:30:01 +01:00
Hugo SOSZYNSKI
263796f2f2 Merge remote-tracking branch 'upstream/master' 2017-11-10 14:02:10 +01:00
Hugo SOSZYNSKI
7d3a2e61ab Now installing mongo-c-driver from the git on Travis. Corrected misunterstood structure initialisation by clang. Added the removed libmongo-client needed for the usertools. 2017-11-10 13:59:43 +01:00
Rainer Gerhards
d178cbf299 travis: fix invalid LLVM repository source 2017-11-07 12:06:50 +01:00
Rainer Gerhards
6f0b2f7524 travies: use clang-5.0 for static analysis 2017-11-04 17:13:26 +01:00
Hugo SOSZYNSKI
e10949041e Changed the libmogo-client-dev for libmongoc-driver-dev in the install.sh for Travis. 2017-11-03 09:04:07 +01:00
Rainer Gerhards
2fd3fcca1e travis: remove temporary package override
this was needed while libfastjson 0.99.7 was not released
We just uncomment it so that the override can be easily re-enabled
the next time a similar need arises (what routinely happens)

closes https://github.com/rsyslog/rsyslog/issues/1848
2017-10-18 08:20:22 +02:00
Rainer Gerhards
3c84972299 travis: add capability to temporarily use v8-devel package
required due to new-version requirement for next release.
Else we could not run the CI tests.
2017-10-13 18:51:37 +02:00
mrworkman
bb60c206f0 Fix broken 'debugless' build. 2017-09-12 08:01:39 -04:00
Rainer Gerhards
657c6ea441 remove very verbose CI output in librdkafka install
was a left-over from debugging the kafka CI system;
spams stdout and thus can even lead to premature abort on
Travis due to too-large output
2017-06-28 12:23:01 +02:00
732d0e21a7 omkafka: Enhanced support to detect kafka broker problems.
Also fixed tryresume and doAction handling when action is
in suspend state.

Calling for Callbacks more often.

Adjusted kafka fail test settings.

diag.sh script detects of kafka server comes up proberly

Added liblz4 to dependencies for omkafka/imkafka needed
for static linking.

This addresses the omkafka queue problem from issue:
https://github.com/rsyslog/rsyslog/issues/1052
Yet it does not fully solve the message problem yet,
but I am working on this.
2017-06-08 15:35:45 +02:00
Rainer Gerhards
4c136cc5ce travis/testbench: silence dependency build a bit 2017-05-22 14:06:25 +02:00
Rainer Gerhards
2cce1a36eb travis: enable imkafka 2017-05-19 14:07:56 +02:00
Rainer Gerhards
fe9e51efac travis: make qpid-proton work again
Package libqpid-proton3-dev is no longer available for precise.
As such, we cannot install it and so we disable both the installation
and related tests when running on precise.
2017-03-20 10:31:04 +01:00
Dan Molik
fde42481b6 enusre we build libmaxmind from source for Ubuntu percise (12.04) and trusty (14.04) 2017-01-13 14:42:08 -05:00
Dan Molik
591eb7dbc2 Compile ip-lookup module during testing
- fix clang compilation by switching to BEGINdoAction_nostring
2017-01-13 13:41:52 -05:00
Rainer Gerhards
388d60535a travis: liblognorm's new version has different package name
old name was incorrect, but we need to modify this here as well
2016-08-22 14:39:47 +02:00
Rainer Gerhards
50b91a4091 travis: alternative way to install autoconf-archive
if we install via .travis.yml, we always get aborts during
the install, so we do it differently
2016-06-03 09:50:14 +02:00
Rainer Gerhards
5b0e8b80d2 Merge branch 'master' into travis-es 2016-05-10 17:33:24 +02:00
Rainer Gerhards
3e68926db7 travis: force install of non-OS-standard components
sometimes the travis build complains about inability to verify
correctness, which is probably related to ongoing build work. This
should solve the issue with PPAs (mostly our own).
2016-05-06 11:46:40 +02:00
Janmejay Singh
510569dcbc enable kafka from travis-config so install script pulls librdkafka in 2016-04-19 15:43:30 +05:30
Rainer Gerhards
e63099c597 travis: merge software install into script step
this provides the ability to use common logic for some
steps (will show benefits in the longer term)
2016-03-14 11:24:24 +01:00