dockerfile: install libzstd-dev for kafka zstd compression

[dbottini2]

Container images were built without libzstd-dev in the builder stage, so
librdkafka's find_package(ZSTD) failed and WITH_ZSTD was disabled. Kafka
producers in the shipped images could not use zstd compression.

Changes:

• dockerfiles/Dockerfile: add libzstd-dev to the builder and debug builder
  stages so librdkafka is compiled with zstd support.
• tests/runtime/out_kafka.c: add a runtime test asserting librdkafka accepts
  compression.codec=zstd (only true when WITH_ZSTD is compiled in).

Addresses #11366 (Packaging: Enable ZSTD compression by default).

---

Enter [N/A] in the box, if an item is not applicable to your change.

Testing
Before we can approve your change; please submit the following in a comment:

• Example configuration file for the change
• Debug log output from testing the change

• Attached Valgrind output that shows no leaks or memory corruption was found

If this is a change to packaging of containers or native binaries then please confirm it works for all targets.

• [N/A] local-build-all.sh builds the native distro packages under packaging/distros/. This PR only changes the container images (dockerfiles/Dockerfile), which that script does not build. The container image build was verified instead (see test evidence above).
• Set ok-package-test label to test for all targets (requires maintainer to do).

Test evidence

Example config (rdkafka.compression.codec=zstd):

[INPUT]
    Name    dummy
    Tag     test

[OUTPUT]
    Name                       kafka
    Match                      test
    Brokers                    127.0.0.1:9092
    Topics                     test
    rdkafka.compression.codec  zstd

Container build (dockerfiles/Dockerfile, --target=production) — zstd is now
detected so librdkafka is built with WITH_ZSTD:

-- Found ZSTD: /usr/lib/aarch64-linux-gnu/libzstd.so
WITH_ZSTD:BOOL=ON

Running the production image with the kafka zstd config — the output starts and
accepts the codec (only failure is the absent broker, as expected). Before this change
the same image rejected compression.codec=zstd at init:

[ info] [output:kafka:kafka.0] brokers='127.0.0.1:9092' topics='test'
[error] [output:kafka:kafka.0] ... Connect to ipv4#127.0.0.1:9092 failed: Connection refused

Runtime test (tests/runtime/out_kafka.c::zstd_compression_available) under Valgrind:

Test zstd_compression_available...              [ OK ]
SUCCESS: All unit tests have passed.
==13903== All heap blocks were freed -- no leaks are possible
==13903== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)

Notes on targets: amd64 and arm64 (the published distroless images) configure with
zstd and build successfully. The README's plain multi-arch example additionally lists
linux/arm/v7 and linux/s390x, which fail at CMakeLists.txt:310 because
-DFLB_SIMD=On is unsupported on those architectures; this is pre-existing and
unrelated to this change.

Documentation

• [N/A] Documentation required for this feature

Backporting

• Backport to latest stable release: dockerfile: install libzstd-dev for kafka zstd compression (4.2 backport)  #12004 (4.2 branch).

---

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

Summary by CodeRabbit

• Chores

  • Added the Zstandard development library to builder and debug images, making its development components available during builds.

• Tests

  • Added a Kafka runtime check that verifies whether Zstandard compression can be configured when Zstandard support is enabled.
  • Updated test configuration to include the required Kafka headers when the check is available.

[coderabbitai]

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

• Last reviewed commit: 39dc8a2

Use the following commands to manage reviews:

• @coderabbitai resume to resume automatic reviews.
• @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

• ▶️ Resume reviews
• 🔍 Trigger review

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

• Configuration used: defaults
• Review profile: CHILL
• Plan: Advanced
• Run ID: 4461749c-8482-4dfc-b041-47d98370ebb2

📥 Commits

Reviewing files that changed from the base of the PR and between 872fb42 and 39dc8a2.

📒 Files selected for processing: 1

• tests/runtime/out_kafka.c

🚧 Files skipped from review as they are similar to previous changes: 1

• tests/runtime/out_kafka.c

Included review availability: This review used your included allowance. Your plan provides up to 8 included reviews per hour; 7 remain after this review.

📝 Walkthrough

Docker images now install libzstd-dev. The Kafka runtime test suite conditionally checks whether librdkafka accepts zstd as the compression codec.

Changes: Zstd support updates

Layer / File(s)	Summary
Docker zstd packages dockerfiles/Dockerfile	The builder and debug images install libzstd-dev.
Kafka zstd availability test tests/runtime/CMakeLists.txt, tests/runtime/out_kafka.c	When the target exists, KAFKA_FOUND is false, and WITH_ZSTD is true, the Kafka runtime test target receives the librdkafka source include directory and FLB_TEST_KAFKA_ZSTD definition. The conditional test checks whether setting compression.codec to zstd succeeds and registers the test. The raw-output test comment is updated to mention selecting a log field.

Priority: ⬇️ Low

Estimated code review effort: 2 (Simple) | ~10 minutes

Change: Bug fix

Merge Risk: ⚪ Minimal · up to 39dc8

The production image includes the library needed for Kafka zstd support. No merge-blocking issue is identified.

Pre-merge checks: 5

Check name	Status	Explanation
Description Check		Check skipped - CodeRabbit’s high-level summary is enabled.
Title check		The title clearly identifies the primary change: adding libzstd-dev to enable Kafka Zstandard compression in the Docker image. It is concise and specific, although it does not mention the supporting r…
Docstring Coverage		Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 2 functions across 1 files.
Linked Issues check		Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check		Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create a new PR

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 99acbc8514

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[chatgpt-codex-connector]

Gate the zstd check to zstd-enabled builds

In regular source builds where FLB_OUT_KAFKA=On but the host lacks a system libzstd-dev, librdkafka still builds with WITH_ZSTD=OFF because the CMake path only lets librdkafka's own find_package(ZSTD) decide; the Dockerfile install does not affect those builds. This unconditional assertion then makes flb-rt-out_kafka fail for an otherwise supported non-container build, so either wire zstd into the CMake Kafka build or only run this check when zstd support was actually required/enabled.

Useful? React with 👍 / 👎.

[dbottini2]

Fixed in 0c6dde112. Zstd assertion now runs only for bundled librdkafka with WITH_ZSTD=ON; optional/system builds retain raw_format. Kafka runtime tests pass with zstd ON and OFF.
Rovo Dev