fluent-bit pod having difficulty connecting to Splunk HEC endpoint

[lifayt]

Bug Report

Describe the bug
We are attempting to add a splunk output to our fluent-bit pods that run as part of an EKS Amazon Cloudwatch addon. We are running into an issue where we are able to manually connect to the HEC endpoint via a curl command like so:

curl --request POST \
  --url https://example.splunkcloud.com/services/collector \
  --header 'Authorization: Splunk <hec-token>' \
  --header 'Content-Type: application/json' \
  --data '{"index": "airflow", "event": "from-fluent-bit-pod"}'

This produces the expected response:

{"text":"Success","code":0}

Similarly, querying the HEC health endpoint works:

curl --request GET \
  --url https://example.splunkcloud.com/services/collector/health 

This also produces the expected response:

{"text":"HEC is healthy","code":17}

but if we try it using the fluent-bit cli, or using a config file, then we get an error saying the domain is not found.

[net] getaddrinfo(host='https://example.splunkcloud.com/services/collector', err=4): Domain name not found

Here are some example commands of how I'm starting up fluent-bit:

/fluent-bit/bin/fluent-bit -i cpu -t cpu -o splunk -p host=https://example.splunkcloud.com/services/collector -p splunk_token=<token> \
  -p tls=on -p tls.verify=off -m '*'

Expected behavior
Since I can connect to the splunk ingestion endpoint using curl, I would expect fluent-bit to also be able to connect.

Your Environment

• Version used:

Fluent Bit v1.9.10
Git commit: f4996b8a8e6c82498e95906153738078039c74bd

• Environment name and version (e.g. Kubernetes? What version?): EKS Server Version: v1.28.12-eks-2f46c53, installed as part of the aws cloudwatch eks addon.

It would be particularly helpful if I could get some feedback on how to better diagnose what the issue is here. I work in a corporate environment, so there's always lots of networking/firewall issues to contend with, but I'm not sure how to get at the guts of what fluent-bit is running into (since my attempts at debugging it by posting events manually to splunk are all working).

[patrick-stephens]

1.9 is a very old version, can you retry with the latest version as there have been a lot of improvements and changes since?