This article explains how to configure and use the Splunk HEC service in Apache APISIX.
The complexity of systems is increasing as technology iterates and enterprise architecture evolves. Logs can support and be compatible with different analysis engines to reduce the cost for users in the selection, operation and maintenance process. Log-based analysis and observation plays a very important role as the cornerstone to ensure system stability.
Apache APISIX is not only an API Gateway with exceptional performance, but also has supported most of the mainstream open source and commercial logging solutions through the communication with community users on data and logging operation and maintenance, including: HTTP Logger, TCP Logger, Kafka Logger, UDP Logger, RocketMQ Logger, SkyWalking Logger, Aliyun Cloud Logging(SLS), Google Cloud Logging, etc.
We now have a new addition to the Apache APISIX Logger support matrix: Splunk HEC Logging.
This article explains how to configure and use the Splunk HEC service in Apache APISIX.
About Splunk HTTP Event Collector
Splunk is a full-text search engine for machine data that can be used to collect, index, search, and analyze data from a variety of applications. According to DB Engines' search engine ranking, Splunk is currently in second place and is a widely used full-text search software. Splunk, like ElasticSearch, is a quasi-real-time data stream that provides uninterrupted search results.
Splunk HTTP Event Collector (HEC) is an HTTP event collector provided by Splunk that provides the ability to send data and application events to Splunk using the HTTP(S) protocol.
About splunk-hec-logging plugin
When the maximum processing capacity of a queue is reached or when the maximum time to refresh the buffer is reached, data in the queue will be committed to Splunk HEC.
How to use the splunk-hec-logging plugin
Splunk Configuration
Deploy Splunk Enterprise
Please refer to Splunk's installation guide for deployment. This article will demonstrate deployment via Docker.
Docker command parameters are as follows.
docker run -p 18088:8088 -p 18000:8000 \ # 8088 is the HEC port, 8000 is the management backend port
-e "SPLUNK_PASSWORD=your-password" \ # Admin Login Password
-e "SPLUNK_START_ARGS=--accept-license" \ # Accept the license terms (Splunk will provide an Enterprise Trial License by default)
-e "SPLUNK_HEC_TOKEN=your-hec-token" \ # Set the default HEC token, this will create a default HEC after configuration
-itd --rm --name splunk-example splunk/splunk:latest
The command parameters are explained in the Docker Splunk Documentation.
Configure Splunk HEC
The default HEC is already configured and created in Docker, so we won't go into the process of creating it here. For details on the manual creation process, please refer to the documentation: Set up and use HTTP Event Collector in Splunk Web.
Login to Splunk Enterprise and check HEC
Access the mapped port of Docker through the browser. Since you need to map the 8000 port of the management backend to the 18000 port of the host, you can access it from the browser by "loopback address plus port" on the host during operation. For example: http://127.0.0.1:18000, the default username for login is admin, and the password is the SPLUNK_PASSWORD value set in the environment variable in the above example.
As shown in the figure below, it means the login is successful.
Click on "Settings > Data Inputs" at the top right of the screen to check if the default HEC is set successfully.
We can already see the number of HECs in the Inputs column of the HTTP Event Collector, indicating successful setup.
At this point, you can click HTTP Event Collector to enter the HEC details list to view the Token information of HECs.
Token Values is the value of SPLUNK_HEC_TOKEN configured in the Docker environment variable above.
Apache APISIX Configuration
Enable the splunk-hec-logging plugin
Run the following command to enable the splunk-hec-logging plugin.
curl http://127.0.0.1:9080/apisix/admin/routes/1 -H 'X-API-KEY: edd1c9f034335f136f87ad84b625c8f1' -X PUT -d '
{
"plugins":{
"splunk-hec-logging":{
"endpoint":{
// HEC endpoint
"uri":"http://127.0.0.1:18088/services/collector",
// HEC Token
"token":"BD274822-96AA-4DA6-90EC-18940FB2414C"
},
// // Maximum time (in seconds) to refresh the batch queue buffer
"inactive_timeout":2,
// Maximum number of log entries per batch queue
"batch_max_size":10
}
},
"upstream":{
"type":"roundrobin",
"nodes":{
"127.0.0.1:1980":1
}
},
"uri":"/splunk.do"
}'
The plug-in parameters are described in the following table.
| Name | Required | Default Value | Description |
|---|---|---|---|
| endpoint | Yes | N/A | Splunk HEC Endpoint Configuration Information |
| endpoint.uri | Yes | N/A | Splunk HEC Event Collection API |
| endpoint.token | Yes | N/A | Splunk HEC Identity Token |
| endpoint.channel | No | N/A | Splunk HEC send channel identification, refer to: About HTTP Event Collector Indexer Acknowledgment |
| endpoint.timeout | No | 10 | Splunk HEC data submission timeout in seconds. |
| ssl_verify | No | TRUE | Enable SSL authentication, refer to: OpenResty Documentation. |
| max_retry_count | No | 0 | Maximum number of retries before removal from the processing pipeline. |
| retry_delay | No | 1 | Number of seconds that process execution should be delayed if execution fails. |
| buffer_duration | No | 60 | The maximum duration (in seconds) of the oldest entry in the batch must be processed first. |
| inactive_timeout | No | 5 | Maximum time to refresh the buffer in seconds. |
| batch_max_size | No | 1000 | Maximum number of entries per batch queue. |
Send the request
Run the following command to send a request to Splunk.
$ curl -i http://127.0.0.1:9080/splink.do
HTTP/1.1 200 OK
Content-Type: text/html; charset=utf-8
Transfer-Encoding: chunked
Connection: keep-alive
Date: Fri, 10 Dec 2021 09:57:52 GMT
Server: APISIX/2.11.0
Hello, Splunk HEC Logging
Verify the log
Log in to the Splunk console and click "Search & Reporting".
Type source="apache-apisix-splunk-hec-logging" in the search box to query the sent request logs.
Disable the splunk-hec-logging plugin
Remove the splunk-hec-logging configuration to disable the plugin.
curl http://127.0.0.1:9080/apisix/admin/routes/1 -H 'X-API-KEY: edd1c9f034335f136f87ad84b625c8f1' -X PUT -d '
{
"uri":"/logging.do",
"upstream":{
"type":"roundrobin",
"nodes":{
"127.0.0.1:1980":1
}
},
"plugins":{
}
}'
Summary
Apache APISIX is also currently working on additional plugins to support integration with more services, so if you're interested, feel free to start a discussion thread in our GitHub Discussion or communicate via the mailing list.