APISIX Integrates ClickHouse to Improve Log Efficiency

The author of this article, Zhendong Qi, is from China Mobile Cloud Competence Center. He has extensive experience in distributed object storage software development and has participated in the construction of several resource pools in the mobile cloud. This article describes how community contributor Zhendong Qi contributed clickhouse-logger to Apache APISIX and how he used the plugin to simplify the business architecture and improve the efficiency of logging.

Background Information​

Apache APISIX is a dynamic, real-time, high-performance API gateway that provides rich traffic management features such as load balancing, dynamic upstream, canary release, circuit breaking, authentication, observability, and more. It not only has many useful plugins, but also supports plugin dynamic change and hot reload.

Developed by Yandex and open sourced in 2016, ClickHouse is not only a database, but also a database management system that allows creating tables and databases, loading data and running queries at runtime without reconfiguring or restarting services.

As more and more companies start to migrate their business to the cloud, how to efficiently implement log sending and receiving and log analysis to enhance the observabilities of the system becomes a challenge. China Mobile Cloud Competence Center, as a company providing public cloud services, the architecture of the initial Apache APISIX-based business log sending and receiving and analysis system is roughly like this.

Click to Preview

As the business grows, the above solution is not only oddly expensive to maintain, but also difficult to meet our needs for fine-grained data analysis. Since the data received by rsyslog is a string rather than a JSON format log, it makes log analysis difficult.

There is a famous saying in computing that "any problem can be solved by adding an indirect middle layer". We actually considered adding another intermediate layer between tcp-logger and rsyslog to convert strings to JSON, but this is obviously not a permanent solution.

So let's look at the problem differently: if we consider the existing architecture of "tcp-logger+rsyslog+Promtail+Loki" as a huge middle layer, then no matter how many additional middle layers we add in between, it only solves the immediate problem. In the meantime, it makes it more bloated and difficult to maintain. Is there a product on the market that can directly replace "tcp-logger+rsyslog+Promtail+Loki"?

With this question in mind, we spent some time researching and ended up choosing ClickHouse for several reasons.

1. ClickHouse provides HTTP interface for easy calling of other modules.
2. ClickHouse-based analysis toolbox is very mature and can meet our needs for log analysis.
3. ClickHouse supports the use of object storage as a storage engine, very convenient.
4. There is no need to repeat the "tool-building" process.

There is only one problem left to solve: how to implement the interface between Apache APISIX and ClickHouse? A good way to do this is in the form of a plugin. As a member of the Apache APISIX community, I rarely speak up in the community. Seeing the recent progress in the ecosystem of Apache APISIX, I was actually a bit excited to use Apache APISIX, but I haven't contributed any code to the community yet, so I thought I'd like to take this opportunity to add some fire to the community's ecosystem development.

How It Works​

The clickhouse-logger plugin acts as an middle layer between Apache APISIX and ClickHouse. As mentioned above, we use Apache APISIX as a seven-layer load balancer and requests passing through Apache APISIX generate logs, such as access log and error log. The clickhouse-logger collects the logs and organizes them according to the log format set by its own metadata. Finally, it relies on a batch processor to send the collated logs to ClickHouse in bulk.

Click to Preview

clickhouse-logger serves as a replacement for "tcp-logger+rsyslog+Promtail+Loki" in our scenario. Eliminating the need for format conversion and data forwarding between multiple components, log data requests can be pushed directly to the ClickHouse server.

Click to Preview

How to Use the ClickHouse Plugin​

Here is a sample process to enable the clickhouse-logger plugin in a route.

Enable the ClickHouse Plugin​

Run the curl command to enable the clickhouse-logger plugin for the specified route.

curl http://127.0.0.1:9080/apisix/admin/routes/1 \
-H 'X-API-KEY: edd1c9f034335f136f87ad84b625c8f1' -X PUT -d '
{
      "plugins": {
            "clickhouse-logger": {
                "user": "default",
                "password": "a",
                "database": "default",
                "logtable": "test",
                "endpoint_addr": "http://127.0.0.1:8123"
            }
       },
      "upstream": {
           "type": "roundrobin",
           "nodes": {
               "127.0.0.1:1980": 1
           }
      },
      "uri": "/hello"
}'

The parameters of clickhouse-logger are listed in the chart below.

Test the ClickHouse Plugin​

1. Use the curl command to test the plugin.

   curl -i http://127.0.0.1:9080/hello
   

   Copy

2. If the following result is returned, it means it is successfully enabled.

   HTTP/1.1 200 OK
   ...
   hello, world
   

   Copy

Advanced Skill: Configure Log Format​

You can set a custom log format using the log_format metadata, as shown in the example below.

1. Configure the log_format metadata parameter.

   curl http://127.0.0.1:9080/apisix/admin/plugin_metadata/clickhouse-logger \
   -H 'X-API-KEY: edd1c9f034335f136f87ad84b625c8f1' -X PUT -d '
   {
       "log_format": {
           "host": "$host",
           "@timestamp": "$time_iso8601",
           "client_ip": "$remote_addr"
       }
   }'
   

   Copy

   Declare the log format as a key-value pair in JSON format. For the value part, only strings are supported. If it starts with$, it indicates that it is to get APISIX built-in variables or Nginx built-in variables. In particular, this setting is global, meaning that specifying log_format will take effect on all routes or services bound to http-logger.

2. Create a table for ClickHouse to write in.

   CREATE TABLE default.test (
     `host` String,
     `client_ip` String,
     `route_id` String,
     `@timestamp` String,
     PRIMARY KEY(`@timestamp`)
   ) ENGINE = MergeTree()
   

   Copy

3. Execute select * from default.test; on ClickHouse, you will get data similar to the following.

   ┌─host──────┬─client_ip─┬─route_id─┬─@timestamp────────────────┐
   │ 127.0.0.1 │ 127.0.0.1 │    1     │ 2022-01-17T10:03:10+08:00 │
   └───────────┴───────────┴──────────┴───────────────────────────┘
   

   Copy

Advanced Skill: Connect Grafana with ClickHouse​

1. Enable the clickhouse-logger plugin globally.

   curl http://127.0.0.1:9080/apisix/admin/global_rules/1 \
   -H 'X-API-KEY: edd1c9f034335f136f87ad84b625c8f1' -X PUT -d '
   {
       "plugins": {
           "clickhouse-logger": {
               "timeout": 3,
               "retry_delay": 1,
               "batch_max_size": 100,
               "user": "default",
               "password": "a",
               "database": "default",
               "logtable": "t",  "max_retry_count": 1,
               "endpoint_addr": "http://127.0.0.1:8123"
           }
       }
   }'
   

   Copy

2. Configure the log_format metadata parameter. log_format must be in the same format as the database table structure, otherwise it will cause write failure.

    curl http://127.0.0.1:9080/apisix/admin/plugin_metadata/clickhouse-logger \
   -H 'X-API-KEY: edd1c9f034335f136f87ad84b625c8f1' -X PUT -d '
   {
       "log_format": {
           "upstream_header_time": "$upstream_header_time",
           "upstream_connect_time": "$upstream_connect_time",
           "status": "$status",
           "host": "$host",
           "body_bytes_sent": "$body_bytes_sent",
           "request": "$request",
           "remote_user": "$remote_user",
           "client_ip": "$remote_addr",
           "content_length": "$content_length",
           "local_time": "$fmt_ms_time_local",
           "http_referer": "$http_referer",
           "http_x_amz_target": "$http_x_amz_target",
           "http_x_request_id": "$http_x_request_id",
           "upstream_response_time": "$upstream_response_time",
           "upstream_status": "$upstream_status",
           "http_user_agent": "$http_user_agent",
           "request_time": "$request_time",
           "upstream_addr": "$upstream_addr",
           "http_host": "$http_host",
           "content_type": "$content_type"
       }
   }'
   

   Copy

Here are some screenshots of the dashboard after interfacing with Clickhouse using Grafana.

Click to Preview

Click to Preview

Click to Preview

Disable the ClickHouse Plugin​

clickhouse-logger can be disabled by removing the appropriate configuration in the plugin configuration. Since the Apache APISIX plugin is hot-loaded, the configuration can be updated without a restart.

curl http://127.0.0.1:9080/apisix/admin/routes/1  \
-H 'X-API-KEY: edd1c9f034335f136f87ad84b625c8f1' -X PUT -d '
{
    "methods": ["GET"],
    "uri": "/hello",
    "plugins": {},
    "upstream": {
        "type": "roundrobin",
        "nodes": {
            "127.0.0.1:1980": 1
        }
    }
}'

Summary​

The above is the whole process of developing clickhouse-logger for Apache APISIX, and how to implement it with Grafana to achieve the same observability with simpler architechure and worlflows. I hope more people in the community will be willing to step out of their comfort zones, switching from followers to contributors is much easier than you think.