Updated at: 2022-12-09 03:49:50
Apache Kafka is an open source distributed message system. Many enterprises will use Kafka to realize high-performance data pipeline, data stream analysis, data integration and other functions. There will be a large number of Kafka clients in their digital systems. Therefore, there will be a large amount of analyzable data in Kafka client, and operator can send the data to AnyRoboot's Kafka server cluster in Kafka protocol through Kafka client to realize Kafka log aggregation.
AnyRobot can realize Kafka data receiving through NodePort to open the external network accessibility of Kafka server cluster.
_15.png)
Note: Kafka log aggregation only supports clients of Kafka 2.3.1 and above.
The configuration process of Kafka log aggregation is as follows:
1. Create NodePort service:
Create the corresponding NodePort service at each Kafka node of AnyRobot, configure the externally accessible IP address for each Kafka node and allocate the NodePort type service port. To extend Kafka node, you can manually add the corresponding NodePort service;
2. Modify the Kafka client configuration file, and realize communication with AnyRobot at the Kafka client through the externally accessible IP address and NodePort port configured in step 1;
3. Receive Kafka log:
At Kafka client, the operator can customize the Topic to be sent to AnyRobot, and the log type of Kafka data received by AnyRobot is the Topic name corresponding to the data by default.
Note:
1. For configuration process details of Kafka log aggregation, please refer to AnyRobot Deployment Guide;
2. When configuring the Topic name at Kafka client, it is not allowed to start with the existing Topic name of the system. The existing Topic names of the system include: kafka_topic, anyrobot_flink, anyrobot_flink_config, alert_rule_intermediate, k8s_event, k8s_log, reindex_source, and reindex_result.
AnyRobot can realize Kafka data receiving through NodePort to open the external network accessibility of Kafka server cluster.
Note: Kafka log aggregation only supports clients of Kafka 2.3.1 and above.The configuration process of Kafka log aggregation is as follows:
1. Create NodePort service:
Create the corresponding NodePort service at each Kafka node of AnyRobot, configure the externally accessible IP address for each Kafka node and allocate the NodePort type service port. To extend Kafka node, you can manually add the corresponding NodePort service;
2. Modify the Kafka client configuration file, and realize communication with AnyRobot at the Kafka client through the externally accessible IP address and NodePort port configured in step 1;
3. Receive Kafka log:
At Kafka client, the operator can customize the Topic to be sent to AnyRobot, and the log type of Kafka data received by AnyRobot is the Topic name corresponding to the data by default.
Note: 1. For configuration process details of Kafka log aggregation, please refer to AnyRobot Deployment Guide;
2. When configuring the Topic name at Kafka client, it is not allowed to start with the existing Topic name of the system. The existing Topic names of the system include: kafka_topic, anyrobot_flink, anyrobot_flink_config, alert_rule_intermediate, k8s_event, k8s_log, reindex_source, and reindex_result.
< Previous:
Next: >