How to integrate Prometheus/Pushgateway into Apache Hudi for Monitoring
This article mainly introduces how Apache Hudi integrates Prometheus/Pushgateway for monitoring. The article is very detailed and has certain reference value. Interested friends must read it!
1. availability
In Hudi's newest master branch, prometheus monitors Hudi application metrics, which will be available in the upcoming version 0.6.0.
2. profile
Prometheus is a popular surveillance service. In the upcoming Apache Hudi 0.6.0 release, in addition to the existing reporter types (Graphite, JMX, DATADOG, user-defined metrics), we will introduce the ability to report Hudi metrics via pushgateway and httpserver.
3. Configuration 3.1 pushgateway
Similar to other supported reporters, enabling pushgateway reporters requires configuring the following properties.
hoodie.metrics.on=true
hoodie.metrics.reporter.type=PROMETHEUS_PUSHGATEWAY
The following properties are used to configure the address and port number of the pushgateway. Address default is localhost, port default is 9091
hoodie.metrics.pushgateway.host=x.x.x.x
hoodie.metrics.pushgateway.port=9091
The following properties are used to configure whether to delete monitoring information from pushgateway at the end of the task. The default is true.
hoodie.metrics.pushgateway.delete.on.shutdown=false
The following attributes are used to configure the task name prefix and whether a random suffix is required. The default is true.
hoodie.metrics.pushgateway.job.name=xiaomm
hoodie.metrics.pushgateway.random.job.name.suffix=false
The following properties are used to configure whether to delete monitoring information from pushgateway at the end of the task. The default is true.
hoodie.metrics.pushgateway.delete.on.shutdown=false3.2 prometheus-httpserverhoodie.metrics.on=true
hoodie.metrics.reporter.type=PROMETHEUS
hoodie.metrics.reporter.host=9090
This concludes the configuration description
4. example demonstrates
In this example, we run a HoodieDeltaStreamer with metric collection enabled and configured accordingly.
The above is "Apache Hudi how to integrate Prometheus/Pushgateway monitoring" all the content of this article, thank you for reading! Hope to share the content to help everyone, more relevant knowledge, welcome to pay attention to the industry information channel!
