A probe into the Mechanism of Spark Streaming reverse pressure

1. Principle of reverse pressure mechanism

The reverse pressure mechanism in Spark Streaming is a new feature introduced by Spark 1.5.0, which can dynamically adjust the intake rate according to the processing efficiency.

When the batch processing time (Batch Processing Time) is greater than the batch interval (Batch Interval, that is, BatchDuration), it means that the speed of data processing is less than the speed of data intake, and the duration is too long or the source data increases sharply, which is easy to cause data to accumulate in memory and eventually lead to Executor OOM or task collapse.

In this case, if it is a Kafka Receiver-based data source, you can control the maximum input rate by setting spark.streaming.receiver.maxRate; if it is a Direct-based data source (such as Kafka Direct Stream), you can control the maximum input rate by setting spark.streaming.kafka.maxRatePerPartition. Of course, it is generally not a problem to set these parameters when the pressure test has been carried out in advance and the flow peak will not exceed expectations. However, the maximum value does not mean that it is the optimal value, and it is best to dynamically estimate the optimal rate of the next batch according to the processing of each batch. Above Spark 1.5.0, it can be realized by back pressure mechanism. Turn on the reverse pressure mechanism, that is, setting spark.streaming.backpressure.enabled to true,Spark Streaming will automatically adjust the input rate according to the processing capacity, so as to ensure maximum throughput and performance at peak traffic.

Override def onBatchCompleted (batchCompleted: StreamingListenerBatchCompleted) {val elements = batchCompleted.batchInfo.streamIdToInputInfo for {/ / processing end time processingEnd