How to realize ReceiverTracker

This article mainly explains "How to realize ReceiverTracker". Interested friends may wish to have a look. The method introduced in this paper is simple, fast and practical. Let's let Xiaobian take you to learn "How to realize ReceiverTracker"!

The ReceiverTacker class is as follows, and you can see what this class does from the source comments.

Manage the execution of ReceiverInputDStreams and record metadata information sent by Receiver. The ReceiverTacker class must be constructed with a StreamingContext object.

ReceiverTackerEndpoint is a message communication body inside the ReceiverTacker class, which is used to communicate with ReceiverSupervisorImpl running on the Executor side, including messages such as registration of Receiver, restarting Receiver, clearing previous Block data, updating current limit value, adding Block metadata information, etc.

Next, the specific processing flow will be explained by receiving an AddBlock message to which metadata information is added from ReceiverSupervisorImpl at the Executor end.

ReceivedBlockInfo class contains StreamID, number of records in Block, metadata, storage result of receiving Block (BlockID and number of records)

The ReceiverBlockTracker class is a concrete implementation of the addBlock method.

1. Call WriteToLog method of ReceiverBlockTracker

2. Call the getReceivedBlockQueue method of ReceiverBlockTracker, where streamIdToUnallocatedBlockQueue is HashMap, Key is StreamID, and Value is ReceivedBlockQueue. ReceivedBlockQueue is defined as private type ReceivedBlockQueue = mutable.Queue[ReceivedBlockInfo]

ReceiverBlockTracker class, you can see from the source code, it will record all received Block information, according to the need to assign Block to Batch. If checkpoint is set and WAL is enabled, all operations will be saved to the pre-written log, so when Driver fails, the state of ReceiverTracker can be restored from checkpoint and WAL.

The important method in the ReceiverBlockTracker class is allocateBlocksToBatch. privateval timeToAllocatedBlocks = new mutable.HashMap[Time, AllocatedBlocks] stores the block data allocated at the batch processing time.

This method is called by ReceiverTracker.

ReceiverTracker's allocateBlocksToBatch method is called by JobGenerator's generateJobs method.

The important method in the ReceiverBlockTracker class is getBlocksOfBatch.

This method is called by ReceiverTracker's getBlocksOfBatch.

ReceiverTracker's getBlocksOfBatch method is called by ReceiverInputDStream's compute method.

At this point, I believe everyone has a deeper understanding of "how to realize ReceiverTracker," so let's actually operate it! Here is the website, more related content can enter the relevant channels for inquiry, pay attention to us, continue to learn!