Analysis of RDD Transformation and Action source code

Wordcount.toDebugString views the inheritance chain of RDD

So in a broad sense, performing an operation on any function can be regarded as an operator, even including finding the power, and the square can be regarded as an operator, but for some operators, we use a symbol to replace the operation he is going to do, so don't struggle when you see the operator. It is no different from the f of f (x), and it is even no different from the basic operation symbol of addition, subtraction, multiplication and division. It's just that he can operate on a single object (some symbols, such as greater than and less than symbols, operate on multiple objects). For example, take the probability P {X

For (v TextOutputFormat (key is null,value and element toString)

* * saveAsObjectFile** (binary)-> saveAsSequenceFile- > SequenceFileOutputFormat (key is null,value and BytesWritable)

Cache\ persist

* * checkpoint () * * mechanism avoids cache loss (insufficient memory) the performance overhead caused by recalculation will result in another job, which is more reliable than cache.

SparkContex.setCheckpointDir sets the directory location