hadoop&spark mapreduce对比 & 框架设计和理解

Hadoop MapReduce:

MapReduce在每次执行的时候都要从磁盘读数据，计算完毕后都要把数据放到磁盘

spark map reduce:

 

 

RDD is everything for dev:

Basic Concepts:

Graph RDD:

Spark Runtime:

schedule:

Depency Type:

Scheduler Optimizations：

Event Flow:

Submit Job:

New Job Instance:

Job In Detail:

executor.launchTask:

Standalone：

 

 

Work Flow:

Standalone detail:

Driver application to Clustor:

Worker Exception:

Executor Exception:

Master Exception:

Master HA: