mapreduce过程以及相关类总结
每个存放文件的split进行处理map过程,map之后放入hash分区的reduce过程,问题:怎么确定map个数
![enter description here enter description here](./images/1585961816402.png)
enter description here
shuffle阶段从map到reduce,放入对应hash好的reduce块中,下图是缓冲过程
![enter description here enter description here](https://raw.githubusercontent.com/miaozhijuan/xiaoshujiang/master/%E5%B0%8F%E4%B9%A6%E5%8C%A0/1585961794204.png)
enter description here
如何确定reduce给谁或者几个,由partition决定
![enter description here enter description here](https://raw.githubusercontent.com/miaozhijuan/xiaoshujiang/master/%E5%B0%8F%E4%B9%A6%E5%8C%A0/1585962105328.png)
enter description here
combine过程,会在写入文件前各个spill块先进行一次combine,然后reduce的时候再进行一次分区上的总combine操作,combine实际和reduce操作做的一样
![enter description here enter description here](https://raw.githubusercontent.com/miaozhijuan/xiaoshujiang/master/%E5%B0%8F%E4%B9%A6%E5%8C%A0/1585962218334.png)
enter description here
hadoop1.0进程关系,maptask和reducetask
![enter description here enter description here](https://raw.githubusercontent.com/miaozhijuan/xiaoshujiang/master/%E5%B0%8F%E4%B9%A6%E5%8C%A0/1585962282157.png)
enter description here