【Hadoop】中map与reduce的个数问题 |
您所在的位置:网站首页 › 素因子个数表达式怎么算出来的 › 【Hadoop】中map与reduce的个数问题 |
在hadoop中当一个任务没有设置的时候,该任务的执行的map的个数是由任务本身的数据量决定的,具体计算方法会在下文说明;而reduce的个数hadoop是默认设置为1的。为何设置为1那,因为一个任务的输出的文件个数是由reduce的个数来决定的。一般一个任务的结果默认是输出到一个文件中,所以reduce的数目设置为1。那如果我们为了提高任务的执行速度如何对map与reduce的个数来进行调整那。 在讲解之前首先,看一下hadoop官方文档是如何说明的。 Number of Maps The number of maps is usually driven by the number of DFS blocks in the input files. Although that causes people to adjust their DFS block size to adjust the number of maps. The right level of parallelism for maps seems to be around 10-100 maps/node, although we have taken it up to 300 or so for very cpu-light map tasks. Task setup takes awhile, so it is best if the maps take at least a minute to execute. Actually controlling the number of maps is subtle. The mapred.map.tasks parameter is just a hint to the InputFormat for the number of maps. The default InputFormat behavior is to split the total number of bytes into the right number of fragments. However, in the default case the DFS block size of the input files is treated as an upper bound for input splits. A lower bound on the split size can be set via mapred.min.split.size. Thus, if you expect 10TB of input data and have 128MB DFS blocks, you'll end up with 82k maps, unless your mapred.map.tasks is even larger. Ultimately the InputFormat determines the number of maps. The number of map tasks can also be increased manually using the JobConf's conf.setNumMapTasks(int num). This can be used to increase the number of map tasks, but will not set the number below that which Hadoop determines via splitting the input data. Number of Reduces The right number of reduces seems to be 0.95 or 1.75 * (nodes * mapred.tasktracker.tasks.maximum). At 0.95 all of the reduces can launch immediately and start transfering map outputs as the maps finish. At 1.75 the faster nodes will finish their first round of reduces and launch a second round of reduces doing a much better job of load balancing. Currently the number of reduces is limited to roughly 1000 by the buffer size for the output files (io.buffer.size * 2 * numReduces |
今日新闻 |
点击排行 |
|
推荐新闻 |
|
图片新闻 |
|
专题文章 |
CopyRight 2018-2019 实验室设备网 版权所有 win10的实时保护怎么永久关闭 |