forked from docs/doc-exports
Yang, Tong
6182f91ba8
Reviewed-by: Hasko, Vladimir <vladimir.hasko@t-systems.com> Co-authored-by: Yang, Tong <yangtong2@huawei.com> Co-committed-by: Yang, Tong <yangtong2@huawei.com>
19 lines
1.9 KiB
HTML
19 lines
1.9 KiB
HTML
<a name="mrs_01_1790"></a><a name="mrs_01_1790"></a>
|
|
|
|
<h1 class="topictitle1">Why Does a MapReduce Task Stay Unchanged for a Long Time?</h1>
|
|
<div id="body1596092470013"><div class="section" id="mrs_01_1790__sd67067b9f09643fbaa0b49d60ecd1fa8"><h4 class="sectiontitle">Question</h4><p id="mrs_01_1790__a672a1717b4c043e0a24ece96a230c83c">MapReduce job is not progressing for long time</p>
|
|
</div>
|
|
<div class="section" id="mrs_01_1790__s9ba4cde5e2db43f29bd7e6973e2a4569"><h4 class="sectiontitle">Answer</h4><p id="mrs_01_1790__a06dee1c583b944e5ad81628540ef2814">This is because of less memory. When the memory is less, the time taken by the job to copy the map output increases significantly.</p>
|
|
</div>
|
|
<p id="mrs_01_1790__afb0662de2adb415db57fb11e1e31011c">In order to reduce the waiting time, increase the heap memory.</p>
|
|
<p id="mrs_01_1790__a5818b1398ef84f4bb51f6ab23f332786">The job configuration should be tuned according to number of mappers and data size processed by each mapper. Based on the input data size, tune the following configurations accordingly for feasible performance.</p>
|
|
<ul id="mrs_01_1790__ufc7a655fe07b485fa772c172e21ef76d"><li id="mrs_01_1790__lf967b36ca5034db69859c97d158dcf26"><span class="parmname" id="mrs_01_1790__pf755239348c64f70a4ac24b37bc1cb7d"><b>mapreduce.reduce.memory.mb</b></span></li><li id="mrs_01_1790__ladb4df0130ed4fdf8c793405f4cfd74f"><span class="parmname" id="mrs_01_1790__p21454099c39e421fa8c7362e3e0d1b07"><b>mapreduce.reduce.java.opts</b></span></li></ul>
|
|
<p id="mrs_01_1790__ae4f2846ba76a458c885ed1b3c56226a0"><strong id="mrs_01_1790__a51a4169c9d3346d6bdb16049ce185f7b">Example</strong>: If the data size is 5 GB with 10 mappers, then the ideal heap memory would be 1.5 GB. Increase the heap memory size according with the increase in data size.</p>
|
|
</div>
|
|
<div>
|
|
<div class="familylinks">
|
|
<div class="parentlink"><strong>Parent topic:</strong> <a href="mrs_01_1788.html">Common Issues About MapReduce</a></div>
|
|
</div>
|
|
</div>
|
|
|