Thursday, 26 December 2013

In-mapper Combiner Program to Calculate Average

In-mapper Combiner Program to Calculate Average

In my previous post I described how we can use in-mapper combiner to make our M/R program more efficient. In the post, we also saw both M/R algorithm for average calculation with/without using in-mapper combiner optimization.

In this post I am posting codes for both the algorithm.

The M/R program to calculate average without in-mapper combiner is given below:

The M/R program to calculate average with in-mapper combiner is given below:


The programs took 56sec and 42sec respectively for execution on my laptop for 10million records. So we can see a 33% improvement in time while using in-mapper combiner program.

4 comments:

  1. Hi please can you share the code of IntPair class?

    ReplyDelete
    Replies
    1. Completely forgot to write about it, just posted it here - http://rishavrohitblog.blogspot.com/2014/02/implementing-custom-writablecomparable.html

      Delete
  2. Hi, your post on combiner programes were very easy to understand thanks for the useful post
    Hadoop Training in Velachery | Hadoop Training .
    Hadoop Training in Chennai | Hadoop .

    ReplyDelete
  3. Nice post ! Thanks for sharing valuable information with us. Keep sharing..Big data hadoop online Training Course

    ReplyDelete