An Example Configurable Partitioner

In the new Hadoop API, if you’re writing a custom Partitioner that needs to be configured, you make it Configurable.

class CustomPartitioner extends Partitioner<KEY, VALUE> implements Configurable {
     private Configuration configuration;

     @Override
      public int getPartition(KEY key, VALUE value, int n) {
           // Calculate the hash code, reading from the configuration as needed.
           return Math.abs(hashCode) % n;
      }

      @Override
      public void setConf(Configuration configuration) {
         this.configuration = configuration;
      }

      @Override
      public Configuration getConf() {
         return configuration;
      }
}

During initialization the Hadoop framework will create an instance of your custom partitioner. When Hadoop’s reflection utilities instantiate a class, they check to see if that class is Configurable. If it is, setConf is called immediately after creation. So in the template above, the private configuration variable will be initialized before the first call to getPartition.

Advertisements
This entry was posted in Those that have just broken the flower vase. Bookmark the permalink.

4 Responses to An Example Configurable Partitioner

  1. Harsh says:

    Great post, I’ll make sure I pass it around when Partitioners ever come up on MLs/etc. 🙂

  2. abhijit says:

    Thanks a lot. Very good piece of information 🙂

  3. Sachin Sudarshana says:

    Hey.. Thanks a lot. This helped a lot. Is it possible to do the same with a Custom Comparator class, that implements WritableComparable? No luck so far.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s