simple way for sorting in secondary keys using hadoop

A Useful Partitioner Class (secondary sort, the -partitioner org.apache.hadoop.mapred.lib.KeyFieldBasedPartitioner option)

Hadoop has a library class, KeyFieldBasedPartitioner, that is useful for many applications. This class allows the Map/Reduce framework to partition the map outputs based on certain key fields, not the whole keys. For example:

$HADOOP_HOME/bin/hadoop  jar $HADOOP_HOME/hadoop-streaming.jar 
    -input myInputDirs 
    -output myOutputDir 
    -mapper org.apache.hadoop.mapred.lib.IdentityMapper 
    -reducer org.apache.hadoop.mapred.lib.IdentityReducer 
    -partitioner org.apache.hadoop.mapred.lib.KeyFieldBasedPartitioner 
    -D stream.map.output.field.separator=. 
    -D stream.num.map.output.key.fields=4 
    -D map.output.key.field.separator=. 
    -D mapred.text.key.partitioner.options=-k1,2
    -D mapred.reduce.tasks=12

Here, -D stream.map.output.field.separator=. and -D stream.num.map.output.key.fields=4 are as explained in previous example. The two variables are used by streaming to identify the key/value pair of mapper.

The map output keys of the above Map/Reduce job normally have four fields separated by ".". However, the Map/Reduce framework will partition the map outputs by the first two fields of the keys using the -D mapred.text.key.partitioner.options=-k1,2 option. Here, -D map.output.key.field.separator=. specifies the separator for the partition. This guarantees that all the key/value pairs with the same first two fields in the keys will be partitioned into the same reducer.

This is effectively equivalent to specifying the first two fields as the primary key and the next two fields as the secondary. The primary key is used for partitioning, and the combination of the primary and secondary keys is used for sorting. A simple illustration is shown here:

Output of map (the keys)

11.12.1.2
11.14.2.3
11.11.4.1
11.12.1.1
11.14.2.2

Partition into 3 reducers (the first 2 fields are used as keys for partition)

11.11.4.1
-----------
11.12.1.2
11.12.1.1
-----------
11.14.2.3
11.14.2.2

Sorting within each partition for the reducer(all 4 fields used for sorting)

11.11.4.1
-----------
11.12.1.1
11.12.1.2
-----------
11.14.2.2
11.14.2.3