Shulou(Shulou.com)06/03 Report--

Author | Bai Song

1. Add a class to write the amount of messages sent in each super step into the Counter of Hadoop. Create a new GiraphMessages class under the org.apache.giraph.counters package to count the number of messages.

The source code is as follows:

Package org.apache.giraph.counters;import java.util.Iterator;import java.util.Map;import org.apache.hadoop.mapreduce.Mapper.Context;import com.google.common.collect.Maps;/** * Hadoop Counters in group "Giraph Messages" for counting every superstep * message count. * / public class GiraphMessages extends HadoopCountersBase {/ * * Counter group name for the giraph Messages * / public static final String GROUP_NAME = "GiraphMessages"; / * * Singleton instance for everyone to use * / private static GiraphMessages INSTANCE; / * * superstep time in msec * / private final Map superstepMessages; private GiraphMessages (Context context) {super (context, GROUP_NAME); superstepMessages = Maps.newHashMap ();} / * * Instantiate with Hadoop Context. * * @ param context * Hadoop Context to use. * / public static void init (Context context) {INSTANCE = new GiraphMessages (context);} / * * Get singleton instance. * * @ return singleton GiraphTimers instance. * / public static GiraphMessages getInstance () {return INSTANCE;} / * Get counter for superstep messages * * @ param superstep * @ return * / public GiraphHadoopCounter getSuperstepMessages (long superstep) {GiraphHadoopCounter counter = superstepMessages.get (superstep); if (counter = = null) {String counterPrefix = "Superstep-" + superstep+ ""; counter = getCounter (counterPrefix) SuperstepMessages.put (superstep, counter);} return counter;} @ Override public Iterator iterator () {return superstepMessages.values () .iterator ();}}

2. Add statistics to the BspServiceMaster class. Each time Master synchronizes, it aggregates the amount of messages sent by each Worker (summation) and stores them in GlobalStats. So you only need to take the total traffic size from the GlobalStats object after each synchronization and write it to the GiraphMessages. The format is actually stored in the Map superstepMessages object defined in the GiraphMessages class in the previous step. In the constructor of BspServiceMaster, the last line of code is appended to initialize the GiraphMessages.

GiraphMessages.init (context)

In the SuperstepState coordinateSuperstep () method of the BspServiceMaster class, add the recording function. The snippet code is as follows:

…… Start recording from the 0th overstep.

3. The experimental results are as follows:

Finish it!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.