最近一直都在看《Hadoop权威指南（中文版）》，虽然的确是在翻译的方面有很多不如意之处，但是对于我这个英语不是很好的人来说，看中文版的书还是能够大大节约我的时间的。我的本科毕业设计就是关于HDFS和MapReduce的，所以我最近非常想马上编写出自己的MapReduce程序。
从网上看到了一个关于题目所说的非常好的配置方法，自己动手试了试，发现果然非常好用，所以将这个文章转载如下：
1、确定eclipse是关闭的，如果不是的话，弄好之后要关了重新打开才可以。找到hadoop的安装路径，我的是hadoop-0.20.2，在home/hadoop/hadoop-0.20.2/contrib/eclipse-plugin/下有hadoop-0.20.2-eclipse-plugin.jar，将这个jar包拷贝到eclipse安装目录下的plugins里，我的是在usr/eclipse/plugins/下，然后打开eclipse，点击主菜单上的window—preferences，在左边栏中找到Hadoop Map/Reduce，点击后在右边对话框里设置hadoop的安装路径即主目录。
2、创建一个MapReduce Project，点击eclipse主菜单上的File—New—Project，在弹出的对话框中选择MapReduce Project，之后输入Project的名，例如wordcount，确定即可，然后就可以象一个普通的 Eclipse Java project 那样，添加Java类，比如你可以定义一个WordCount 类，然后将你安装的hadoop程序里的WordCount源程序代码（版本不同会有区别），我的是在hadoop-0.20.2/src/examples/org/apache/hadoop/examples/WordCount.java写到此类中(以下是源程序代码)，如果是19版本以前的，添加入必要的 import 语句 ( Eclipse 快捷键 ctrl+shift+o 可以帮你)，即可形成一个完整的 wordcount 程序，然后运行。
import java.io.IOException;
import java.util.StringTokenizer;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.mapreduce.Reducer;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
import org.apache.hadoop.util.GenericOptionsParser;
public class WordCount {
public static class TokenizerMapper
extends Mapper<Object, Text, Text, IntWritable>{
private final static IntWritable one = new IntWritable(1);
private Text word = new Text();
public void map(Object key, Text value, Context context
) throws IOException, InterruptedException {
StringTokenizer itr = new StringTokenizer(value.toString());
while (itr.hasMoreTokens()) {
word.set(itr.nextToken());
context.write(word, one);
}
}
}
public static class IntSumReducer
extends Reducer<Text,IntWritable,Text,IntWritable> {
private IntWritable result = new IntWritable();
public void reduce(Text key, Iterable<IntWritable> values,
Context context
) throws IOException, InterruptedException {
int sum = 0;
for (IntWritable val : values) {
sum += val.get();
}
result.set(sum);
context.write(key, result);
}
}
public static void main(String[] args) throws Exception {
Configuration conf = new Configuration();
String[] otherArgs = new GenericOptionsParser(conf, args).getRemainingArgs();
if (otherArgs.length != 2) {
System.err.println("Usage: wordcount <in> <out>");
System.exit(2);
}
Job job = new Job(conf, "word count");
job.setJarByClass(WordCount.class);
job.setMapperClass(TokenizerMapper.class);
job.setCombinerClass(IntSumReducer.class);
job.setReducerClass(IntSumReducer.class);
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(IntWritable.class);
FileInputFormat.addInputPath(job, new Path(otherArgs[0]));
FileOutputFormat.setOutputPath(job, new Path(otherArgs[1]));
System.exit(job.waitForCompletion(true) ? 0 : 1);
}
}
3、运行时要设置参数，点击主菜单上的Run—Run Configurations对话框左边里选java Application点右键New，右边出现对话框中Arguments，设定程序运行时的两个参数，即输入目录和输出目录，这里的路径是相对于workspace的，不是相对于Hadoop安装路径的，我运行时写的是/home/hadoop/in /home/hadoop/out，之前我在/home/hadoop/里面创建了in文件夹（里面创建文件并写入内容，我的创建了两个文件f1和f2并写入了一些单词），out文件夹不用创建，否则运行时会出现错误，提示out文件夹已经存在，参数写了后点Run即可。
我的运行过程：
10/11/23 17:10:55 INFO jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId=
10/11/23 17:10:55 WARN mapred.JobClient: No job jar file set. User classes may not be found. See JobConf(Class) or JobConf#setJar(String).
10/11/23 17:10:55 INFO input.FileInputFormat: Total input paths to process : 4
10/11/23 17:10:55 INFO mapred.JobClient: Running job: job_local_0001
10/11/23 17:10:55 INFO input.FileInputFormat: Total input paths to process : 4
10/11/23 17:10:55 INFO mapred.MapTask: io.sort.mb = 100
10/11/23 17:10:55 INFO mapred.MapTask: data buffer = 79691776/99614720
10/11/23 17:10:55 INFO mapred.MapTask: record buffer = 262144/327680
10/11/23 17:10:55 INFO mapred.MapTask: Starting flush of map output
10/11/23 17:10:55 INFO mapred.TaskRunner: Task:attempt_local_0001_m_000000_0 is done. And is in the process of commiting
10/11/23 17:10:55 INFO mapred.LocalJobRunner:
10/11/23 17:10:55 INFO mapred.TaskRunner: Task 'attempt_local_0001_m_000000_0' done.
10/11/23 17:10:55 INFO mapred.MapTask: io.sort.mb = 100
10/11/23 17:10:56 INFO mapred.MapTask: data buffer = 79691776/99614720
10/11/23 17:10:56 INFO mapred.MapTask: record buffer = 262144/327680
10/11/23 17:10:56 INFO mapred.MapTask: Starting flush of map output
10/11/23 17:10:56 INFO mapred.TaskRunner: Task:attempt_local_0001_m_000001_0 is done. And is in the process of commiting
10/11/23 17:10:56 INFO mapred.LocalJobRunner:
10/11/23 17:10:56 INFO mapred.TaskRunner: Task 'attempt_local_0001_m_000001_0' done.
10/11/23 17:10:56 INFO mapred.MapTask: io.sort.mb = 100
10/11/23 17:10:56 INFO mapred.MapTask: data buffer = 79691776/99614720
10/11/23 17:10:56 INFO mapred.MapTask: record buffer = 262144/327680
10/11/23 17:10:56 INFO mapred.MapTask: Starting flush of map output
10/11/23 17:10:56 INFO mapred.MapTask: Finished spill 0
10/11/23 17:10:56 INFO mapred.TaskRunner: Task:attempt_local_0001_m_000002_0 is done. And is in the process of commiting
10/11/23 17:10:56 INFO mapred.LocalJobRunner:
10/11/23 17:10:56 INFO mapred.TaskRunner: Task 'attempt_local_0001_m_000002_0' done.
10/11/23 17:10:56 INFO mapred.MapTask: io.sort.mb = 100
10/11/23 17:10:56 INFO mapred.MapTask: data buffer = 79691776/99614720
10/11/23 17:10:56 INFO mapred.MapTask: record buffer = 262144/327680
10/11/23 17:10:56 INFO mapred.MapTask: Starting flush of map output
10/11/23 17:10:56 INFO mapred.JobClient: map 100% reduce 0%
10/11/23 17:10:56 INFO mapred.MapTask: Finished spill 0
10/11/23 17:10:56 INFO mapred.TaskRunner: Task:attempt_local_0001_m_000003_0 is done. And is in the process of commiting
10/11/23 17:10:56 INFO mapred.LocalJobRunner:
10/11/23 17:10:56 INFO mapred.TaskRunner: Task 'attempt_local_0001_m_000003_0' done.
10/11/23 17:10:56 INFO mapred.LocalJobRunner:
10/11/23 17:10:56 INFO mapred.Merger: Merging 4 sorted segments
10/11/23 17:10:56 INFO mapred.Merger: Down to the last merge-pass, with 2 segments left of total size: 135 bytes
10/11/23 17:10:56 INFO mapred.LocalJobRunner:
10/11/23 17:10:56 INFO mapred.TaskRunner: Task:attempt_local_0001_r_000000_0 is done. And is in the process of commiting
10/11/23 17:10:56 INFO mapred.LocalJobRunner:
10/11/23 17:10:56 INFO mapred.TaskRunner: Task attempt_local_0001_r_000000_0 is allowed to commit now
10/11/23 17:10:56 INFO output.FileOutputCommitter: Saved output of task 'attempt_local_0001_r_000000_0' to /home/hadoop/out
10/11/23 17:10:56 INFO mapred.LocalJobRunner: reduce > reduce
10/11/23 17:10:56 INFO mapred.TaskRunner: Task 'attempt_local_0001_r_000000_0' done.
10/11/23 17:10:57 INFO mapred.JobClient: map 100% reduce 100%
10/11/23 17:10:57 INFO mapred.JobClient: Job complete: job_local_0001
10/11/23 17:10:57 INFO mapred.JobClient: Counters: 12
10/11/23 17:10:57 INFO mapred.JobClient: FileSystemCounters
10/11/23 17:10:57 INFO mapred.JobClient: FILE_BYTES_READ=66060
10/11/23 17:10:57 INFO mapred.JobClient: FILE_BYTES_WRITTEN=136124
10/11/23 17:10:57 INFO mapred.JobClient: Map-Reduce Framework
10/11/23 17:10:57 INFO mapred.JobClient: Reduce input groups=9
10/11/23 17:10:57 INFO mapred.JobClient: Combine output records=12
10/11/23 17:10:57 INFO mapred.JobClient: Map input records=2
10/11/23 17:10:57 INFO mapred.JobClient: Reduce shuffle bytes=0
10/11/23 17:10:57 INFO mapred.JobClient: Reduce output records=9
10/11/23 17:10:57 INFO mapred.JobClient: Spilled Records=24
10/11/23 17:10:57 INFO mapred.JobClient: Map output bytes=107
10/11/23 17:10:57 INFO mapred.JobClient: Combine input records=12
10/11/23 17:10:57 INFO mapred.JobClient: Map output records=12
10/11/23 17:10:57 INFO mapred.JobClient: Reduce input records=12
运行结果：查看运行时设置的输出目录/home/hadoop/out，即为运行结果
come 1
fly 2
follow 1
heart 1
hello 2
moto 1
my 1
on 1
tree 2
真的很神奇，原来我苦苦寻找编写MapReduce程序的办法竟然如此简单，我一直都在网上找各种方法，以为要下载什么Eclipse的插件等等，原来Hadoop-0.20.2里就自带了这个插件，而Eclipse竟然也原生的支持MapReduce程序的开发，看来开源的力量真的很伟大啊！