Hadoop Eclipse plug-in (installed in Eclipse IDE or Spring Tool Suite) eases the experience of Map/Reduce on Hadoop. Hadoop distribution does not include Hadoop Eclipse plug-in jar, but includes source code of that plug-in.
In this post, we'll see all the steps to build Hadoop eclipse plug-in from source code and install that plug-in in Eclipse / Spring Tool Suite to access Hadoop Environment in "Map/Reduce" perspective of the IDE.
Tools and Technologies used in this article
Navigate to
1. Try Ant Build
Lets us first see, what happens if we try to build Hadoop Eclipse plug-in from the provided source code.
Terminal
static-217:hadoop srccodes$ cd /Users/srccodes/hadoop/src/contrib/eclipse-plugin
static-217:eclipse-plugin srccodes$ ant jar
Buildfile: /Users/srccodes/hadoop/src/contrib/eclipse-plugin/build.xml
check-contrib:
[echo] eclipse.home unset: skipping eclipse plugin
init:
ivy-download:
[get] Getting: http://repo2.maven.org/maven2/org/apache/ivy/ivy/2.1.0/ivy-2.1.0.jar
[get] To: /Users/srccodes/hadoop/ivy/ivy-2.1.0.jar
[get] Not modified - so not downloaded
ivy-probe-antlib:
ivy-init-antlib:
ivy-init:
[ivy:configure] :: Ivy 2.1.0 - 20090925235825 :: http://ant.apache.org/ivy/ ::
[ivy:configure] :: loading settings :: file = /Users/srccodes/hadoop/ivy/ivysettings.xml
ivy-resolve-common:
ivy-retrieve-common:
[ivy:cachepath] DEPRECATED: 'ivy.conf.file' is deprecated, use 'ivy.settings.file' instead
[ivy:cachepath] :: loading settings :: file = /Users/srccodes/hadoop/ivy/ivysettings.xml
compile:
jar:
BUILD SUCCESSFUL
Total time: 2 seconds
Above echo message indicates that eclipse.home is not set and build of the plugin is skipped.
2. Modify build.properties
Open build.properties in a text editor and set eclipse.home with your Eclipse / STS installation directory. Also define version of different jars required by Hadoop Eclipse plug-in.
build.properties
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
output.. = bin/
bin.includes = META-INF/,\
plugin.xml,\
resources/,\
classes/,\
classes/,\
lib/
# set eclipse installation path
eclipse.home=/Users/srccodes/software/springsource/sts-3.2.0.RELEASE
# version of different jars required by Hadoop eclipse plugin
version=1.2.0
commons-cli.version=1.2
commons-configuration.version=1.6
commons-httpclient.version=3.0.1
commons-lang.version=2.4
jackson-core-asl.version=1.8.8
jackson-mapper-asl.version=1.8.8
3. Try Ant Build Once Again
Now eclipse.home is set, but we are not done yet. To find out why, try ant build once again. This time we will get java compilation errors as shown below.
Terminal
static-166:eclipse-plugin srccodes$ ant jar
Buildfile: /Users/srccodes/hadoop/src/contrib/eclipse-plugin/build.xml
check-contrib:
init:
[echo] contrib: eclipse-plugin
init-contrib:
ivy-download:
[get] Getting: http://repo2.maven.org/maven2/org/apache/ivy/ivy/2.1.0/ivy-2.1.0.jar
[get] To: /Users/srccodes/hadoop/ivy/ivy-2.1.0.jar
[get] Not modified - so not downloaded
ivy-probe-antlib:
ivy-init-antlib:
ivy-init:
[ivy:configure] :: Ivy 2.1.0 - 20090925235825 :: http://ant.apache.org/ivy/ ::
[ivy:configure] :: loading settings :: file = /Users/srccodes/hadoop/ivy/ivysettings.xml
ivy-resolve-common:
ivy-retrieve-common:
[ivy:cachepath] DEPRECATED: 'ivy.conf.file' is deprecated, use 'ivy.settings.file' instead
[ivy:cachepath] :: loading settings :: file = /Users/srccodes/hadoop/ivy/ivysettings.xml
compile:
[echo] contrib: eclipse-plugin
[javac] /Users/srccodes/hadoop/src/contrib/eclipse-plugin/build.xml:62: warning: 'includeantruntime' was not set, defaulting to build.sysclasspath=last; set to false for repeatable builds
[javac] Compiling 45 source files to /Users/srccodes/hadoop/build/contrib/eclipse-plugin/classes
[javac] /Users/srccodes/hadoop/src/contrib/eclipse-plugin/src/java/org/apache/hadoop/eclipse/dfs/DFSFolder.java:28: package org.apache.hadoop.fs does not exist
[javac] import org.apache.hadoop.fs.FileStatus;
[javac] ^
[javac] /Users/srccodes/hadoop/src/contrib/eclipse-plugin/src/java/org/apache/hadoop/eclipse/dfs/DFSFolder.java:29: package org.apache.hadoop.fs does not exist
[javac] import org.apache.hadoop.fs.Path;
[javac] ^
[javac] /Users/srccodes/hadoop/src/contrib/eclipse-plugin/src/java/org/apache/hadoop/eclipse/dfs/DFSPath.java:25: package org.apache.hadoop.hdfs does not exist
[javac] import org.apache.hadoop.hdfs.DistributedFileSystem;
[javac] ^
:
:
:
:
[javac] ^
[javac] /Users/srccodes/hadoop/src/contrib/eclipse-plugin/src/java/org/apache/hadoop/eclipse/server/HadoopServer.java:291: cannot find symbol
[javac] symbol : class Configuration
[javac] location: class org.apache.hadoop.eclipse.server.HadoopServer
[javac] public Configuration getConfiguration() {
[javac] ^
[javac] /Users/srccodes/hadoop/src/contrib/eclipse-plugin/src/java/org/apache/hadoop/eclipse/server/ConfProp.java:24: package org.apache.hadoop.conf does not exist
[javac] import org.apache.hadoop.conf.Configuration;
[javac] ^
[javac] /Users/srccodes/hadoop/src/contrib/eclipse-plugin/src/java/org/apache/hadoop/eclipse/server/HadoopServer.java:468: cannot find symbol
[javac] symbol : class FileSystem
[javac] location: class org.apache.hadoop.eclipse.server.HadoopServer
[javac] public FileSystem getDFS() throws IOException {
[javac] ^
:
:
:
:
[javac] /Users/srccodes/hadoop/src/contrib/eclipse-plugin/src/java/org/apache/hadoop/eclipse/server/HadoopJob.java:302: package Counters does not exist
[javac] Counters.Group group = counters.getGroup(groupName);
[javac] ^
[javac] /Users/srccodes/hadoop/src/contrib/eclipse-plugin/src/java/org/apache/hadoop/eclipse/server/HadoopJob.java:305: package Counters does not exist
[javac] for (Counters.Counter counter : group) {
[javac] ^
[javac] /Users/srccodes/hadoop/src/contrib/eclipse-plugin/src/java/org/apache/hadoop/eclipse/dfs/DFSFile.java:74: cannot find symbol
[javac] symbol : class FileStatus
[javac] location: class org.apache.hadoop.eclipse.dfs.DFSFile
[javac] FileStatus fs = getDFS().getFileStatus(path);
[javac] ^
[javac] Note: Some input files use or override a deprecated API.
[javac] Note: Recompile with -Xlint:deprecation for details.
[javac] Note: Some input files use unchecked or unsafe operations.
[javac] Note: Recompile with -Xlint:unchecked for details.
[javac] 100 errors
BUILD FAILED
/Users/srccodes/hadoop/src/contrib/eclipse-plugin/build.xml:62: Compile failed; see the compiler error output for details.
Total time: 3 seconds
4. Modify build.xml
Open build.xml and modify path (id="classpath") and target (name="jar") as highlighted below.
build.xml
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<!--
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
<project default="jar" name="eclipse-plugin">
<import file="../build-contrib.xml"/>
<path id="eclipse-sdk-jars">
<fileset dir="${eclipse.home}/plugins/">
<include name="org.eclipse.ui*.jar"/>
<include name="org.eclipse.jdt*.jar"/>
<include name="org.eclipse.core*.jar"/>
<include name="org.eclipse.equinox*.jar"/>
<include name="org.eclipse.debug*.jar"/>
<include name="org.eclipse.osgi*.jar"/>
<include name="org.eclipse.swt*.jar"/>
<include name="org.eclipse.jface*.jar"/>
<include name="org.eclipse.team.cvs.ssh2*.jar"/>
<include name="com.jcraft.jsch*.jar"/>
</fileset>
</path>
<!-- Override classpath to include Eclipse SDK jars -->
<path id="classpath">
<pathelement location="${build.classes}"/>
<pathelement location="${hadoop.root}/build/classes"/>
<path refid="eclipse-sdk-jars"/>
<fileset dir="${hadoop.root}">
<include name="*.jar" />
</fileset>
</path>
<!-- Skip building if eclipse.home is unset. -->
<target name="check-contrib" unless="eclipse.home">
<property name="skip.contrib" value="yes"/>
<echo message="eclipse.home unset: skipping eclipse plugin"/>
</target>
<target name="compile" depends="init, ivy-retrieve-common" unless="skip.contrib">
<echo message="contrib: ${name}"/>
<javac
encoding="${build.encoding}"
srcdir="${src.dir}"
includes="**/*.java"
destdir="${build.classes}"
debug="${javac.debug}"
deprecation="${javac.deprecation}">
<classpath refid="classpath"/>
</javac>
</target>
<!-- Override jar target to specify manifest -->
<target name="jar" depends="compile" unless="skip.contrib">
<mkdir dir="${build.dir}/lib"/>
<copy file="${hadoop.root}/hadoop-core-${version}.jar" tofile="${build.dir}/lib/hadoop-core.jar" verbose="true"/>
<copy file="${hadoop.root}/lib/commons-cli-${commons-cli.version}.jar" todir="${build.dir}/lib" verbose="true"/>
<copy file="${hadoop.root}/lib/commons-configuration-${commons-configuration.version}.jar" todir="${build.dir}/lib" verbose="true"/>
<copy file="${hadoop.root}/lib/commons-httpclient-${commons-httpclient.version}.jar" todir="${build.dir}/lib" verbose="true"/>
<copy file="${hadoop.root}/lib/commons-lang-${commons-lang.version}.jar" todir="${build.dir}/lib" verbose="true"/>
<copy file="${hadoop.root}/lib/jackson-core-asl-${jackson-core-asl.version}.jar" todir="${build.dir}/lib" verbose="true"/>
<copy file="${hadoop.root}/lib/jackson-mapper-asl-${jackson-mapper-asl.version}.jar" todir="${build.dir}/lib" verbose="true"/>
<jar
jarfile="${build.dir}/hadoop-${name}-${version}.jar"
manifest="${root}/META-INF/MANIFEST.MF">
<fileset dir="${build.dir}" includes="classes/ lib/"/>
<fileset dir="${root}" includes="resources/ plugin.xml"/>
</jar>
</target>
</project>
Note: Hadoop Eclipse Plug-in code is dependent on Hadoop jars. So it is required to add those in classpath of 'compile' target. Also we need to include jars required by the eclipse plugin in 'jar' target.
5. Modify MANIFEST.MF
Open MANIFEST.MF in text editor and modify Bundle-ClassPath as highlighted below to incorporate the changes made in build.xml.
META-INF/MANIFEST.MF
Manifest-Version: 1.0
Bundle-ManifestVersion: 2
Bundle-Name: MapReduce Tools for Eclipse
Bundle-SymbolicName: org.apache.hadoop.eclipse;singleton:=true
Bundle-Version: 0.18
Bundle-Activator: org.apache.hadoop.eclipse.Activator
Bundle-Localization: plugin
Require-Bundle: org.eclipse.ui,
org.eclipse.core.runtime,
org.eclipse.jdt.launching,
org.eclipse.debug.core,
org.eclipse.jdt,
org.eclipse.jdt.core,
org.eclipse.core.resources,
org.eclipse.ui.ide,
org.eclipse.jdt.ui,
org.eclipse.debug.ui,
org.eclipse.jdt.debug.ui,
org.eclipse.core.expressions,
org.eclipse.ui.cheatsheets,
org.eclipse.ui.console,
org.eclipse.ui.navigator,
org.eclipse.core.filesystem,
org.apache.commons.logging
Eclipse-LazyStart: true
Bundle-ClassPath: classes/,
lib/hadoop-core.jar,
lib/commons-cli-1.2.jar,
lib/commons-configuration-1.6.jar,
lib/jackson-core-asl-1.8.8.jar,
lib/commons-httpclient-3.0.1.jar,
lib/jackson-mapper-asl-1.8.8.jar,
lib/commons-lang-2.4.jar
Bundle-Vendor: Apache Hadoop
6. Build Hadoop Eclipse Plug-in
Execute ant jar or ant clean package. On successful build, Hadoop Eclipse Plug-in jar (hadoop-eclipse-plugin-1.2.0.jar) will be created inside
Terminal
static-217:eclipse-plugin srccodes$ ant jar
Buildfile: /Users/srccodes/hadoop/src/contrib/eclipse-plugin/build.xml
check-contrib:
init:
[echo] contrib: eclipse-plugin
[mkdir] Created dir: /Users/srccodes/hadoop/build/contrib/eclipse-plugin
[mkdir] Created dir: /Users/srccodes/hadoop/build/contrib/eclipse-plugin/classes
[mkdir] Created dir: /Users/srccodes/hadoop/build/contrib/eclipse-plugin/test
[mkdir] Created dir: /Users/srccodes/hadoop/build/contrib/eclipse-plugin/system
[mkdir] Created dir: /Users/srccodes/hadoop/build/contrib/eclipse-plugin/system/classes
[mkdir] Created dir: /Users/srccodes/hadoop/build/contrib/eclipse-plugin/examples
[mkdir] Created dir: /Users/srccodes/hadoop/build/contrib/eclipse-plugin/test/logs
init-contrib:
ivy-download:
[get] Getting: http://repo2.maven.org/maven2/org/apache/ivy/ivy/2.1.0/ivy-2.1.0.jar
[get] To: /Users/srccodes/hadoop/ivy/ivy-2.1.0.jar
[get] Not modified - so not downloaded
ivy-probe-antlib:
ivy-init-antlib:
ivy-init:
[ivy:configure] :: Ivy 2.1.0 - 20090925235825 :: http://ant.apache.org/ivy/ ::
[ivy:configure] :: loading settings :: file = /Users/srccodes/hadoop/ivy/ivysettings.xml
ivy-resolve-common:
ivy-retrieve-common:
[ivy:cachepath] DEPRECATED: 'ivy.conf.file' is deprecated, use 'ivy.settings.file' instead
[ivy:cachepath] :: loading settings :: file = /Users/srccodes/hadoop/ivy/ivysettings.xml
compile:
[echo] contrib: eclipse-plugin
[javac] /Users/srccodes/hadoop/src/contrib/eclipse-plugin/build.xml:64: warning: 'includeantruntime' was not set, defaulting to build.sysclasspath=last; set to false for repeatable builds
[javac] Compiling 45 source files to /Users/srccodes/hadoop/build/contrib/eclipse-plugin/classes
[javac] Note: Some input files use or override a deprecated API.
[javac] Note: Recompile with -Xlint:deprecation for details.
[javac] Note: Some input files use unchecked or unsafe operations.
[javac] Note: Recompile with -Xlint:unchecked for details.
jar:
[mkdir] Created dir: /Users/srccodes/hadoop/build/contrib/eclipse-plugin/lib
[copy] Copying 1 file to /Users/srccodes/hadoop/build/contrib/eclipse-plugin/lib
[copy] Copying /Users/srccodes/hadoop/hadoop-core-1.2.0.jar to /Users/srccodes/hadoop/build/contrib/eclipse-plugin/lib/hadoop-core.jar
[copy] Copying 1 file to /Users/srccodes/hadoop/build/contrib/eclipse-plugin/lib
[copy] Copying /Users/srccodes/hadoop/lib/commons-cli-1.2.jar to /Users/srccodes/hadoop/build/contrib/eclipse-plugin/lib/commons-cli-1.2.jar
[copy] Copying 1 file to /Users/srccodes/hadoop/build/contrib/eclipse-plugin/lib
[copy] Copying /Users/srccodes/hadoop/lib/commons-configuration-1.6.jar to /Users/srccodes/hadoop/build/contrib/eclipse-plugin/lib/commons-configuration-1.6.jar
[copy] Copying 1 file to /Users/srccodes/hadoop/build/contrib/eclipse-plugin/lib
[copy] Copying /Users/srccodes/hadoop/lib/commons-httpclient-3.0.1.jar to /Users/srccodes/hadoop/build/contrib/eclipse-plugin/lib/commons-httpclient-3.0.1.jar
[copy] Copying 1 file to /Users/srccodes/hadoop/build/contrib/eclipse-plugin/lib
[copy] Copying /Users/srccodes/hadoop/lib/commons-lang-2.4.jar to /Users/srccodes/hadoop/build/contrib/eclipse-plugin/lib/commons-lang-2.4.jar
[copy] Copying 1 file to /Users/srccodes/hadoop/build/contrib/eclipse-plugin/lib
[copy] Copying /Users/srccodes/hadoop/lib/jackson-core-asl-1.8.8.jar to /Users/srccodes/hadoop/build/contrib/eclipse-plugin/lib/jackson-core-asl-1.8.8.jar
[copy] Copying 1 file to /Users/srccodes/hadoop/build/contrib/eclipse-plugin/lib
[copy] Copying /Users/srccodes/hadoop/lib/jackson-mapper-asl-1.8.8.jar to /Users/srccodes/hadoop/build/contrib/eclipse-plugin/lib/jackson-mapper-asl-1.8.8.jar
[jar] Building jar: /Users/srccodes/hadoop/build/contrib/eclipse-plugin/hadoop-eclipse-plugin-1.2.0.jar
BUILD SUCCESSFUL
Total time: 4 seconds
7. Install Hadoop Eclipse Plug-in
Copy Hadoop Eclipse Plug-in jar (hadoop-eclipse-plugin-1.2.0.jar) and paste it to
8. Check Hadoop Eclipse Plug-in
Navigate Window --> Open Perspective --> Other and select Map/ReduceOpen perspective. You'll be able to see Map/Reduce Locations window where New Hadoop Location can be added.
Now, Eclipse environment for Hadoop is ready for you to explore.
Comments