Hello Tez

1) 编译protobuffer

$ tar zxf protobuf-2.5.0.tar.gz

$ cd protobuf-2.5.0

$ ./configure && make && sudo make install

$ protoc --version

libprotoc 2.5.0

2) 修改nodejs和npm的版本

由于本机已经安装了node.js和npm

hadoop@hadoop:~$ node --version

v0.10.33

hadoop@hadoop:~$ npm --version

1.4.28

为了和版本一致, 修改tez-ui的pom.xml. 试过注释掉那段代码, 但是会报错.

<webappDir>src/main/webapp</webappDir>

<node.executable>${basedir}/src/main/webapp/node/node</node.executable>

<fileName>${artifactId}-${parent.version}</fileName>

</properties>

3) 编译tez

hadoop@hadoop:~/github/apache/tez$ mvn clean package -DskipTests=true -Dmaven.javadoc.skip=true

Tez on YARN

http://blog.woopi.org/wordpress/?p=96

http://hadooptutorial.info/apache-tez-successor-mapreduce-framework/

http://www.voidcn.com/article/p-clutuinh-rt.html

$ cd /home/hadoop/github/apache/tez/tez-dist/target/tez-0.7.0-SNAPSHOT

$ mkdir conf

$ hadoop fs -mkdir /apps

$ hadoop fs -mkdir /apps/tez

$ hadoop fs -put /home/hadoop/github/apache/tez/tez-dist/target/tez-0.7.0-SNAPSHOT /apps/tez

$ vi conf/tez-site.xml

<?xml version="1.0"?>

<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<name>tez.version</name>

<value>tez-0.7.0-SNAPSHOT</value>

</property>

<value>${fs.default.name}/apps/tez/${tez.version},${fs.default.name}/apps/tez/${tez.version}/lib/</value>

</configuration>

$ vi ~/.bashrc

export TEZ_HOME=/home/hadoop/github/apache/tez/tez-dist/target/tez-0.7.0-SNAPSHOT

export HADOOP_CLAsspATH=$HADOOP_CLAsspATH:${TEZ_HOME}/conf:${TEZ_HOME}/*:${TEZ_}/lib/*

另一种方案是复制tez-site.xml到$HADOOP_HOME/etc/hadoop下, 并修改hadoop-env.sh文件. 这种侵入性大, 不建议.

vi ${HADOOP_INSTALL}/etc/hadoop/hadoop-env.sh

export HADOOP_CLAsspATH=$HADOOP_HOME:$HADOOP_HOME/etc/hadoop

for f in /home/hadoop/github/apache/tez/tez-dist/target/tez-0.7.0-SNAPSHOT/*.jar; do

if [ "$HADOOP_CLAsspATH" ]; then

export HADOOP_CLAsspATH=$HADOOP_CLAsspATH:$f

else

export HADOOP_CLAsspATH=$f

done

for f in /home/hadoop/github/apache/tez/tez-dist/target/tez-0.7.0-SNAPSHOT/lib/*.jar; do

done

可选步骤:

$ vi ${HADOOP_HOME}/etc/hadoop/mapred-site.xml

<name>mapreduce.framework.name</name>

</property>

Tez WorldCount

$ stop-yarn.sh

$ start-yarn.sh

$ cd ~/github/apache/tez/tez-dist/target/tez-0.7.0-SNAPSHOT

$ hadoop fs -put ~/data/helloworld.txt /input/tez

$ hadoop jar tez-examples-0.7.0-SNAPSHOT.jar orderedwordcount /input/tez/helloworld.txt /output/tez/helloworld

hadoop@hadoop:~/github/apache/tez/tez-dist/target/tez-0.7.0-SNAPSHOT$ hadoop fs -ls /output/tez/helloworld

-rw-r--r-- 3 hadoop supergroup 0 2015-01-20 19:33 /output/tez/helloworld/_SUCCESS

-rw-r--r-- 3 hadoop supergroup 130 2015-01-20 19:33 /output/tez/helloworld/part-v002-o000-00000

hadoop@hadoop:~/github/apache/tez/tez-dist/target/tez-0.7.0-SNAPSHOT$ hadoop fs -cat /output/tez/helloworld/part-v002-o000-00000

again 1

system. 1

Spark 1

system, 1

process 1

Now 1

hot 1

today 1

batch 1

also 1

.. 2

bigdata 2

a 2

Hadoop 2

Hello 3

world 3

is 3

可以看到结果安装word的count进行升序排列. 在 http://localhost:8088/cluster

$ hadoop jar tez-tests-0.7.0-SNAPSHOT.jar testorderedwordcount -DUSE_TEZ_SESSION=true \

/input/tez/helloworld.txt /output/tez/helloworld2 /input/tez/helloworld2.txt /output/tez/helloworld3

15/01/20 20:01:04 INFO examples.TestOrderedWordCount: Creating Tez Session

15/01/20 20:01:04 INFO client.TezClient: Tez Client Version: [ component=tez-api, version=0.7.0-SNAPSHOT, revision=83261659809f7904b786c9c81def4451dca27078, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20150120-1554 ]

15/01/20 20:01:04 INFO client.RMProxy: Connecting to ResourceManager at localhost/127.0.0.1:8032

15/01/20 20:01:04 INFO client.TezClient: Session mode. Starting session.

15/01/20 20:01:04 INFO Configuration.deprecation: fs.default.name is deprecated. Instead, use fs.defaultFS

15/01/20 20:01:04 INFO client.TezClientUtils: Using tez.lib.uris value from configuration:

hdfs://localhost:9000/apps/tez/tez-0.7.0-SNAPSHOT,hdfs://localhost:9000/apps/tez/tez-0.7.0-SNAPSHOT/lib/

15/01/20 20:01:04 INFO client.TezClient: Tez system stage directory

hdfs://localhost:9000/tmp/hadoop/tez/staging/1421755263939/.tez/application_1421753603786_0002 doesn't exist and is created

15/01/20 20:01:04 INFO impl.YarnClientImpl: Submitted application application_1421753603786_0002

15/01/20 20:01:04 INFO client.TezClient: The url to track the Tez Session: http://localhost:8088/proxy/application_1421753603786_0002/

15/01/20 20:01:04 INFO examples.TestOrderedWordCount: Running OrderedWordCount DAG,

dagIndex=1, inputPath=/input/tez/helloworld.txt, outputPath=/output/tez/helloworld2

15/01/20 20:01:05 INFO examples.TestOrderedWordCount: Checking DAG specific ACLS

15/01/20 20:01:05 INFO examples.TestOrderedWordCount: Waiting for TezSession to get into ready state

15/01/20 20:01:08 INFO examples.TestOrderedWordCount: Submitting DAG to Tez Session, dagIndex=1

15/01/20 20:01:08 INFO client.TezClient: Submitting dag to TezSession,

sessionName=OrderedWordCountSession, applicationId=application_1421753603786_0002, dagName=OrderedWordCount1

15/01/20 20:01:08 INFO client.RMProxy: Connecting to ResourceManager at localhost/127.0.0.1:8032

15/01/20 20:01:08 INFO examples.TestOrderedWordCount: Submitted DAG to Tez Session, dagIndex=1

15/01/20 20:01:15 INFO examples.TestOrderedWordCount: DAG 1 completed. FinalState=SUCCEEDED

examples.TestOrderedWordCount: Running OrderedWordCount DAG, dagIndex=2, inputPath=/input/tez/helloworld2.txt, outputPath=/output/tez/helloworld3

15/01/20 20:01:15 INFO examples.TestOrderedWordCount: Checking DAG specific ACLS

15/01/20 20:01:15 INFO examples.TestOrderedWordCount: Waiting for TezSession to get into ready state

15/01/20 20:01:15 INFO examples.TestOrderedWordCount:

15/01/20 20:01:15 INFO client.TezClient: Submitting dag to TezSession, dagName=OrderedWordCount2

15/01/20 20:01:15 INFO client.RMProxy: Connecting to ResourceManager at localhost/127.0.0.1:8032

15/01/20 20:01:15 INFO examples.TestOrderedWordCount: Submitted DAG to Tez Session, dagIndex=2

15/01/20 20:01:16 INFO examples.TestOrderedWordCount: DAG 2 completed. FinalState=SUCCEEDED

15/01/20 20:01:16 INFO examples.TestOrderedWordCount: Shutting down session

client.TezClient: Shutting down Tez Session, sessionName=OrderedWordCountSession, applicationId=application_1421753603786_0002

查看yarn web

Tez

Tez on YARN

Tez WorldCount

相关推荐