【甘道夫】Ubuntu14 server + Hadoop2.2.0环境下Sqoop1.99.3部署记录-白红宇

【甘道夫】Ubuntu14 server + Hadoop2.2.0环境下Sqoop1.99.3部署记录

阅读量：4283 次

发布时间：2019-05-27

本文共 3368 字，大约阅读时间需要 11 分钟。

第一步，下载、解压、配置环境变量：

官网下载sqoop1.99.3

http://mirrors.cnnic.cn/apache/sqoop/1.99.3/

将sqoop解压到目标目录，我的是

/home/fulong/Sqoop/sqoop-1.99.3-bin-hadoop200

配置环境变量：

export SQOOP_HOME=/home/fulong/Sqoop/sqoop-1.99.3-bin-hadoop200

export PATH=$PATH:$SQOOP_HOME/bin

export CATALINA_BASE=$SQOOP_HOME/server

export LOGDIR=$SQOOP_HOME/logs/

第二步，修改配置文件：

修改配置文件catalina.properties：

vim $SQOOP_HOME/server/conf/catalina.properties

在文件中找到

common.loader，在其后添加hadoop相关jar包路径，最终效果如下

common.loader=${catalina.base}/lib,${catalina.base}/lib/*.jar,${catalina.home}/lib,${catalina.home}/lib/*.jar,${catalina.home}/../lib/*.jar,/home/fulong/Hadoop/hadoop-2.2.0/share/hadoop/common/*.jar,/home/fulong/Hadoop/hadoop-2.2.0/share/hadoop/common/lib/*.jar,/home/fulong/Hadoop/hadoop-2.2.0/share/hadoop/hdfs/*.jar,/home/fulong/Hadoop/hadoop-2.2.0/share/hadoop/hdfs/lib/*.jar,/home/fulong/Hadoop/hadoop-2.2.0/share/hadoop/mapreduce/*.jar,/home/fulong/Hadoop/hadoop-2.2.0/share/hadoop/mapreduce/lib/*.jar,/home/fulong/Hadoop/hadoop-2.2.0/share/hadoop/tools/*.jar,/home/fulong/Hadoop/hadoop-2.2.0/share/hadoop/tools/lib/*.jar,/home/fulong/Hadoop/hadoop-2.2.0/share/hadoop/yarn/*.jar,/home/fulong/Hadoop/hadoop-2.2.0/share/hadoop/yarn/lib/*.jar

修改配置文件sqoop.properties：

vim $SQOOP_HOME/conf/sqoop.properties

在文件中找到

org.apache.sqoop.submission.engine.mapreduce.configuration.directory，用Hadoop的配置文件目录赋值，最终效果如下：

org.apache.sqoop.submission.engine.mapreduce.configuration.directory=/home/fulong/Hadoop/hadoop-2.2.0/etc/hadoop

！！！！！！切记！！！！！！

sqoop配置文件中不能用环境变量（比如$HADOOP_HOME），必须用全路径！！！

第三步，jar包相关工作：

1）首先，拷入需要用到的Oracle的jdbc包classes12.jar：

我拷贝到了两个目录下

/home/fulong/Sqoop/sqoop-1.99.3-bin-hadoop200/server/webapps/sqoop/WEB-INF/lib

/home/fulong/Sqoop/sqoop-1.99.3-bin-hadoop200/server/lib

2）解决log4j包冲突问题

由于在

catalina.properties配置文件的common.loader属性中配置引入了所有的hadoop相关jar包，而

hadoop的lib目录下是有log4j包的，只是版本和sqoop自带的log4j不一致。

我们需要将

/home/fulong/Sqoop/sqoop-1.99.3-bin-hadoop200/server/webapps/sqoop/WEB-INF/lib目录下的log4j-1.2.16.jar移除即可。

启动：

Sqoop分为server端和client端，server端只需要在一个hadoop节点上运行即可，client端可以在任意能运行hadoop client的节点上运行。

首先启动server端：

fulong@FBI003:~/Sqoop/sqoop-1.99.3-bin-hadoop200/server/webapps/sqoop/WEB-INF/lib$

sqoop.sh server start

Sqoop home directory: /home/fulong/Sqoop/sqoop-1.99.3-bin-hadoop200

Setting SQOOP_HTTP_PORT: 12000

Setting SQOOP_ADMIN_PORT: 12001

Using CATALINA_OPTS:

Adding to CATALINA_OPTS: -Dsqoop.http.port=12000 -Dsqoop.admin.port=12001

Using CATALINA_BASE: /home/fulong/Sqoop/sqoop-1.99.3-bin-hadoop200/server

Using CATALINA_HOME: /home/fulong/Sqoop/sqoop-1.99.3-bin-hadoop200/server

Using CATALINA_TMPDIR: /home/fulong/Sqoop/sqoop-1.99.3-bin-hadoop200/server/temp

Using JRE_HOME: /usr/lib/jvm/java

Using CLASSPATH: /home/fulong/Sqoop/sqoop-1.99.3-bin-hadoop200/server/bin/bootstrap.jar

然后启动client端，设置指向服务器端，查看版本：

fulong@FBI008:~$

sqoop.sh client

Sqoop home directory: /home/fulong/Sqoop/sqoop-1.99.3-bin-hadoop200

Sqoop Shell: Type 'help' or '\h' for help.

sqoop:000>

set server --host FBI003 --port 12000 --webapp sqoop

Server is set successfully

sqoop:000>

show version --all

client version:

Sqoop 1.99.3 revision 2404393160301df16a94716a3034e31b03e27b0b

Compiled by mengweid on Fri Oct 18 14:15:53 EDT 2013

server version:

Sqoop 1.99.3 revision 2404393160301df16a94716a3034e31b03e27b0b

Compiled by mengweid on Fri Oct 18 14:15:53 EDT 2013

Protocol version:

[1]

sqoop:000>

到此，Sqoop1.99.3正常启动！

转载地址：http://hbcgi.baihongyu.com/

你可能感兴趣的文章