部署
# 下载
https://archive.apache.org/dist/sqoop/1.4.7/sqoop-1.4.7.bin__hadoop-2.6.0.tar.gz
# 解压
tar -zxvf sqoop-1.4.7.bin__hadoop-2.6.0.tar.gz -C /opt/
# 设置环境变量，并应用生效
echo 'export SQOOP_HOME=/opt/sqoop-1.4.7.bin__hadoop-2.6.0' >> /etc/profile
echo 'export PATH="$SQOOP_HOME/bin:$PATH"' >> /etc/profile
echo 'export CLASSPATH="$SQOOP_HOME/lib:$CLASSPATH"' >> /etc/profile
source /etc/profile
# 复制配置模板文件，并编辑
cp $SQOOP_HOME/conf/sqoop-env-template.sh $SQOOP_HOME/conf/sqoop-env.sh
vi $SQOOP_HOME/conf/sqoop-env.sh
#Set path to where bin/hadoop is available
export HADOOP_COMMON_HOME=/opt/hadoop-3.3.0
#Set path to where hadoop-*-core.jar is available
export HADOOP_MAPRED_HOME=/opt/hadoop-3.3.0
#set the path to where bin/hbase is available
export HBASE_HOME=/opt/hbase-2.5.6
#Set the path to where bin/hive is available
export HIVE_HOME=/opt/hive-3.1.3
#Set the path for where zookeper config dir is
export ZOOCFGDIR=/opt/zookeeper-3.7.2
# 测试sqoop是否能够正常打开，会输出版本
sqoop version
# 复制hive内的mysql连接jar包到sqoop内
cp /opt/hive-3.1.3/lib/mysql-connector-j-8.0.33.jar $SQOOP_HOME/lib/
#
sqoop list-databases --connect jdbc:mysql://192.168.10.75:3306/ --username root -password HoMf@123
报错
java.lang.NoClassDefFoundError: org/apache/commons/lang/StringUtils
- 原因：
Sqoop1.4.7默认只加载了commons-lang3-3.4.jar的jar包，里面的StringUtils类的package为：org/apache/commons/lang3/StringUtils，所以直接使用sqoop命令时报上述错误。
[root@linux-master ~]# sqoop list-databases --connect jdbc:mysql://192.168.10.75:3306/ --username root -password HoMf@123
Warning: /opt/sqoop-1.4.7.bin__hadoop-2.6.0/../hcatalog does not exist! HCatalog jobs will fail.
Please set $HCAT_HOME to the root of your HCatalog installation.
Warning: /opt/sqoop-1.4.7.bin__hadoop-2.6.0/../accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
Warning: /opt/sqoop-1.4.7.bin__hadoop-2.6.0/../zookeeper does not exist! Accumulo imports will fail.
Please set $ZOOKEEPER_HOME to the root of your Zookeeper installation.
2024-11-21 10:30:16,518 INFO sqoop.Sqoop: Running Sqoop version: 1.4.7
2024-11-21 10:30:16,630 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
2024-11-21 10:30:16,711 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/commons/lang/StringUtils
at org.apache.sqoop.manager.MySQLManager.initOptionDefaults(MySQLManager.java:73)
at org.apache.sqoop.manager.SqlManager.<init>(SqlManager.java:89)
at com.cloudera.sqoop.manager.SqlManager.<init>(SqlManager.java:33)
at org.apache.sqoop.manager.GenericJdbcManager.<init>(GenericJdbcManager.java:51)
at com.cloudera.sqoop.manager.GenericJdbcManager.<init>(GenericJdbcManager.java:30)
at org.apache.sqoop.manager.CatalogQueryManager.<init>(CatalogQueryManager.java:46)
at com.cloudera.sqoop.manager.CatalogQueryManager.<init>(CatalogQueryManager.java:31)
at org.apache.sqoop.manager.InformationSchemaManager.<init>(InformationSchemaManager.java:38)
at com.cloudera.sqoop.manager.InformationSchemaManager.<init>(InformationSchemaManager.java:31)
at org.apache.sqoop.manager.MySQLManager.<init>(MySQLManager.java:65)
at org.apache.sqoop.manager.DefaultManagerFactory.accept(DefaultManagerFactory.java:67)
at org.apache.sqoop.ConnFactory.getManager(ConnFactory.java:184)
at org.apache.sqoop.tool.BaseSqoopTool.init(BaseSqoopTool.java:272)
at org.apache.sqoop.tool.ListDatabasesTool.run(ListDatabasesTool.java:44)
at org.apache.sqoop.Sqoop.run(Sqoop.java:147)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:183)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:234)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:243)
at org.apache.sqoop.Sqoop.main(Sqoop.java:252)
Caused by: java.lang.ClassNotFoundException: org.apache.commons.lang.StringUtils
at java.net.URLClassLoader.findClass(URLClassLoader.java:387)
at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:359)
at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
... 20 more
- 解决:
将压缩包内的commons-lang-2.6.jar导入到sqoop目录下的lib目录下即可
下载地址：
http://mirrors.tuna.tsinghua.edu.cn/apache//commons/lang/binaries/commons-lang-2.6-bin.zip
Warning警告
Warning: /opt/sqoop-1.4.7.bin__hadoop-2.6.0/../hcatalog does not exist! HCatalog jobs will fail.
Please set $HCAT_HOME to the root of your HCatalog installation.
Warning: /opt/sqoop-1.4.7.bin__hadoop-2.6.0/../accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
Warning: /opt/sqoop-1.4.7.bin__hadoop-2.6.0/../zookeeper does not exist! Accumulo imports will fail.
Please set $ZOOKEEPER_HOME to the root of your Zookeeper installation.
- 解决：
进入$SQOOP_HOME/bin下，修改configure-sqoop文件，将下面的内容注释掉
vi $SQOOP_HOME/bin/configure-sqoop
## Moved to be a runtime check in sqoop.
##
if [ ! -d "${HCAT_HOME}" ]; then
echo "Warning: $HCAT_HOME does not exist! HCatalog jobs will fail."
echo 'Please set $HCAT_HOME to the root of your HCatalog installation.'
fi
if [ ! -d "${ACCUMULO_HOME}" ]; then
echo "Warning: $ACCUMULO_HOME does not exist! Accumulo imports will fail."
echo 'Please set $ACCUMULO_HOME to the root of your Accumulo installation.'
fi
if [ ! -d "${ZOOKEEPER_HOME}" ]; then
echo "Warning: $ZOOKEEPER_HOME does not exist! Accumulo imports will fail."
echo 'Please set $ZOOKEEPER_HOME to the root of your Zookeeper installation.'
fi
##
转换数据过程中问题
2024-11-21 14:23:20,771 ERROR hive.HiveConfig: Could not load org.apache.hadoop.hive.conf.HiveConf. Make sure HIVE_CONF_DIR is set correctly.
2024-11-21 14:23:20,772 ERROR tool.ImportTool: Import failed: java.io.IOException: java.lang.ClassNotFoundException: org.apache.hadoop.hive.conf.HiveConf
原因：
sqoop缺少hive jar包解决：
将hive/lib 包下的hive-common-x.x.x.jar和 hive-exec-x.x.x.jar 拷贝到sqoop/lib包下
x.x.x代表具体的版本，hive版本不同jar包版本也就会不同
[root@linux-master ~]# cp /opt/hive-3.1.3/lib/hive-common-3.1.3.jar /opt/sqoop-1.4.7.bin__hadoop-2.6.0/lib/
[root@linux-master ~]# cp /opt/hive-3.1.3/lib/hive-exec-3.1.3.jar /opt/sqoop-1.4.7.bin__hadoop-2.6.0/lib/