sqoop搭建配置使用

1.下载文件，并解压
http://archive.apache.org/dist/sqoop/
下面是解压之后的：

2.文件配置
进入conf文件，将sqoop-env-template.sh命名为sqoop-env.sh，并修改sqoop-env.sh：

[root@hadoop001 software]# cat /opt/sqoop/conf/sqoop-env.sh
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements.  See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License.  You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.# included in all the hadoop scripts with source command
# should not be executable directly
# also should not be passed any arguments, since we need original $*# Set Hadoop-specific environment variables here.#Set path to where bin/hadoop is available
export HADOOP_COMMON_HOME=/opt/hadoop/hadoop-2.8.0/#Set path to where hadoop-*-core.jar is available
export HADOOP_MAPRED_HOME=/opt/hadoop/hadoop-2.8.0/#set the path to where bin/hbase is available
#export HBASE_HOME=#Set the path to where bin/hive is available
export HIVE_HOME=/opt/hive/apache-hive-2.3.3-bin#Set the path for where zookeper config dir is
export ZOOCFGDIR=/opt/zookeeper/

3.将mysql驱动包加入到sqoop下lib中

[root@hadoop001 software]# ll /opt/sqoop/lib/...
-rw-r--r-- 1 root root 2428320 3月  29 13:20 mysql-connector-java-8.0.25.jar...

4.配置环境变量：

[root@hadoop001 software]# cat ~/.bashrc
#Sqoop
export SQOOP_HOME=/opt/sqoop
export PATH=$PATH:$SQOOP_HOME/binsource ~/.bashrc

5.验证是否成功

 sqoop-version 或者 sqoop version

6.查看基本命令

sqoop help

7.对mysql简单操作
（1）查看mysql有哪些数据库

[root@hadoop001 software]# sqoop list-databases --connect jdbc:mysql://hadoop001:3306/  --username root --password 000000
Warning: /opt/sqoop/../hbase does not exist! HBase imports will fail.
Please set $HBASE_HOME to the root of your HBase installation.
Warning: /opt/sqoop/../hcatalog does not exist! HCatalog jobs will fail.
Please set $HCAT_HOME to the root of your HCatalog installation.
Warning: /opt/sqoop/../accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
22/04/08 16:58:06 INFO sqoop.Sqoop: Running Sqoop version: 1.4.7
22/04/08 16:58:06 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
22/04/08 16:58:06 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
Loading class `com.mysql.jdbc.Driver'. This is deprecated. The new driver class is `com.mysql.cj.jdbc.Driver'. The driver is automatically registered via the SPI and manual loading of the driver class is generally unnecessary.
hive
information_schema
mysql
performance_schema

（2）查看mysql有哪些表

[root@hadoop001 software]# sqoop list-tables --connect jdbc:mysql://hadoop001:3306/mysql  --username root --password 000000
Warning: /opt/sqoop/../hbase does not exist! HBase imports will fail.
Please set $HBASE_HOME to the root of your HBase installation.
Warning: /opt/sqoop/../hcatalog does not exist! HCatalog jobs will fail.
Please set $HCAT_HOME to the root of your HCatalog installation.
Warning: /opt/sqoop/../accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
22/04/08 17:01:04 INFO sqoop.Sqoop: Running Sqoop version: 1.4.7
22/04/08 17:01:04 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
22/04/08 17:01:04 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
Loading class `com.mysql.jdbc.Driver'. This is deprecated. The new driver class is `com.mysql.cj.jdbc.Driver'. The driver is automatically registered via the SPI and manual loading of the driver class is generally unnecessary.
columns_priv
db
event
func
general_log
help_category
help_keyword
help_relation
help_topic
host
ndb_binlog_index

8.sqoop数据导入
（1）将mysql数据导入到hdfs中（未指定hdfs地址，选择默认路径）

sqoop import --connect jdbc:mysql://hadoop001:3306/mysql --username root  --password 000000  --table user   -m 1

查看结果：

[root@hadoop001 software]# hadoop fs -ls /user/root/user
Found 2 items
-rw-r--r--   2 root supergroup          0 2022-04-08 17:26 /user/root/user/_SUCCESS
-rw-r--r--   2 root supergroup        631 2022-04-08 17:26 /user/root/user/part-m-00000
[root@hadoop001 software]# hadoop fs -cat /user/root/user/part-m-00000
localhost,root,*032197AE5731D4664921A6CCAC7CFCE6A0698693,Y,Y,Y,Y,Y,Y,Y,Y,Y,Y,Y,Y,Y,Y,Y,Y,Y,Y,Y,Y,Y,Y,Y,Y,Y,Y,Y,Y,Y,,,,,0,0,0,0,,
hadoop001,root,*032197AE5731D4664921A6CCAC7CFCE6A0698693,Y,Y,Y,Y,Y,Y,Y,Y,Y,Y,Y,Y,Y,Y,Y,Y,Y,Y,Y,Y,Y,Y,Y,Y,Y,Y,Y,Y,Y,,,,,0,0,0,0,,
127.0.0.1,root,*032197AE5731D4664921A6CCAC7CFCE6A0698693,Y,Y,Y,Y,Y,Y,Y,Y,Y,Y,Y,Y,Y,Y,Y,Y,Y,Y,Y,Y,Y,Y,Y,Y,Y,Y,Y,Y,Y,,,,,0,0,0,0,,
::1,root,*032197AE5731D4664921A6CCAC7CFCE6A0698693,Y,Y,Y,Y,Y,Y,Y,Y,Y,Y,Y,Y,Y,Y,Y,Y,Y,Y,Y,Y,Y,Y,Y,Y,Y,Y,Y,Y,Y,,,,,0,0,0,0,,
%,root,*032197AE5731D4664921A6CCAC7CFCE6A0698693,Y,Y,Y,Y,Y,Y,Y,Y,Y,Y,N,Y,Y,Y,Y,Y,Y,Y,Y,Y,Y,Y,Y,Y,Y,Y,Y,Y,Y,,,,,0,0,0,0,,

（2）将mysql数据导入到hdfs中（指定hdfs地址和分隔符）

sqoop import  --connect jdbc:mysql://hadoop001:3306/mysql  --username root --password root  --table help_keyword  --target-dir /user/root/user/user_table  --fields-terminated-by '&' -m 2

查看结果：

root@hadoop001 software]# hdfs dfs -cat /user/root/user/user_table/part-m-00000
0&JOIN
1&HOST
2&REPEAT
3&SERIALIZABLE
4&REPLACE
5&AT
6&SCHEDULE
7&RETURNS
8&STARTS
9&MASTER_SSL_CA
10&NCHAR
11&COLUMNS
12&COMPLETION
13&WORK
14&DATETIME

sqoop搭建配置使用相关推荐

基于认证的代理平台搭建配置squid-20130730
基于认证的代理平台搭建配置squid-20130730 功能:通过squid代理实现 (1)基于用户名密码认证的出口ip路由选择 (2)基于client源ip的出口ip路由选择 (3)基于连接本机ip ...
Linux下svn搭建配置流程
Linux下svn搭建配置流程一. 源文件编译安装.源文件共两个,为: 1. 下载subversion源文件 subversion-1.6.1.tar.gz http://d136 ...
Linux下的NFS搭建配置
Linux下的NFS搭建配置一.NFS介绍 1.NFS简介 2.NFS特点二.环境介绍三.服务端配置 1.创建共享目录 2.检查NFS包是否安装 3.安装nfs相关包 4.编辑配置文件 5.重启 ...
linux下本地的源的格式,[Linux]CentOS7搭建/配置：YUM仓库/源[本地源/HTTP源/自建源仓库...
[Linux]CentOS7搭建/配置:YUM仓库/源[本地源/HTTP源/自建源仓库 1 本地YUM源 1.1 [YUM仓库/服务端] 获取:软件包资源 1.2 [YUM仓库/服务端] 挂载/搭建: ...
Linux下svn搭建配置
Linux下svn搭建配置 1.安装svn客户端 yum install subversion svnserve --version :如果成功 ...
DreamweaverCS6搭建配置php本地站点（图文教程）- 教程篇
通过Dreamweaver.exe 配置PHP本地服务站点一.软件安装说明: Dreamweaver_CS6.exe 二.步骤图解: 第一步: 第二步: 第三步: 第四步: 三.其他版本: Drea ...
PyQt5 环境搭建+配置+怎样运行生成的.py程序
PyQt5 环境搭建+配置+怎样运行生成的.py程序 PyQt5安装及配置安装配置 Error while finding module specification for 'PyQt5.uic. ...
Linux下svn 安装搭建配置流程
Linux下svn搭建配置流程一. 源文件编译安装.源文件共两个,为: 1. 下载subversion源文件 subversion-1.6.1.tar.gz http://d136 ...
Redis高可用集群-哨兵模式（Redis-Sentinel）搭建配置教程【Windows环境】
================================================= 人工智能教程.零基础!通俗易懂!风趣幽默!大家可以看看是否对自己有帮助! 点击查看高清无码教程 == ...

sqoop搭建配置使用

sqoop搭建配置使用相关推荐

最新文章

热门文章