Hadoop学习笔记之HDFS

  • HDFS （Hadoop Distributed File System）
      • 优点
      • 缺点
    • HDFS操作
      • 命令操作HDFS
      • Web端操作HDFS
      • Java操作HDFS

HDFS （Hadoop Distributed File System）

分布式存储系统

支持海量数据的存储，成百上千的计算机组成存储集群，HDFS可以运行在低成本的硬件之上，具有的高容错、高可靠性、高可扩展性、高吞吐率等特征，非常适合大规模数据集上的应用。

优点

  • 高容错性
  • 适合批处理
  • 适合大数据处理
  • 流式文件访问
  • 可构建在廉价机器上

缺点

  • 不适合低延迟数据访问
  • 不适合小文件存取
  • 不适合并发写入、文件随机修改

HDFS操作

命令操作HDFS

# 显示目录 / 下的文件
hdfs dfs -ls /
# 新建文件夹，绝对路径
hdfs dfs -mkdir /test
# 上传文件
hdfs dfs -put test.txt /test/
# 下载文件
hdfs dfs -get /test/test.txt
# 输出文件内容
hdfs dfs -cat /test/test.txt

Web端操作HDFS

打开http://192.168.9.200:9870/，可以对HDFS进行操作

文件在这里管理

可视化操作还是简单一些

Java操作HDFS

  1. 添加依赖
<dependency><groupId>org.apache.hadoop</groupId><artifactId>hadoop-client</artifactId><version>3.3.2</version>
</dependency>
  1. 建立连接
public void setUp() throws Exception {System.out.println("开始建立与HDFS的连接");configuration = new Configuration();fileSystem = FileSystem.get(new URI(HDFS_PATH), configuration, "hadoop");
}
  1. 文件操作
    Configuration configuration = null;FileSystem fileSystem = null;public static final String HDFS_PATH = "hdfs://192.168.9.200:9000";/*** 在 hdfs中新建文件夹** @throws Exception*/@Testpublic void mkdir() throws Exception {fileSystem.mkdirs(new Path("/JavaDemo/test"));}/*** 创建文件** @throws Exception*/@Testpublic void create() throws Exception {FSDataOutputStream outputStream = fileSystem.create(new Path("/JavaDemo/test/haha.txt"));outputStream.write("hello bigdata from javaDemo".getBytes());outputStream.flush();outputStream.close();}/*** 查看文件 hdfs -fs -cat file** @throws Exception*/@Testpublic void cat() throws Exception {FSDataInputStream in = fileSystem.open(new Path("/JavaDemo/test/haha.txt"));IOUtils.copyBytes(in, System.out, 1024);in.close();}/*** 重命名文件** @throws Exception*/@Testpublic void rename() throws Exception {Path oldPath = new Path("/JavaDemo/test/haha.txt");Path newPath = new Path("/JavaDemo/test/hehe.txt");fileSystem.rename(oldPath, newPath);}/*** 上传文件到HDFS** @throws Exception*/@Testpublic void copyFromLocalFile() throws Exception {Path loacalPath = new Path("hello.txt");Path hdfsPath = new Path("/");fileSystem.copyFromLocalFile(loacalPath, hdfsPath);}/*** 上传文件到HDFS带进度信息** @throws Exception*/@Testpublic void copyFromLocalFileWithProgress() throws Exception {InputStream in = new BufferedInputStream(Files.newInputStream(new File("hbase-2.2.7-bin.tar.gz").toPath()));FSDataOutputStream ouput = fileSystem.create(new Path("/JavaDemo/test/hbase-2.2.7-bin.tar.gz"), () -> {System.out.print(".");});IOUtils.copyBytes(in, ouput, 4096);}/*** 下载文件到HDFS** @throws Exception*/@Testpublic void copyToLocalFile() throws Exception {Path hdfsPath = new Path("/JavaDemo/test/haha.txt");Path loacalPath = new Path("./haha.txt");//  useRawLocalFileSystemfileSystem.copyToLocalFile(false, hdfsPath, loacalPath, true);}/*** 查看某个目录下所有文件** @throws Exception*/@Testpublic void listFiles() throws Exception {FileStatus[] fileStatuses = fileSystem.listStatus(new Path("/"));for (FileStatus f : fileStatuses) {String isDir = f.isDirectory() ? "文件夹" : "文件";short replication = f.getReplication();long len = f.getLen();String path = f.getPath().toString();System.out.println(isDir + "\t" + replication + "\t" + len + "\t" + path);}}/*** 删除文件** @throws Exception*/@Testpublic void delete() throws IOException {fileSystem.delete(new Path("/JavaDemo/haha.txt"), true);}
  1. 关闭连接
public void tearDown() {configuration = null;fileSystem = null;System.out.println("关闭与HDFS的连接");
}

完整测试文件

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.*;
import org.apache.hadoop.io.IOUtils;
import org.junit.After;
import org.junit.Before;
import org.junit.Test;import java.io.*;
import java.net.URI;
import java.nio.file.Files;/*** @author Gettler* Java 操作HDFS*/
public class HdfsDemo {Configuration configuration = null;FileSystem fileSystem = null;public static final String HDFS_PATH = "hdfs://192.168.9.200:9000";/*** 在 hdfs中新建文件夹** @throws Exception*/@Testpublic void mkdir() throws Exception {fileSystem.mkdirs(new Path("/JavaDemo/test"));}/*** 创建文件** @throws Exception*/@Testpublic void create() throws Exception {FSDataOutputStream outputStream = fileSystem.create(new Path("/JavaDemo/test/haha.txt"));outputStream.write("hello bigdata from javaDemo".getBytes());outputStream.flush();outputStream.close();}/*** 查看文件 hdfs -fs -cat file** @throws Exception*/@Testpublic void cat() throws Exception {FSDataInputStream in = fileSystem.open(new Path("/JavaDemo/test/haha.txt"));IOUtils.copyBytes(in, System.out, 1024);in.close();}/*** 重命名文件** @throws Exception*/@Testpublic void rename() throws Exception {Path oldPath = new Path("/JavaDemo/test/haha.txt");Path newPath = new Path("/JavaDemo/test/hehe.txt");fileSystem.rename(oldPath, newPath);}/*** 上传文件到HDFS** @throws Exception*/@Testpublic void copyFromLocalFile() throws Exception {Path loacalPath = new Path("hello.txt");Path hdfsPath = new Path("/");fileSystem.copyFromLocalFile(loacalPath, hdfsPath);}/*** 上传文件到HDFS带进度信息** @throws Exception*/@Testpublic void copyFromLocalFileWithProgress() throws Exception {InputStream in = new BufferedInputStream(Files.newInputStream(new File("hbase-2.2.7-bin.tar.gz").toPath()));FSDataOutputStream ouput = fileSystem.create(new Path("/JavaDemo/test/hbase-2.2.7-bin.tar.gz"), () -> {System.out.print(".");});IOUtils.copyBytes(in, ouput, 4096);}/*** 下载文件到HDFS** @throws Exception*/@Testpublic void copyToLocalFile() throws Exception {Path hdfsPath = new Path("/JavaDemo/test/haha.txt");Path loacalPath = new Path("./haha.txt");//  useRawLocalFileSystemfileSystem.copyToLocalFile(false, hdfsPath, loacalPath, true);}/*** 查看某个目录下所有文件** @throws Exception*/@Testpublic void listFiles() throws Exception {FileStatus[] fileStatuses = fileSystem.listStatus(new Path("/"));for (FileStatus f : fileStatuses) {String isDir = f.isDirectory() ? "文件夹" : "文件";short replication = f.getReplication();long len = f.getLen();String path = f.getPath().toString();System.out.println(isDir + "\t" + replication + "\t" + len + "\t" + path);}}/*** 删除文件** @throws Exception*/@Testpublic void delete() throws IOException {fileSystem.delete(new Path("/JavaDemo/haha.txt"), true);}//测试之前执行的代码@Beforepublic void setUp() throws Exception {System.out.println("开始建立与HDFS的连接");configuration = new Configuration();fileSystem = FileSystem.get(new URI(HDFS_PATH), configuration, "hadoop");}//测试之完执行的代码@Afterpublic void tearDown() {configuration = null;fileSystem = null;System.out.println("关闭与HDFS的连接");}
}

9999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999ssssssssss999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999ssssssssss999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999ssssssssss999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999ssssssssss999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999ssssssssss99999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999

Hadoop学习笔记之HDFS相关推荐

  1. Hadoop学习笔记（四）HDFS部分下

    Hadoop学习笔记(四)HDFS部分下 一.HDFS 的数据流 1.1 HDFS的写数据流程 客户端通过 Distributed FileSystem 模块向 NameNode 请求上传文件,Nam ...

  2. hadoop学习笔记（八）：Java HDFS API

    hadoop学习笔记(八):Java HDFS API 阅读目录 一.使用HDFS FileSystem详解 回到顶部 一.使用HDFS FileSystem详解 HDFS依赖的第三方包: hadoo ...

  3. Hadoop学习笔记一 简要介绍

    Hadoop学习笔记一 简要介绍 这里先大致介绍一下Hadoop.     本文大部分内容都是从官网Hadoop上来的.其中有一篇介绍HDFS的pdf文档,里面对Hadoop介绍的比较全面了.我的这一 ...

  4. Hadoop学习笔记(1) ——菜鸟入门

     Hadoop学习笔记(1) --菜鸟入门 Hadoop是什么?先问一下百度吧: [百度百科]一个分布式系统基础架构,由Apache基金会所开发.用户能够在不了解分布式底层细节的情况下.开发分布式 ...

  5. Hadoop学习笔记(1)

    原文:http://www.cnblogs.com/zjfstudio/p/3859704.html Hadoop学习笔记(1) --菜鸟入门 Hadoop是什么?先问一下百度吧: [百度百科]一个分 ...

  6. Hadoop学习笔记—18.Sqoop框架学习

    Hadoop学习笔记-18.Sqoop框架学习 一.Sqoop基础:连接关系型数据库与Hadoop的桥梁 1.1 Sqoop的基本概念 Hadoop正成为企业用于大数据分析的最热门选择,但想将你的数据 ...

  7. Hadoop学习笔记—20.网站日志分析项目案例（一）项目介绍

    Hadoop学习笔记-20.网站日志分析项目案例(一)项目介绍 网站日志分析项目案例(一)项目介绍:当前页面 网站日志分析项目案例(二)数据清洗:http://www.cnblogs.com/edis ...

  8. Hadoop学习笔记(8) ——实战 做个倒排索引

    Hadoop学习笔记(8) --实战 做个倒排索引 倒排索引是文档检索系统中最常用数据结构.根据单词反过来查在文档中出现的频率,而不是根据文档来,所以称倒排索引(Inverted Index).结构如 ...

  9. Hadoop学习笔记—4.初识MapReduce

    一.神马是高大上的MapReduce MapReduce是Google的一项重要技术,它首先是一个 编程模型 ,用以进行大数据量的计算.对于大 数据量的计算,通常采用的处理手法就是并行计算.但对许多开 ...

最新文章

  1. oracle乘法运算,oracle实现相乘话语
  2. 区块链课程笔记-第一课哈希算法在加密中的应用
  3. 004：STM32启动文件详解及SystemInit函数分析（转）
  4. 解决Eclipse查看源代码出现Source not found的问题
  5. 使用IntelliJ IDEA 2016创建maven管理的Java Web项目
  6. linux安装R3.5.0
  7. 小米路由r2d论坛_维修一台亮黄灯的R2D小米路由器2
  8. Unity3D 制作按钮开关，旋钮开关，摇杆手柄 插件
  9. 计算机二级职业生涯规划讲座心得,职业规划讲座心得体会-
  10. 101个著名的管理学及心理学效应
  11. c语言实现动画的方法,C语言动画技术的实现方法
  12. OPCClient远程连接OPC服务器配置手册
  13. CondaUpgradeError: This environment has previously been operated on by a conda version that‘s newer
  14. 图片不能置于底层怎么办_ps怎么把图片置于底层
  15. 高通骁龙875夺安卓处理器桂冠，但外挂5G基带成为它的弊病
  16. 此CA证书不受信任的解决办法
  17. vue3 baseCss 初始化
  18. linux qt fscanf,fscanf QT小部件C++
  19. Idea Intellij 远程开发调试
  20. idea启动报符号缺失，无法启动与构建

热门文章

  1. 适合上班族做的副业，四个下班后可以操作的副业项目
  2. python均值插补法填补缺失值_R语言笔记（四）：特殊值处理
  3. 为什么python性能差
  4. 私域电商个人号IP运营指南 私域流量池之微信个人号基础操作手册
  5. 河南财经政法大学CSDN俱乐部招新圆满结束
  6. linux下使用vlc命令行推流
  7. Java并发编程实战--FutureTask
  8. 团队环境搭建：MTK平台，Android 9.0内销TV编译服务器配置
  9. 通达信指标公式编写基础（二）
  10. 计算机图形学基础第二版,计算机图形学基础(第2版)课后习题答案__陆枫.pdf