Kudu的介绍及使用

前文：

过往采用Hive的离线处理时效性低，计算任务过于集中，查询效率低。SparkStreaming+Hive的数据清洗线使得多套数据流过于复杂。未来的数据仓库场景越来越趋向于实时数仓。

一、引入

二、架构图

2、架构及数据量

3、文件结构

4、目录结构

5、读写流程及分区策略

三、分区及策略

四、代码

1、通过SparkSql操作数据

object kudu_02_Main {def main(args: Array[String]): Unit = {val spark = SparkSession.builder().master("local[*]").appName("kudu_02_Main").getOrCreate()val df = spark.read.options(Map("kudu.master" -> "kudu-01:7051","kudu.table" -> "impala::test.my_table_kudu")).format("org.apache.kudu.spark.kudu").load()//.kududf.show();/*** +------+---------------------+---------+* | name | time                | message |* +------+---------------------+---------+* | joey | 2018-01-01 00:00:00 | hello   |* | ross | 2018-02-01 00:00:00 | goodbye |* +------+---------------------+---------+*/df.select("name").where("name = 'joey'").show()println("######################################################################")//    //注册全局视窗
//    df.createGlobalTempView("my_table_kudu1")
//    spark.sql("select * from global_temp.my_table_kudu1").show()//注册临时视窗df.createOrReplaceTempView("my_table_kudu2")spark.sql("select * from my_table_kudu2 where name = 'joey'").show()/*** create table test.my_tb2_kudu(* name string,* age int,* city string,* primary key(name)* )* partition by hash(name) partitions 2* stored as kudu;*/val students: Array[Student] = Array(Student("jack", 30, "new york"),Student("jason", 20, "beijing"),Student("cesar", 18, "guangzhou"),Student("anderson", 25, "dongguang"))import spark.implicits._val stuRDD: RDD[Student] = spark.sparkContext.parallelize(students)val stuDF = stuRDD.toDF()stuDF.write.options(Map("kudu.master"-> "bt-03:7051", "kudu.table"-> "impala::test.my_tb2_kudu")).mode("append").format("org.apache.kudu.spark.kudu").save}
}

2、通过kuduContext操作数据

object kudu_03_Main {case class Student(name: String, age: Int, city: String)case class delStu(name:String)def main(args: Array[String]): Unit = {val spark = SparkSession.builder().master("local[*]").appName("kudu_03_Main").getOrCreate()val kuduContext = new KuduContext("kudu-01:7051", spark.sparkContext)println(kuduContext.tableExists("impala::test.my_table_kudu"))/*** 建表语句* create table test.my_tb3_kudu(* name string,* age int,* city string,* primary key(name)* )* partition by hash(name) partitions 2* stored as kudu;*///增加val students: Array[Student] = Array(Student("jack", 30, "new york"),Student("jason", 20, "beijing"),Student("cesar", 18, "guangzhou"),Student("anderson", 25, "dongguang"))//RDD与DF互相转换import spark.implicits._val studentsRDD: RDD[Student] = spark.sparkContext.parallelize(students)val df: DataFrame = studentsRDD.toDF()df.show()//kuduContext.insertRows(df,"impala::test.my_tb3_kudu")val new_stu: Array[Student] = Array(Student("jack", 31, "dongbei"),Student("jason", 21, "shanghai"))val new_setRDD: RDD[Student] = spark.sparkContext.parallelize(new_stu)val df4 = new_setRDD.toDF()kuduContext.upsertRows(df4,"impala::test.my_tb3_kudu")//删除val delStus: Seq[delStu] = Seq(delStu("jason"),delStu("cesar"))//RDD与DF互相转换import spark.implicits._val delStusRDD: RDD[delStu] = spark.sparkContext.parallelize(delStus)val df1: DataFrame = delStusRDD.toDF()df1.show()//kuduContext.deleteRows(df1,"impala::test.my_tb3_kudu")val df2 = spark.read.options(Map("kudu.master" -> "bt-03:7051","kudu.table" -> "impala::test.my_tb3_kudu")).format("org.apache.kudu.spark.kudu").load()df2.show()df2.createOrReplaceTempView("my_tb3_kudu")val df3: DataFrame = spark.sql("select name from my_tb3_kudu where name = 'anderson'")//kuduContext.deleteRows(df3,"impala::test.my_tb3_kudu")}}

五、Kudu和HBase的对比

六、配置

master.gflagfile

# Do not modify these two lines. If you wish to change these variables,
# modify them in /etc/default/kudu-master.
--fromenv=rpc_bind_addresses
--fromenv=log_dir--fs_wal_dir=/data/kudu/wal/master
--fs_data_dirs=/data/kudu/data/master
--master_addresses=kudu-01:7051,kudu-02:7051,kudu-03:7051
-rpc_encryption=disabled
-rpc_authentication=disabled
-trusted_subnets=0.0.0.0/0--rpc_num_acceptors_per_address=5
--master_ts_rpc_timeout_ms=60000
--rpc_service_queue_length=1000

tserver.gflagfile

# Do not modify these two lines. If you wish to change these variables,
# modify them in /etc/default/kudu-tserver.
--fromenv=rpc_bind_addresses
--fromenv=log_dir--fs_wal_dir=/data/kudu/wal/server
--fs_data_dirs=/data/kudu/data/server
--tserver_master_addrs=kudu-01:7051,kudu-02:7051,kudu-03:7051
-rpc_encryption=disabled
-rpc_authentication=disabled
-trusted_subnets=0.0.0.0/0#性能调优
--memory_limit_hard_bytes=27917287424
--block_cache_capacity_mb=7680
--maintenance_manager_num_threads=3--rpc_num_acceptors_per_address=5
--master_ts_rpc_timeout_ms=60000
--rpc_service_queue_length=1000

Kudu的介绍及使用相关推荐

[Kudu基础]--Kudu+Impala介绍 | 微店数据科学团队博客
感谢原文作者:https://juejin.im/entry/5a72d3d1f265da3e4d730b37 Kudu+Impala介绍概述 Kudu和Impala均是Cloudera贡献给Apa ...
Apache Kudu的介绍
一.Apache Kudu的介绍 1.1.背景介绍在kudu之前,大数据主要以两种方式存储: (1)静态数据 : 以HDFS引擎作为存储,适用于高吞吐量的离线大数据分析场景. 这类存储的局限性是 ...
Kudu+Impala介绍
概述 Kudu和Impala均是Cloudera贡献给Apache基金会的顶级项目.Kudu作为底层存储,在支持高并发低延迟kv查询的同时,还保持良好的Scan性能,该特性使得其理论上能够同时兼顾OL ...
kudu简介与操作方式
kudu简介与操作方式 0.932018.06.09 16:13:35字数 750阅读 30042 1.kudu整体介绍 Kudu是cloudera开源的运行在hadoop平台上的列式存储系统,拥有H ...
笔记：分布式大数据技术原理（二）构建在 Hadoop 框架之上的 Hive 与 Impala
""" 有了 MapReduce,Tez 和 Spark 之后,程序员发现,MapReduce 的程序写起来真麻烦.他们希望简化这个过程.这就好比你有了汇编语言,虽然你几 ...
kudu介绍：为什么要使用?
前言近两年,KUDU 在大数据平台的应用越来越广泛.在阿里.小米.网易等公司的大数据架构中,KUDU 都有着不可替代的地位.本文通过分析 KUDU 的设计, 试图解释为什么 KUDU 会被广泛应用于 ...
Kudo介绍 + Spark\Python\Scala开发Kudu应用程序
前半部分文章摘自:http://blog.csdn.net/a1043498776/article/details/72681890 Kudu的背景 Hadoop中有很多组件,为了实现复杂的功能通常都 ...
kudu接受kafka消息_Kafka 在车好多
在 2016 年中旬,车好多集团(瓜子二手车&毛豆新车)开始引入 Kafka 技术栈,起初的目标是作为大数据系统的数据输入,随着使用范围的推广,逐步成为集团数据链路的核心组件. Kafka 在 ...
Kudu command tools
为什么80%的码农都做不了架构师?>>> 介绍 kudu集群部署完毕后,如何管理kudu集群?官方提供了kudu command tool工具来管理kudu集群,通过kudu ...

Kudu的介绍及使用

Kudu的介绍及使用相关推荐

最新文章

热门文章