背景

部门老大说熟悉一下GPkafka的使用，昨天把ZK和kafka刚装完，今天就要实验一下kafka与GP的数据交互，从网上参考了一下教程，现在把他们整理一下，准备走一遍流程。

Kafka是分布式消息订阅系统，有非常好的横向扩展性，可实时存储海量数据，是流数据处理中间件的事实标准。当通过Kafka和GP搭建流处理管道时，如何高速可靠的完成流数据加载，这是个问题。从5.10开始，GP发布了新的工具GPKafka，为GP提供了流数据加载的能力。
GPkafka工具：kafka —> Greenplum

环境

kafka：kafka_2.11-2.4.1
GP: 5.19

正式开始今天的工作

1、启动kafka
参照前面的博客：先启动ZooKeeper，然后启动kafka
https://blog.csdn.net/weixin_43120559/article/details/105539016
https://blog.csdn.net/weixin_43120559/article/details/105531275
2、创建gpss扩展
在将Kafka消息数据加载到Greenplum数据库之前，必须在将Kafka数据写入Greenplum表的每个数据库中注册Greenplum-Kafka集成格式化程序函数；示例在edw数据库

[gpadmin@mdw ~]$ psql -d edw
psql (8.3.23)
Type "help" for help.edw=# CREATE EXTENSION gpss;
CREATE EXTENSION

3、创建示例库
kafka的数据格式json形式；样式：

{"time":1550198435941,"type":"type_mobileinfo","phone_imei":"861738033581011","phone_imsi":"","phone_mac":"00:27:1c:95:47:09","appkey":"307A5C626E6C2F6472636E6E6A2F736460656473","phone_udid":"8F137BFFB2289784A5EA2DCADCE519C2","phone_udid2":"744DD04CE29652F4F1D2DFFC8D3204A9","appUdid":"D21C76419E54B18DDBB94BF2E6990183","phone_resolution":"1280*720","phone_apn":"","phone_model":"BF T26","phone_firmware_version":"5.1","phone_softversion":"3.19.0","phone_softname":"com.esbook.reader","sdk_version":"3.1.8","cpid":"blp1375_13621_001","currentnetworktype":"wifi","phone_city":"","os":"android","install_path":"\/data\/app\/com.esbook.reader-1\/base.apk","last_cpid":"","package_name":"com.esbook.reader","src_code":"WIFIMAC:00:27:1c:95:47:09"}

ods层的建表语句：

CREATE TABLE tbl_novel_mobile_log (package_name text,appkey text,ts bigint,phone_udid text,os character varying(20),idfa character varying(64),phone_imei character varying(20),cpid text,last_cpid text,phone_number character varying(20)
) ;

4、创建gpkafka.yaml配置文件

DATABASE: edw
USER: gpadmin
HOST: 192.168.0.66
PORT: 5432
KAFKA:INPUT:SOURCE:BROKERS: 192.168.0.66:9092TOPIC: mobile_infoCOLUMNS:- NAME: jdataTYPE: jsonFORMAT: jsonERROR_LIMIT: 10OUTPUT:TABLE: tbl_novel_mobile_logMAPPING:- NAME: package_nameEXPRESSION: (jdata->>'package_name')::text- NAME: appkeyEXPRESSION: (jdata->>'appkey')::text- NAME: tsEXPRESSION: (jdata->>'time')::bigint- NAME: phone_udidEXPRESSION: (jdata->>'phone_udid')::text- NAME: osEXPRESSION: (jdata->>'os')::text- NAME: idfaEXPRESSION: (jdata->>'idfa')::text- NAME: phone_imeiEXPRESSION: (jdata->>'phone_imei')::text- NAME: cpidEXPRESSION: (jdata->>'cpid')::text- NAME: last_cpidEXPRESSION: (jdata->>'last_cpid')::text- NAME: phone_numberEXPRESSION: (jdata->>'phone_number')::textCOMMIT:MAX_ROW: 1000

5、创建mobile_info topic

/opt/apps/kafka/bin/kafka-topics.sh --create --zookeeper 192.168.0.66:2181 --replication-factor 1 --partitions 1  --topic mobile_info

6、创建kafka的发布者
执行下列命令；并添加kafka记录
，下面是五条数据一条条执行

/opt/apps/kafka/bin/kafka-console-producer.sh  --broker-list 192.168.0.76:9092 --topic mobile_info
{"time":1550198435941,"type":"type_mobileinfo","phone_imei":"861738033581011","phone_imsi":"","phone_mac":"00:27:1c:95:47:09","appkey":"307A5C626E6C2F6472636E6E6A2F736460656473","phone_udid":"8F137BFFB2289784A5EA2DCADCE519C2","phone_udid2":"744DD04CE29652F4F1D2DFFC8D3204A9","appUdid":"D21C76419E54B18DDBB94BF2E6990183","phone_resolution":"1280*720","phone_apn":"","phone_model":"BF T26","phone_firmware_version":"5.1","phone_softversion":"3.19.0","phone_softname":"com.esbook.reader","sdk_version":"3.1.8","cpid":"blp1375_13621_001","currentnetworktype":"wifi","phone_city":"","os":"android","install_path":"\/data\/app\/com.esbook.reader-1\/base.apk","last_cpid":"","package_name":"com.esbook.reader","src_code":"WIFIMAC:00:27:1c:95:47:09"}
{"time":1550198437885,"type":"type_mobileinfo","phone_imei":"862245038046551","phone_imsi":"","phone_mac":"02:00:00:00:00:00","appkey":"307A5C626F2F76646B74606F2F736460656473","phone_udid":"A3BB70A0218AEFC7908B1D79C0C02D77","phone_udid2":"E3976E0453010FC7F32B6143AA3A164E","appUdid":"4FBEF77BC076254ED0407CAD653E6954","phone_resolution":"1920*1080","phone_apn":"","phone_model":"Le X620","phone_firmware_version":"6.0","phone_softversion":"1.9.0","phone_softname":"cn.wejuan.reader","sdk_version":"3.1.8","cpid":"blf1298_14411_001","currentnetworktype":"wifi","phone_city":"","os":"android","install_path":"\/data\/app\/cn.wejuan.reader-1\/base.apk","last_cpid":"","package_name":"cn.wejuan.reader","src_code":"ffffffff-9063-8e34-0000-00007efffeff"}
{"time":1550198438311,"type":"type_mobileinfo","phone_number":"","phone_imei":"867520045576831","phone_imsi":"460001122544742","phone_mac":"02:00:00:00:00:00","appkey":"307A5C626E6C2F6472636E6E6A2F736460656473","phone_udid":"A00407EF9D6EBCC207A514CDA452EB76","phone_udid2":"A00407EF9D6EBCC207A514CDA452EB76","appUdid":"1C35633F4EB8218789EFD8666C763485","phone_resolution":"2086*1080","phone_apn":"CMCC","phone_model":"ONEPLUS A6000","phone_firmware_version":"9","phone_softversion":"3.19.0","phone_softname":"com.esbook.reader","sdk_version":"3.1.8","cpid":"blf1298_12242_001","currentnetworktype":"4gnet","phone_city":"","os":"android","install_path":"\/data\/app\/com.esbook.reader-TlgFCk6ANgEDRnXDCem8uQ==\/base.apk","last_cpid":"","package_name":"com.esbook.reader","src_code":"460001122544742"}
{"time":1550198433102,"type":"type_mobileinfo","phone_number":"15077113477","phone_imei":"860364049874919","phone_imsi":"460023771256711","phone_mac":"02:00:00:00:00:00","appkey":"307A5C626E6C2F6472636E6E6A2F736460656473","phone_udid":"EEF566CB5253AA62B653347A203815C3","phone_udid2":"0845931539AE39B3B0D4EB42B85D98EC","appUdid":"9570DCA2D574E6B69B24137035209D42","phone_resolution":"2340*1080","phone_apn":"CHINA MOBILE","phone_model":"PBEM00","phone_firmware_version":"8.1.0","phone_softversion":"3.19.0","phone_softname":"com.esbook.reader","sdk_version":"3.1.8","cpid":"blf1298_12242_001","currentnetworktype":"4gnet","phone_city":"","os":"android","install_path":"\/data\/app\/com.esbook.reader-NBToXQo14TOeNuPxo_aA4w==\/base.apk","last_cpid":"","package_name":"com.esbook.reader","src_code":"13598c2d-efc4-4957-8d4d-22eb145d15fd"}
{"time":1550198440577,"type":"type_mobileinfo","phone_imei":"869800021106037","phone_imsi":"","phone_mac":"2c:5b:b8:fb:79:af","appkey":"307A5C626E6C2F6472636E6E6A2F736460656473","phone_udid":"2BC16C4AC07070BA9608BBD0EE2EE320","phone_udid2":"A7F9FA4772D31FADEECFDB445BA3BEBB","appUdid":"DC6BEE2F6E5D6A133E26131887AE788A","phone_resolution":"960*540","phone_apn":"","phone_model":"OPPO A33","phone_firmware_version":"5.1.1","phone_softversion":"3.19.0","phone_softname":"com.esbook.reader","sdk_version":"3.1.8","cpid":"blp1375_14526_003","currentnetworktype":"wifi","phone_city":"","os":"android","install_path":"\/data\/app\/com.esbook.reader-1\/base.apk","last_cpid":"","package_name":"com.esbook.reader","src_code":"WIFIMAC:2c:5b:b8:fb:79:af"}
{"time":1506944701166,"type":"type_mobileinfo","phone_number":"+8618602699126","phone_imei":"865902038154143","phone_imsi":"460012690618403","phone_mac":"02:00:00:00:00:00","appkey":"307A5C626E6C2F6472636E6E6A2F736460656473","phone_udid":"388015DA70C0AEA6D59D3CE37B0C4BA2","phone_udid2":"388015DA70C0AEA6D59D3CE37B0C4BA2","appUdid":"EC0A105297D55075526018078A4A1B84","phone_resolution":"1920*1080","phone_apn":"中国联通","phone_model":"MI MAX 2","phone_firmware_version":"7.1.1","phone_softversion":"3.19.0","phone_softname":"com.esbook.reader","sdk_version":"3.1.8","cpid":"blf1298_10928_001","currentnetworktype":"wifi","phone_city":"","os":"android","install_path":"\/data\/app\/com.esbook.reader-1\/base.apk","last_cpid":"","package_name":"com.esbook.reader","src_code":"460012690618403"}

验证topic：

./bin/kafka-console-consumer.sh --bootstrap-server 192.168.0.76:9092 --topic mobile_info --from-beginning

在没输入上面5条数据时，这个命令的是没有输出的，当输入了之后就会有相应的数据输出。
7.、执行 gpkafka 加载数据

 gpkafka load --quit-at-eof ./gpkafka_mobile_yaml

8、检查加载操作的进度 (非必要)

 gpkafka check ./gpkafka_mobile_yaml

9、查看表中数据。

select * from tbl_novel_mobile_log ;

GreenPlum--GPkafka使用教程相关推荐

centos7 数据库 Greenplum 的单机安装及异常解决方案
centos7 数据库Greenplum-1.13的安装缘环境下载依赖部署安装配置及数据库初始化配置目录配置服务器地址配置用户环境变量数据库配置初始化 **配置远程登录数据库* ...
Greenplum【环境搭建 04】使用GPKafka实现Kafka数据导入Greenplum数据库（扩展安装文件网盘分享）
分享资源地址及文件列表: 链接:https://pan.baidu.com/s/1XVTxKLkOYrL4pCZpFfs-Tg 提取码:sq90 包含文件: # 命令执行 gpkafka # 扩展安装 ...
Greenplum【部署 04】GPSS扩展安装并使用GPKafka实现Kafka数据导入Greenplum数据库（安装包网盘分享）
链接:https://pan.baidu.com/s/1MO-qL0Pxe6PojfZKsw3_qA 提取码:o7fl Greenplum Stream Server (GPSS)是一个ETL(提取. ...
Greenplum Hadoop分布式平台大数据解决方案实战教程
基于Greenplum Hadoop分布式平台的大数据解决方案及商业应用案例剖析 [上集]百度网盘下载:链接:http://pan.baidu.com/s/1eQJFXZ0 密码:kdx9 [下集]百 ...
navicate连接greenplum数据库教程
准备条件 1.已经安装了greenplum数据库.我是在自己电脑CentOS7上安装的单机版. 具体安装方法可以参考我的文章: (还没写完,还需要整理一下,需要的话可以先存着,最近就会补上) 2.确定 ...
【Greenplum 6.9.0】Greenplum Command Center 6.2安装失败教程
需要这个? gpperfmon_install --enable --password gpmon --port 5432 -bash: gpperfmon_install: command not ...
Greenplum 6.0安装教程
一.Greenplum简介 Greenplum数据库系统体系结构的高级概述. Greenplum数据库通过在多个服务器或主机之间分配负载来存储和处理大量数据. Greenplum中的逻辑数据库是一组单 ...
Greenplum常用问题汇总【持续更新】
Greenplum是世界领先的MPP架构数据库,底层基于PostgreSQL,能处理大规模分析型查询.当前Greenplum在Pivotal被VMWare收购后,已经归属于VMWare Tanzu产品 ...
hive外部表改为内部表_3000字揭秘Greenplum的外部数据加载——外部表
外部表是greenplum的一种数据表,它与普通表不同的地方是:外部表是用来访问存储在greenplum数据库之外的数据.如普通表一样,可使用SQL对外部表进行查询和插入操作.外部表主要用于Green ...

GreenPlum--GPkafka使用教程

背景

环境

正式开始今天的工作

GreenPlum--GPkafka使用教程相关推荐

最新文章

热门文章