使用HIVE 自带的JSON 函数进行json解析同时解析JSON数组

数据展示
这是遗传JSON字符串，是一个还款计划，其中包含了很多的还款计划。

{"plan":[{"principal":"1114.09","interest":"489.14","date":"2018-11-02"},{"principal":"1124.30","interest":"423.03","date":"2018-12-02"},{"principal":"1134.61","interest":"412.72","date":"2019-01-02"},{"principal":"1145.01","interest":"402.32","date":"2019-02-02"},{"principal":"1155.50","interest":"391.83","date":"2019-03-02"},{"principal":"1166.10","interest":"381.23","date":"2019-04-02"},{"principal":"1176.78","interest":"370.55","date":"2019-05-02"},{"principal":"1187.57","interest":"359.76","date":"2019-06-02"},{"principal":"1198.46","interest":"348.87","date":"2019-07-02"},{"principal":"1209.44","interest":"337.89","date":"2019-08-02"},{"principal":"1220.53","interest":"326.80","date":"2019-09-02"},{"principal":"1231.72","interest":"315.61","date":"2019-10-02"},{"principal":"1243.01","interest":"304.32","date":"2019-11-02"},{"principal":"1254.40","interest":"292.93","date":"2019-12-02"},{"principal":"1265.90","interest":"281.43","date":"2020-01-02"},{"principal":"1277.51","interest":"269.82","date":"2020-02-02"},{"principal":"1289.22","interest":"258.11","date":"2020-03-02"},{"principal":"1301.03","interest":"246.30","date":"2020-04-02"},{"principal":"1312.96","interest":"234.37","date":"2020-05-02"},{"principal":"1325.00","interest":"222.33","date":"2020-06-02"},{"principal":"1337.14","interest":"210.19","date":"2020-07-02"},{"principal":"1349.40","interest":"197.93","date":"2020-08-02"},{"principal":"1361.77","interest":"185.56","date":"2020-09-02"},{"principal":"1374.25","interest":"173.08","date":"2020-10-02"},{"principal":"1386.85","interest":"160.48","date":"2020-11-02"},{"principal":"1399.56","interest":"147.77","date":"2020-12-02"},{"principal":"1412.39","interest":"134.94","date":"2021-01-02"},{"principal":"1425.34","interest":"121.99","date":"2021-02-02"},{"principal":"1438.40","interest":"108.93","date":"2021-03-02"},{"principal":"1451.59","interest":"95.74","date":"2021-04-02"},{"principal":"1464.90","interest":"82.43","date":"2021-05-02"},{"principal":"1478.32","interest":"69.01","date":"2021-06-02"},{"principal":"1491.87","interest":"55.46","date":"2021-07-02"},{"principal":"1505.55","interest":"41.78","date":"2021-08-02"},{"principal":"1519.35","interest":"27.98","date":"2021-09-02"},{"principal":"1533.28","interest":"14.05","date":"2021-10-02"}]}

目标表建表语句，现在的需求是要将一个JSON转化为多条数据，并且增加还款期数

CREATE TABLE `app.app_cpdji_repayment_plan`(`platform_no` string COMMENT '社会信用编码', `project_id` string COMMENT '项目编号', `contract_id` string COMMENT '合同编号', `repayment_periods` INT COMMENT '还款期数', `repayment_date` string COMMENT '还款日期', `principal` DECIMAL(20,2) COMMENT '应还本金', `interest` DECIMAL(20,2) COMMENT '应还利息')COMMENT '还款计划' STORED AS PARQUET;

获取JSON 中计划的内容

 select get_json_object(repay_plan, '$.plan');

将JSON串中的各行进行区分，分出多行数据

 split(regexp_replace( regexp_replace( regexp_replace(get_json_object(repay_plan, '$.plan'),'\\[','') , '\\]','') ,'\\}\\,\\{' ,'\\}\\;\\{') ,'\\;')) course_scores AS json) AS table1 LATERAL VIEW json_tuple(json,'date','principal','interest')

将数组进行列转行

 explode( split(regexp_replace( regexp_replace( regexp_replace(get_json_object(repay_plan, '$.plan'),'\\[','') , '\\]','') ,'\\}\\,\\{' ,'\\}\\;\\{') ,'\\;')) course_scores AS json) AS table1 LATERAL VIEW json_tuple(json,'date','principal','interest') d AS DATE1, principal,interest;

完成数据加工，进行按照日期排序

INSERT INTO app.app_cpdji_repayment_plan
SELECT platform_no,project_no,contract_no,ROW_NUMBER() over(PARTITION BY project_no,contract_no,platform_noORDER BY DATE1 ) rt,DATE1,principal,interest
FROM ( SELECT json,project_no,contract_no,platform_noFROM app.app_cpdji_view_ods_prodc_inv_contract LATERAL VIEW explode( split(regexp_replace( regexp_replace( regexp_replace(get_json_object(repay_plan, '$.plan'),'\\[','') , '\\]','') ,'\\}\\,\\{' ,'\\}\\;\\{') ,'\\;')) course_scores AS json) AS table1 LATERAL VIEW json_tuple(json,'date','principal','interest') d AS DATE1, principal,interest;

这样一个简单的数据解析就完成了，不得不说HIVE进行JSON解析是真的很不错。

花费了 1100多秒。

一共200亿的数据量，这个效率还是太棒了。

使用HIVE 自带的JSON 函数进行json解析同时解析JSON数组相关推荐

oracle 11g json 函数,在Oracle 11g中支持JSON(Support for JSON in Oracle 11g)
在Oracle 11g中支持JSON(Support for JSON in Oracle 11g) Oracle 11g是否支持JSON? 我的意思是像在PL / SQL查询中操纵JSON对象. 我 ...
大数据框架之Hive：第8章函数
第8章函数 8.1 函数简介 Hive会将常用的逻辑封装成函数给用户进行使用,类似于Java中的函数. 好处:避免用户反复写逻辑,可以直接拿来使用. 重点:用户需要知道函数叫什么,能做什么. Hiv ...
大数据基础之Hive（四）—— 常用函数和压缩存储
作者:duktig 博客:https://duktig.cn (文章首发) 优秀还努力.愿你付出甘之如饴,所得归于欢喜. 更多文章参看github知识库:https://github.com/dukt ...
Hive学习笔记三之函数操作
文章目录 5 函数 5.1 系统内置函数 5.2 常用内置函数 5.2.1 空字段赋值 5.2.2 CASE WHEN THEN ELSE END(类似于java中的switch case) 5.2. ...
sql与nosql_SQL Server JSON函数：NoSQL与关系世界之间的桥梁
sql与nosql In this article, we will explain what JSON is, what are the SQL Server JSON functions intr ...
mediarecorder添加时间戳_Python脚本实现数据处理（官方实例）和Hive自带时间函数...
官网示例官网地址: https://cwiki.apache.org/confluence/display/Hive/GettingStarted#GettingStarted-ApacheWebl ...
深入浅出学Hive：Hive内建操作符与函数开发
目录: 初始Hive Hive安装与配置 Hive内建操作符与函数开发 Hive JDBC Hive参数 Hive高级编程 Hive QL Hive Shell基本操作 Hive优化 Hive体系结构 ...
Hive系列（四）函数
Hive函数一.常用内置函数内置函数查看关系运算数学运算逻辑运算数值计算日期函数条件函数字符串函数集合统计函数二.lateral view 与 explode以及reflect和 ...
json函数 presto_Hive sql和Presto sql的一些对比
最近由于工作上和生活上的一些事儿好久没来博客园了,但是写博客的习惯还是得坚持,新的一年需要更加努力,困知勉行,终身学习,每天都保持空杯心态.废话不说,写一些最近使用到的Presto SQL和Hive ...
Hive常见查询操作与函数汇总
目录一.查询操作 1.基本查询(Like VS RLike) 2.Join语句 3.分组 4.排序 sort by 和 distribute by 6.分桶抽样二.函数汇总 1.查询函数行与列的 ...

使用HIVE 自带的JSON 函数进行json解析同时解析JSON数组

使用HIVE 自带的JSON 函数进行json解析同时解析JSON数组相关推荐

最新文章

热门文章

使用HIVE 自带的JSON 函数进行json解析 同时解析JSON数组

使用HIVE 自带的JSON 函数进行json解析 同时解析JSON数组相关推荐

最新文章

热门文章

使用HIVE 自带的JSON 函数进行json解析同时解析JSON数组

使用HIVE 自带的JSON 函数进行json解析同时解析JSON数组相关推荐