电商App项目的离线数仓

参考项目网址：（1）https://hub.fastgit.org/HammingMa/gmall-dw

（2）warehouse: 电商数据仓库

原始数据

从日志中抽取用户行为数据（json格式），包括启动数据和事件行为数据（这里模拟生成），存到指定文件。还有从数据库表里获得的相关数据，也存到指定文件。

（1）启动数据包括：

//设备id // 用户id // 程序版本号 5,6等 //程序版本名 v1.1.1 // 安卓系统版本 //设置日志类型 start appStart.setEn("start"); // 语言 es,en,pt // 渠道号从哪个渠道来的 appStart.setSr(getRandomChar(1)); // 区域 // 手机品牌 ba ,手机型号 md，就取2位数字了 // 嵌入sdk的版本 // gmail // 屏幕宽高 hw // 客户端产生日志时间 // 手机网络模式 3G,4G,WIFI // 拉丁美洲西经34°46′至西经117°09；北纬32°42′至南纬53°54′ // 经度 // 纬度 // 入口 // 开屏广告类型 // 状态 // 加载时长 // 失败码 // 扩展字段

数据例子

{"action":"1","ar":"MX","ba":"HTC","detail":"201","en":"start","entry":"1","extend1":"","g":"RWT7HN31@gmail.com","hw":"750*1134","l":"pt","la":"-9.0","ln":"-54.0","loading_time":"1","md":"HTC-17","mid":"31","nw":"WIFI","open_ad_type":"2","os":"8.2.6","sr":"C","sv":"V2.0.8","t":"1632675539796","uid":"31","vc":"11","vn":"1.3.6"}

（2）事件行为数据分为下面几种：

1.商品点击，展示 2.商品详情页 3. 商品列表页 4.广告 5.消息通知 6.用户前台活跃 7.用户后台活跃 8. 故障日志 9.用户评论 10. 用户收藏 11.用户点赞

数据例子

1632709453025|{"cm":{"ln":"-44.7","sv":"V2.8.4","os":"8.1.1","g":"TTF03B8I@gmail.com","mid":"32","nw":"3G","l":"en","vc":"17","hw":"750*1134","ar":"MX","uid":"32","t":"1632706083536","la":"13.4","md":"sumsung-5","vn":"1.0.3","ba":"Sumsung","sr":"M"},"ap":"app","et":[{"ett":"1632637800076","en":"loading","kv":{"extend2":"","loading_time":"0","action":"3","extend1":"","type":"2","type1":"201","loading_way":"2"}},{"ett":"1632628780999","en":"notification","kv":{"ap_time":"1632652229365","action":"2","type":"3","content":""}},{"ett":"1632652586798","en":"active_foreground","kv":{"access":"","push_id":"1"}},{"ett":"1632649878013","en":"favorites","kv":{"course_id":7,"id":0,"add_time":"1632616387020","userid":3}},{"ett":"1632631836363","en":"praise","kv":{"target_id":5,"id":8,"type":3,"add_time":"1632668604153","userid":4}}]}

（3）调用sqoop的脚本，db-》文件，从数据库表里获得相关数据，有全量和增量两种方法：

1.订单详情表表（增量） 2.SKU商品表（全量表） 3.用户表（增量及更新） 4.商品一级分类表（全量）5.商品二级分类表（全量）6.商品三级分类表（全量）7.支付流水表（增量）8.商品省份表（全量） 9.商品地区表（全量） 10.商品品牌表（全量） 11.订单状态表（增量）12.SPU商品表（全量）13.商品评论表（增量）14，退单表（增量）15.加购表（全量）16.商品收藏表（全量）17.优惠券领用表（新增及变化）18.优惠券表（全量）19.活动表（全量）20.活动订单关联表（增量）21.优惠规则表（全量）22.编码字典表（全量）

#! /bin/bash##这个是sqoop的路径，date的日期
sqoop=/opt/module/sqoop/bin/sqoop
do_date=`date -d '-1 day' +%F`if [ -n "$2" ]; thendo_date=$2
fi##这是其中的一个函数，通过该函数可以进行sqoop的传输
##注意第一 $1 不是传入的参数，而是传入该函数的sql语句，$2 是传入数据库表名
##最后的hadoop是传入的lzo压缩格式
import_data(){
##如果是CDH环境，请把上面的路径注释掉，然后改为   sqoop import \
#sqoop import \
$sqoop import \
--connect jdbc:mysql://hadoop103:3306/gmall \
--username root \
--password 123456 \
--target-dir /origin_data_cdh/gmall/db/$1/$do_date \
--delete-target-dir \
--query "$2 and  \$CONDITIONS" \
--num-mappers 1 \
--fields-terminated-by '\t' \
--compress \
--compression-codec lzop \
--null-string '\\N' \
--null-non-string '\\N'hadoop jar /opt/module/hadoop-2.7.2/share/hadoop/common/hadoop-lzo-0.4.20.jar com.hadoop.compression.lzo.DistributedLzoIndexer /origin_data_cdh/gmall/db/$1/$do_date
}
## 函数中也分增量和全量表，看到where 1=1 的都是全量表，带有日期的都是增量表import_order_info(){import_data order_info "selectid, final_total_amount, order_status, user_id, out_trade_no, create_time, operate_time,province_id,benefit_reduce_amount,original_total_amount,feight_fee      from order_infowhere (date_format(create_time,'%Y-%m-%d')='$do_date' or date_format(operate_time,'%Y-%m-%d')='$do_date')"
}##这里省略其他表
##进行参数调用
case $1 in"order_info")import_order_info
;;
##这里省略其他表
;;
"all")import_base_category1import_base_category2import_base_category3import_order_infoimport_order_detailimport_sku_infoimport_user_infoimport_payment_infoimport_base_trademarkimport_activity_infoimport_activity_orderimport_cart_infoimport_comment_infoimport_coupon_useimport_coupon_infoimport_favor_infoimport_order_refund_infoimport_order_status_logimport_spu_infoimport_activity_ruleimport_base_dic
;;
esac

ODS层

特点：

保持数据原貌不做任何修改，起到备份数据的作用。

数据采用压缩，减少磁盘存储空间（例如：原始数据100G，可以压缩到10G左右）。

创建分区表，防止后续的全表扫描。

在用户行为数据中，需要创建两张Hive表，一张是启动数据表，一张是事件行为表。

log（文件）->ods (外部表）相关代码：

CREATE DATABASE gmall;
use gmall;DROP TABLE IF EXISTS ods_start_log;
CREATE EXTERNAL TABLE ods_start_log (line string)
partitioned by (dt string)
STORED AS INPUTFORMAT  "com.hadoop.mapred.DeprecatedLzoTextInputFormat"
OUTPUTFORMAT "org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat"
LOCATION '/warehouse/gamll/ods/ods_start_log';LOAD DATA INPATH '/origin_data_cdh/gmall/log/topic_start/2020-11-19' INTO TABLE ods_start_log PARTITION (dt='2020-11-19');
LOAD DATA INPATH '/origin_data_cdh/gmall/log/topic_start/2020-11-20' INTO TABLE ods_start_log PARTITION (dt='2020-11-20');SELECT * FROM ods_start_log WHERE dt='2020-11-19' LIMIT 2;SELECT * FROM ods_start_log WHERE dt='2020-11-20' LIMIT 2;SELECT count(*) FROM ods_start_log WHERE dt='2020-11-20'#创建索引
sudo -u hdfs hadoop jar /opt/module/LZO/hadoop-lzo-0.4.15-cdh5.8.3.jar  com.hadoop.compression.lzo.LzoIndexer /warehouse/gamll/ods/ods_start_log/dt=2020-11-19#################################################################################
DROP TABLE IF EXISTS ods_event_log;
CREATE EXTERNAL TABLE ods_event_log(line STRING)
PARTITIONED BY (dt STRING)
STORED AS
INPUTFORMAT  "com.hadoop.mapred.DeprecatedLzoTextInputFormat"
OUTPUTFORMAT "org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat"
LOCATION '/warehouse/gamll/ods/ods_event_log';LOAD DATA INPATH '/origin_data_cdh/gmall/log/topic_event/2020-11-19' INTO TABLE ods_event_log PARTITION (dt='2020-11-19');SELECT * from ods_event_log WHERE dt='2020-11-19' LIMIT 2;LOAD DATA INPATH '/origin_data_cdh/gmall/log/topic_event/2020-11-20' INTO TABLE ods_event_log PARTITION (dt='2020-11-20');SELECT * from ods_event_log WHERE dt='2020-11-20' LIMIT 2;SELECT count(*) FROM ods_event_log WHERE dt='2020-11-19';sudo -u hdfs hadoop jar /opt/module/LZO/hadoop-lzo-0.4.15-cdh5.8.3.jar  com.hadoop.compression.lzo.LzoIndexer /warehouse/gamll/ods/ods_event_log/dt=2020-11-19

sqoop导出文件->ods

例如先建立订单详情表

drop table if exists ods_order_detail;
create external table ods_order_detail( `id` string COMMENT '订单编号',`order_id` string  COMMENT '订单号', `user_id` string COMMENT '用户id',`sku_id` string COMMENT '商品id',`sku_name` string COMMENT '商品名称',`order_price` decimal(10,2) COMMENT '商品价格',`sku_num` bigint COMMENT '商品数量',`create_time` string COMMENT '创建时间'
) COMMENT '订单详情表'
PARTITIONED BY (`dt` string)
row format delimited fields terminated by '\t'
STORED ASINPUTFORMAT 'com.hadoop.mapred.DeprecatedLzoTextInputFormat'OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat'
location '/warehouse/gmall/ods/ods_order_detail/';

然后存入ods

#!/bin/bashAPP=gmall
hive=/opt/module/hive/bin/hive# 如果是输入的日期按照取输入日期；如果没输入日期取当前时间的前一天
if [ -n "$2" ] ;thendo_date=$2
else do_date=`date -d "-1 day" +%F`
fisql1="
load data inpath '/origin_data/$APP/db/order_info/$do_date' OVERWRITE into table ${APP}.ods_order_info partition(dt='$do_date');load data inpath '/origin_data/$APP/db/order_detail/$do_date' OVERWRITE into table ${APP}.ods_order_detail partition(dt='$do_date');load data inpath '/origin_data/$APP/db/sku_info/$do_date' OVERWRITE into table ${APP}.ods_sku_info partition(dt='$do_date');load data inpath '/origin_data/$APP/db/user_info/$do_date' OVERWRITE into table ${APP}.ods_user_info partition(dt='$do_date');load data inpath '/origin_data/$APP/db/payment_info/$do_date' OVERWRITE into table ${APP}.ods_payment_info partition(dt='$do_date');load data inpath '/origin_data/$APP/db/base_category1/$do_date' OVERWRITE into table ${APP}.ods_base_category1 partition(dt='$do_date');load data inpath '/origin_data/$APP/db/base_category2/$do_date' OVERWRITE into table ${APP}.ods_base_category2 partition(dt='$do_date');load data inpath '/origin_data/$APP/db/base_category3/$do_date' OVERWRITE into table ${APP}.ods_base_category3 partition(dt='$do_date'); load data inpath '/origin_data/$APP/db/base_trademark/$do_date' OVERWRITE into table ${APP}.ods_base_trademark partition(dt='$do_date'); load data inpath '/origin_data/$APP/db/activity_info/$do_date' OVERWRITE into table ${APP}.ods_activity_info partition(dt='$do_date'); load data inpath '/origin_data/$APP/db/activity_order/$do_date' OVERWRITE into table ${APP}.ods_activity_order partition(dt='$do_date'); load data inpath '/origin_data/$APP/db/cart_info/$do_date' OVERWRITE into table ${APP}.ods_cart_info partition(dt='$do_date'); load data inpath '/origin_data/$APP/db/comment_info/$do_date' OVERWRITE into table ${APP}.ods_comment_info partition(dt='$do_date'); load data inpath '/origin_data/$APP/db/coupon_info/$do_date' OVERWRITE into table ${APP}.ods_coupon_info partition(dt='$do_date'); load data inpath '/origin_data/$APP/db/coupon_use/$do_date' OVERWRITE into table ${APP}.ods_coupon_use partition(dt='$do_date'); load data inpath '/origin_data/$APP/db/favor_info/$do_date' OVERWRITE into table ${APP}.ods_favor_info partition(dt='$do_date'); load data inpath '/origin_data/$APP/db/order_refund_info/$do_date' OVERWRITE into table ${APP}.ods_order_refund_info partition(dt='$do_date'); load data inpath '/origin_data/$APP/db/order_status_log/$do_date' OVERWRITE into table ${APP}.ods_order_status_log partition(dt='$do_date'); load data inpath '/origin_data/$APP/db/spu_info/$do_date' OVERWRITE into table ${APP}.ods_spu_info partition(dt='$do_date'); load data inpath '/origin_data/$APP/db/activity_rule/$do_date' OVERWRITE into table ${APP}.ods_activity_rule partition(dt='$do_date'); load data inpath '/origin_data/$APP/db/base_dic/$do_date' OVERWRITE into table ${APP}.ods_base_dic partition(dt='$do_date');
"sql2="
load data inpath '/origin_data/$APP/db/base_province/$do_date' OVERWRITE into table ${APP}.ods_base_province;load data inpath '/origin_data/$APP/db/base_region/$do_date' OVERWRITE into table ${APP}.ods_base_region;
"
case $1 in
"first"){
#$hive -e "$sql1"
#$hive -e "$sql2"sudo -u hive  hive -e "$sql1"sudo -u hive  hive -e "$sql2"
};;
"all"){
#$hive -e "$sql1"sudo -u hive  hive -e "$sql1"
};;
esac

DWD层：

DWD层创建基础明细表，明细表用于存储ODS层原始表转换过来的明细数据。

1. 启动json数据展开，实现DWD层

建表

drop table if exists dwd_start_log;
CREATE EXTERNAL TABLE dwd_start_log(
`mid_id` string,
`user_id` string,
`version_code` string,
`version_name` string,
`lang` string,
`source` string,
`os` string,
`area` string,
`model` string,
`brand` string,
`sdk_version` string,
`gmail` string,
`height_width` string,
`app_time` string,
`network` string,
`lng` string,
`lat` string,
`entry` string,
`open_ad_type` string,
`action` string,
`loading_time` string,
`detail` string,
`extend1` string
)
PARTITIONED BY (dt string)
stored as parquet
location '/warehouse/gmall/dwd/dwd_start_log/'
TBLPROPERTIES('parquet.compression'='lzo');

ods->dwd

#!/bin/bash# 定义变量方便修改
APP=gmall# 如果是输入的日期按照取输入日期；如果没输入日期取当前时间的前一天
if [ -n "$1" ] ;thendo_date=$1
else do_date=`date -d "-1 day" +%F`
fi sql="
set hive.exec.dynamic.partition.mode=nonstrict;insert overwrite table "$APP".dwd_start_log
PARTITION (dt='$do_date')
select get_json_object(line,'$.mid') mid_id,get_json_object(line,'$.uid') user_id,get_json_object(line,'$.vc') version_code,get_json_object(line,'$.vn') version_name,get_json_object(line,'$.l') lang,get_json_object(line,'$.sr') source,get_json_object(line,'$.os') os,get_json_object(line,'$.ar') area,get_json_object(line,'$.md') model,get_json_object(line,'$.ba') brand,get_json_object(line,'$.sv') sdk_version,get_json_object(line,'$.g') gmail,get_json_object(line,'$.hw') height_width,get_json_object(line,'$.t') app_time,get_json_object(line,'$.nw') network,get_json_object(line,'$.ln') lng,get_json_object(line,'$.la') lat,get_json_object(line,'$.entry') entry,get_json_object(line,'$.open_ad_type') open_ad_type,get_json_object(line,'$.action') action,get_json_object(line,'$.loading_time') loading_time,get_json_object(line,'$.detail') detail,get_json_object(line,'$.extend1') extend1
from "$APP".ods_start_log
where dt='$do_date';
"$hive -e "$sql"

2. 事件行为json数据展开，实现DWD层，最后实现的表包括：商品点击表，商品详情页表，商品列表页表，评论表，

参考1，复杂一些，这里省略相关语句。

3.业务数据的展开，实现DWD层

维度表（全量表）：

商品维度表，优惠券信息表，活动维度表，地区维度表，时间维度表，用户维度表（拉链表），编码字典维度表


DROP TABLE IF EXISTS `dwd_dim_sku_info`;
CREATE EXTERNAL TABLE `dwd_dim_sku_info` (`id` string COMMENT '商品id',`spu_id` string COMMENT 'spuid',`price` double COMMENT '商品价格',`sku_name` string COMMENT '商品名称',`sku_desc` string COMMENT '商品描述',`weight` double COMMENT '重量',`tm_id` string COMMENT '品牌id',`tm_name` string COMMENT '品牌名称',`category3_id` string COMMENT '三级分类id',`category2_id` string COMMENT '二级分类id',`category1_id` string COMMENT '一级分类id',`category3_name` string COMMENT '三级分类名称',`category2_name` string COMMENT '二级分类名称',`category1_name` string COMMENT '一级分类名称',`spu_name` string COMMENT 'spu名称',`create_time` string COMMENT '创建时间'
)
COMMENT '商品维度表'
PARTITIONED BY (`dt` string)
stored as parquet
location '/warehouse/gmall/dwd/dwd_dim_sku_info/'
tblproperties ("parquet.compression"="lzo");insert overwrite table dwd_dim_sku_info partition(dt='2020-03-29')
select  sku.id,sku.spu_id,sku.price,sku.sku_name,sku.sku_desc,sku.weight,sku.tm_id,ob.tm_name,sku.category3_id,c2.id category2_id,c1.id category1_id,c3.name category3_name,c2.name category2_name,c1.name category1_name,spu.spu_name,sku.create_time
from
(select * from ods_sku_info where dt='2020-03-29'
)sku
join
(select * from ods_base_trademark where dt='2020-03-29'
)ob on sku.tm_id=ob.tm_id
join
(select * from ods_spu_info where dt='2020-03-29'
)spu on spu.id = sku.spu_id
join
(select * from ods_base_category3 where dt='2020-03-29'
)c3 on sku.category3_id=c3.id
join
(select * from ods_base_category2 where dt='2020-03-29'
)c2 on c3.category2_id=c2.id
join
(select * from ods_base_category1 where dt='2020-03-29'
)c1 on c2.category1_id=c1.id;

事实表：订单明细事实表（事务型快照事实表），支付事实表，退款事实表（事务型快照事实表），评价事实表（事务型快照事实表），加购事实表（周期型快照事实表，每日快照），收藏事实表（周期型快照事实表，每日快照），优惠券领用事实表（累积型快照事实表），订单事实表（累积型快照事实表）

例如：订单明细事实表

drop table if exists ods_order_detail;
create external table ods_order_detail( `id` string COMMENT '订单编号',`order_id` string  COMMENT '订单号', `user_id` string COMMENT '用户id',`sku_id` string COMMENT '商品id',`sku_name` string COMMENT '商品名称',`order_price` decimal(10,2) COMMENT '商品价格',`sku_num` bigint COMMENT '商品数量',`create_time` string COMMENT '创建时间'
) COMMENT '订单详情表'
PARTITIONED BY (`dt` string)
row format delimited fields terminated by '\t'
STORED ASINPUTFORMAT 'com.hadoop.mapred.DeprecatedLzoTextInputFormat'OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat'
location '/warehouse/gmall/ods/ods_order_detail/';


insert overwrite table dwd_dim_sku_info partition(dt='2020-03-29')
select  sku.id,sku.spu_id,sku.price,sku.sku_name,sku.sku_desc,sku.weight,sku.tm_id,ob.tm_name,sku.category3_id,c2.id category2_id,c1.id category1_id,c3.name category3_name,c2.name category2_name,c1.name category1_name,spu.spu_name,sku.create_time
from
(select * from ods_sku_info where dt='2020-03-29'
)sku
join
(select * from ods_base_trademark where dt='2020-03-29'
)ob on sku.tm_id=ob.tm_id
join
(select * from ods_spu_info where dt='2020-03-29'
)spu on spu.id = sku.spu_id
join
(select * from ods_base_category3 where dt='2020-03-29'
)c3 on sku.category3_id=c3.id
join
(select * from ods_base_category2 where dt='2020-03-29'
)c2 on c3.category2_id=c2.id
join
(select * from ods_base_category1 where dt='2020-03-29'
)c1 on c2.category1_id=c1.id;

DWS层

dws层的数据从dwd层来选取，在建模上进行主题分类，分类后每个主题都制作一张表，按照dwd层的数据，进行每日的轻度聚合，一般是建宽表。

需要下面的表

例如 DWS层每日设备行为（用户行为）

drop table if exists dws_uv_detail_daycount;
create external table dws_uv_detail_daycount
(`mid_id` string COMMENT '设备唯一标识',`user_id` string COMMENT '用户标识',`version_code` string COMMENT '程序版本号', `version_name` string COMMENT '程序版本名', `lang` string COMMENT '系统语言', `source` string COMMENT '渠道号', `os` string COMMENT '安卓系统版本', `area` string COMMENT '区域', `model` string COMMENT '手机型号', `brand` string COMMENT '手机品牌', `sdk_version` string COMMENT 'sdkVersion', `gmail` string COMMENT 'gmail', `height_width` string COMMENT '屏幕宽高',`app_time` string COMMENT '客户端日志产生时的时间',`network` string COMMENT '网络模式',`lng` string COMMENT '经度',`lat` string COMMENT '纬度',`login_count` bigint COMMENT '活跃次数'
)
partitioned by(dt string)
stored as parquet
location '/warehouse/gmall/dws/dws_uv_detail_daycount'
TBLPROPERTIES('parquet.compression'='lzo');insert overwrite table dws_uv_detail_daycount partition(dt='2020-03-29')
select  mid_id,concat_ws('|', collect_set(user_id)) user_id,concat_ws('|', collect_set(version_code)) version_code,concat_ws('|', collect_set(version_name)) version_name,concat_ws('|', collect_set(lang))lang,concat_ws('|', collect_set(source)) source,concat_ws('|', collect_set(os)) os,concat_ws('|', collect_set(area)) area, concat_ws('|', collect_set(model)) model,concat_ws('|', collect_set(brand)) brand,concat_ws('|', collect_set(sdk_version)) sdk_version,concat_ws('|', collect_set(gmail)) gmail,concat_ws('|', collect_set(height_width)) height_width,concat_ws('|', collect_set(app_time)) app_time,concat_ws('|', collect_set(network)) network,concat_ws('|', collect_set(lng)) lng,concat_ws('|', collect_set(lat)) lat,count(*) login_count
from dwd_start_log
where dt='2020-03-29'
group by mid_id;

DWT层

DWT层将DWS层每日聚合的数据进行积累，DWT层不是分区表，是一个累积型全量表，并且数据来源于DWS层。

累积性全量表：查询要改动的旧数据，查询新增和变化的新数据，新旧关联，以新换旧，导入覆盖。

DWT层

设备主题宽表

会员主题宽表

商品主题宽表

优惠券主题宽表

活动主题宽表

会员主题（ADS层

例如

DWT设备主题宽表

drop table if exists dwt_uv_topic;
create external table dwt_uv_topic
(`mid_id` string COMMENT '设备唯一标识',`user_id` string COMMENT '用户标识',`version_code` string COMMENT '程序版本号',`version_name` string COMMENT '程序版本名',`lang` string COMMENT '系统语言',`source` string COMMENT '渠道号',`os` string COMMENT '安卓系统版本',`area` string COMMENT '区域',`model` string COMMENT '手机型号',`brand` string COMMENT '手机品牌',`sdk_version` string COMMENT 'sdkVersion',`gmail` string COMMENT 'gmail',`height_width` string COMMENT '屏幕宽高',`app_time` string COMMENT '客户端日志产生时的时间',`network` string COMMENT '网络模式',`lng` string COMMENT '经度',`lat` string COMMENT '纬度',`login_date_first` string  comment '首次活跃时间',`login_date_last` string  comment '末次活跃时间',`login_day_count` bigint comment '当日活跃次数',`login_count` bigint comment '累积活跃天数'
)
stored as parquet
location '/warehouse/gmall/dwt/dwt_uv_topic'
tblproperties ("parquet.compression"="lzo");insert overwrite table dwt_uv_topic
selectnvl(new.mid_id,old.mid_id),nvl(new.user_id,old.user_id),nvl(new.version_code,old.version_code),nvl(new.version_name,old.version_name),nvl(new.lang,old.lang),nvl(new.source,old.source),nvl(new.os,old.os),nvl(new.area,old.area),nvl(new.model,old.model),nvl(new.brand,old.brand),nvl(new.sdk_version,old.sdk_version),nvl(new.gmail,old.gmail),nvl(new.height_width,old.height_width),nvl(new.app_time,old.app_time),nvl(new.network,old.network),nvl(new.lng,old.lng),nvl(new.lat,old.lat),if(old.mid_id is null,'2020-03-29',old.login_date_first),if(new.mid_id is not null,'2020-03-29',old.login_date_last),if(new.mid_id is not null, new.login_count,0),nvl(old.login_count,0)+if(new.login_count>0,1,0)
from
(select*from dwt_uv_topic
)old
full outer join
(select*from dws_uv_detail_daycountwhere dt='2020-03-29'
)new
on old.mid_id=new.mid_id;

ADS层