序列化Writable接口
2024-05-11 01:34:20
基本序列化类型往往不能满足所有需求，比如在Hadoop框架内部传递一个自定义bean对象，那么该对象就需要实现Writable序列化接口。
实现Writable序列化步骤如下
（1）必须实现Writable接口
（2）反序列化时，需要反射调用空参构造函数，所以必须有空参构造
public CustomBean() {super();
}（3）重写序列化方法
@Override
public void write(DataOutput out) throws IOException {....
}（4）重写反序列化方法
@Override
public void readFields(DataInput in) throws IOException {....
}（5）反序列化的字段顺序和序列化字段的顺序必须完全一致
（6）方便展示结果数据，需要重写bean对象的toString()方法，可以自定义分隔符
（7）如果自定义Bean对象需要放在Mapper输出KV中的K,则该对象还需实现Comparable接口，因为因为MapReduce框中的Shuffle过程要求对key必须能排序！！
@Override
public int compareTo(CustomBean o) {// 自定义排序规则return this.num > o.getNum() ? -1 : 1;
}Writable接口案例
1、需求
统计每台智能音箱设备内容播放时长
原始日志格式
001 001577c3 kar_890809 120.196.100.99 1116 954 200
日志id 设备id appkey(合作硬件厂商) 网络ip 自有内容时长(秒) 第三方内容时长(秒) 网络状态码输出结果
001577c3 11160 9540 20700
设备id 自有内容时长(秒) 第三方内容时长(秒) 总时长mr编程总结
- map()方法输出的kv以及具体类型如何确定？mr中map()方法输出的key如果是相同key则会去往同个reduce调用reduce方法。
- reduce()方法中输入参数kv，key:map()方法输出某个key，而value:value是一个集合，这个集合中的数据都来自于map输出的kv，而且是k相同的所有kv对的value集合在一起。
整体思路分析
Map阶段：
- 读取一行文本数据，按照制表符切分
- 抽取出自由内容时长，第三方内容时长，设备id
- 输出：key-->设备id，value:封装一个bean对象，bean对象携带自由时长，第三方内容时长，设备id
- 自定义bean对象作为value输出，需要实现writable序列化接口
Reduce阶段：
在reduce方法中直接遍历迭代器，累加时长即可
原始数据：
01 a00df6s kar 120.196.100.99 384 33 200
11 0sfs01 kar 120.196.100.99 198 86 200
21 adfd00fd5 pandora 120.196.100.99 513 261 200
31 0ad0s7 pandora 120.196.100.99 840 413 200
41 0sfs01 kar 120.196.100.99 190 401 200
51 00fdaf3 kar 120.196.100.99 273 527 200
61 00fdaf3 pandora 120.196.100.99 950 526 200
71 0a0fe2 kar 120.196.100.99 730 496 200
81 0a0fe2 pandora 120.196.100.99 367 759 200
91 0sfs01 kar 120.196.100.99 529 484 200
101 adfd00fd5 kar 120.196.100.99 516 18 200
111 a00df6s pandora 120.196.100.99 300 745 200
121 0a0fe2 kar 120.196.100.99 416 847 200
131 00fdaf3 kar 120.196.100.99 268 489 200
141 00wersa4 kar 120.196.100.99 74 815 200
151 00wersa4 pandora 120.196.100.99 445 293 200
161 00fdaf3 kar 120.196.100.99 673 77 200
171 a00df6s kar 120.196.100.99 378 138 200
181 0sfs01 pandora 120.196.100.99 55 80 200
191 00fdaf3 kar 120.196.100.99 699 7 200
201 00fdaf3 pandora 120.196.100.99 752 687 200
211 00wersa4 kar 120.196.100.99 448 117 200
221 a00df6s pandora 120.196.100.99 17 771 200
231 00fdaf3 kar 120.196.100.99 732 701 200
241 0ad0s7 pandora 120.196.100.99 923 24 200
251 0sfs01 pandora 120.196.100.99 476 781 200
261 adfd00fd5 pandora 120.196.100.99 655 163 200
271 0a0fe2 pandora 120.196.100.99 128 251 200
281 00fdaf3 kar 120.196.100.99 704 149 200
291 a00df6s pandora 120.196.100.99 839 383 200
301 0sfs01 pandora 120.196.100.99 230 38 200
311 adfd00fd5 kar 120.196.100.99 100 661 200
321 adfd00fd5 kar 120.196.100.99 839 442 200
331 0sfs01 pandora 120.196.100.99 165 256 200
341 a00df6s pandora 120.196.100.99 950 247 200
351 0ad0s7 pandora 120.196.100.99 791 760 200
361 0a0fe2 pandora 120.196.100.99 118 180 200
371 adfd00fd5 kar 120.196.100.99 749 581 200
381 0a0fe2 pandora 120.196.100.99 69 324 200
391 0ad0s7 pandora 120.196.100.99 337 144 200
401 0a0fe2 kar 120.196.100.99 43 120 200
411 adfd00fd5 pandora 120.196.100.99 311 608 200
421 00wersa4 pandora 120.196.100.99 10 680 200
431 00wersa4 pandora 120.196.100.99 206 843 200
441 00fdaf3 kar 120.196.100.99 733 58 200
451 00wersa4 kar 120.196.100.99 247 528 200
461 00fdaf3 pandora 120.196.100.99 403 121 200
471 adfd00fd5 pandora 120.196.100.99 494 771 200
481 00fdaf3 kar 120.196.100.99 770 511 200
491 0ad0s7 kar 120.196.100.99 686 313 200
501 adfd00fd5 pandora 120.196.100.99 566 914 200
511 00fdaf3 kar 120.196.100.99 895 687 200
521 00wersa4 kar 120.196.100.99 217 322 200
531 0sfs01 kar 120.196.100.99 585 766 200
541 adfd00fd5 kar 120.196.100.99 450 209 200
551 00fdaf3 kar 120.196.100.99 392 658 200
561 a00df6s kar 120.196.100.99 33 689 200
571 0a0fe2 kar 120.196.100.99 71 550 200
581 0sfs01 pandora 120.196.100.99 742 345 200
591 a00df6s pandora 120.196.100.99 335 640 200
601 00fdaf3 kar 120.196.100.99 807 752 200
611 00fdaf3 kar 120.196.100.99 447 811 200
621 00wersa4 kar 120.196.100.99 32 120 200
631 a00df6s pandora 120.196.100.99 170 472 200
641 0sfs01 kar 120.196.100.99 538 915 200
651 0sfs01 pandora 120.196.100.99 870 659 200
661 0a0fe2 pandora 120.196.100.99 158 97 200
671 0ad0s7 pandora 120.196.100.99 97 276 200
681 adfd00fd5 pandora 120.196.100.99 713 80 200
691 0a0fe2 pandora 120.196.100.99 324 818 200
701 00fdaf3 pandora 120.196.100.99 808 386 200
711 00wersa4 kar 120.196.100.99 418 129 200
721 0sfs01 kar 120.196.100.99 994 842 200
731 0ad0s7 pandora 120.196.100.99 639 525 200
741 a00df6s kar 120.196.100.99 827 134 200
751 00wersa4 kar 120.196.100.99 594 44 200
761 0ad0s7 kar 120.196.100.99 197 558 200
771 00wersa4 pandora 120.196.100.99 477 832 200
781 0a0fe2 pandora 120.196.100.99 870 283 200
791 00fdaf3 pandora 120.196.100.99 50 50 200
801 a00df6s pandora 120.196.100.99 158 488 200
811 00wersa4 kar 120.196.100.99 701 736 200
821 0sfs01 pandora 120.196.100.99 127 316 200
831 00wersa4 pandora 120.196.100.99 670 719 200
841 adfd00fd5 kar 120.196.100.99 405 630 200
851 00wersa4 kar 120.196.100.99 78 976 200
861 00wersa4 kar 120.196.100.99 510 527 200
871 adfd00fd5 kar 120.196.100.99 62 407 200
881 00fdaf3 kar 120.196.100.99 873 403 200
891 0a0fe2 kar 120.196.100.99 157 431 200
901 a00df6s pandora 120.196.100.99 112 319 200
911 adfd00fd5 kar 120.196.100.99 144 14 200
921 00fdaf3 kar 120.196.100.99 879 444 200
931 a00df6s pandora 120.196.100.99 882 47 200
941 adfd00fd5 pandora 120.196.100.99 620 114 200
951 00wersa4 kar 120.196.100.99 35 59 200
961 00wersa4 kar 120.196.100.99 79 181 200
971 00wersa4 kar 120.196.100.99 34 758 200
981 a00df6s pandora 120.196.100.99 649 907 200
991 0ad0s7 kar 120.196.100.99 206 408 200
1001 0a0fe2 pandora 120.196.100.99 710 300 200
1011 0sfs01 pandora 120.196.100.99 326 232 200
1021 0ad0s7 pandora 120.196.100.99 749 648 200
1031 00fdaf3 kar 120.196.100.99 749 522 200
1041 0ad0s7 kar 120.196.100.99 516 692 200
1051 adfd00fd5 kar 120.196.100.99 242 706 200
1061 a00df6s kar 120.196.100.99 6 477 200
1071 00wersa4 pandora 120.196.100.99 401 79 200
1081 00wersa4 kar 120.196.100.99 238 178 200
1091 0ad0s7 pandora 120.196.100.99 613 640 200
1101 00fdaf3 pandora 120.196.100.99 83 925 200
1111 0ad0s7 pandora 120.196.100.99 424 937 200
1121 00wersa4 pandora 120.196.100.99 468 194 200
1131 0sfs01 pandora 120.196.100.99 790 503 200
1141 00fdaf3 pandora 120.196.100.99 51 839 200
1151 0sfs01 kar 120.196.100.99 480 399 200
1161 00fdaf3 pandora 120.196.100.99 884 130 200
1171 a00df6s kar 120.196.100.99 580 955 200
1181 0ad0s7 pandora 120.196.100.99 316 358 200
1191 a00df6s pandora 120.196.100.99 255 884 200
1201 adfd00fd5 pandora 120.196.100.99 244 813 200
1211 0sfs01 pandora 120.196.100.99 693 504 200
1221 adfd00fd5 pandora 120.196.100.99 708 222 200
1231 0a0fe2 pandora 120.196.100.99 125 845 200
1241 0sfs01 kar 120.196.100.99 274 528 200
1251 0ad0s7 pandora 120.196.100.99 679 196 200
1261 00wersa4 pandora 120.196.100.99 859 541 200
1271 0sfs01 kar 120.196.100.99 279 606 200
1281 00wersa4 pandora 120.196.100.99 62 720 200
1291 adfd00fd5 pandora 120.196.100.99 833 466 200
1301 0ad0s7 kar 120.196.100.99 622 438 200
1311 00wersa4 pandora 120.196.100.99 356 916 200
1321 adfd00fd5 pandora 120.196.100.99 214 640 200
1331 00wersa4 kar 120.196.100.99 369 15 200
1341 0a0fe2 kar 120.196.100.99 404 367 200
1351 0sfs01 kar 120.196.100.99 827 916 200
1361 00fdaf3 pandora 120.196.100.99 88 798 200
1371 0sfs01 kar 120.196.100.99 175 136 200
1381 00wersa4 pandora 120.196.100.99 132 988 200
1391 0sfs01 kar 120.196.100.99 307 369 200
1401 a00df6s pandora 120.196.100.99 483 951 200
1411 adfd00fd5 pandora 120.196.100.99 760 689 200
1421 0sfs01 pandora 120.196.100.99 75 449 200
1431 00fdaf3 kar 120.196.100.99 17 969 200
1441 a00df6s pandora 120.196.100.99 685 541 200
1451 00fdaf3 pandora 120.196.100.99 690 935 200
1461 0a0fe2 kar 120.196.100.99 161 794 200
1471 0ad0s7 kar 120.196.100.99 772 400 200
1481 0sfs01 kar 120.196.100.99 885 567 200
1491 0a0fe2 kar 120.196.100.99 597 646 200
1501 0sfs01 pandora 120.196.100.99 93 157 200
1511 0a0fe2 pandora 120.196.100.99 290 18 200
1521 adfd00fd5 pandora 120.196.100.99 241 368 200
1531 a00df6s kar 120.196.100.99 47 658 200
1541 0a0fe2 pandora 120.196.100.99 899 878 200
1551 00wersa4 pandora 120.196.100.99 787 956 200
1561 a00df6s pandora 120.196.100.99 888 21 200
1571 adfd00fd5 kar 120.196.100.99 439 41 200
1581 0a0fe2 kar 120.196.100.99 564 792 200
1591 adfd00fd5 pandora 120.196.100.99 661 682 200
1601 0a0fe2 kar 120.196.100.99 680 11 200
1611 a00df6s kar 120.196.100.99 18 566 200
1621 a00df6s kar 120.196.100.99 613 658 200
1631 00fdaf3 kar 120.196.100.99 574 242 200
1641 0ad0s7 pandora 120.196.100.99 992 453 200
1651 0a0fe2 kar 120.196.100.99 592 334 200
1661 0sfs01 kar 120.196.100.99 30 719 200
1671 00wersa4 pandora 120.196.100.99 988 494 200
1681 0a0fe2 kar 120.196.100.99 654 306 200
1691 0sfs01 kar 120.196.100.99 519 198 200
1701 adfd00fd5 pandora 120.196.100.99 796 83 200
1711 0a0fe2 pandora 120.196.100.99 318 701 200
1721 0a0fe2 pandora 120.196.100.99 98 244 200
1731 adfd00fd5 pandora 120.196.100.99 293 634 200
1741 00wersa4 pandora 120.196.100.99 62 812 200
1751 0sfs01 kar 120.196.100.99 548 130 200
1761 0a0fe2 pandora 120.196.100.99 496 120 200
1771 0sfs01 kar 120.196.100.99 737 223 200
1781 0a0fe2 kar 120.196.100.99 498 262 200
1791 adfd00fd5 kar 120.196.100.99 218 193 200
1801 00fdaf3 kar 120.196.100.99 120 564 200
1811 adfd00fd5 pandora 120.196.100.99 177 209 200
1821 00fdaf3 kar 120.196.100.99 392 397 200
1831 0a0fe2 pandora 120.196.100.99 824 336 200
1841 a00df6s pandora 120.196.100.99 721 395 200
1851 0sfs01 kar 120.196.100.99 764 660 200
1861 0ad0s7 pandora 120.196.100.99 589 101 200
1871 00wersa4 pandora 120.196.100.99 356 356 200
1881 0sfs01 pandora 120.196.100.99 216 307 200
1891 00fdaf3 kar 120.196.100.99 1 582 200
1901 00wersa4 kar 120.196.100.99 912 436 200
1911 0sfs01 pandora 120.196.100.99 405 719 200
1921 a00df6s kar 120.196.100.99 941 542 200
1931 00wersa4 kar 120.196.100.99 966 490 200
1941 00fdaf3 pandora 120.196.100.99 960 454 200
1951 0ad0s7 kar 120.196.100.99 545 39 200
1961 00fdaf3 pandora 120.196.100.99 293 299 200
1971 0a0fe2 kar 120.196.100.99 825 462 200
1981 adfd00fd5 kar 120.196.100.99 34 751 200
1991 a00df6s kar 120.196.100.99 807 569 200
2001 0a0fe2 pandora 120.196.100.99 160 973 200
2011 00wersa4 pandora 120.196.100.99 497 637 200
2021 00fdaf3 kar 120.196.100.99 302 27 200
2031 0sfs01 pandora 120.196.100.99 653 104 200
2041 0a0fe2 kar 120.196.100.99 963 65 200
2051 0ad0s7 kar 120.196.100.99 204 828 200
2061 00wersa4 kar 120.196.100.99 91 656 200
2071 0a0fe2 pandora 120.196.100.99 937 34 200
2081 adfd00fd5 pandora 120.196.100.99 350 748 200
2091 0a0fe2 pandora 120.196.100.99 97 688 200
2101 0ad0s7 kar 120.196.100.99 430 338 200
2111 0sfs01 pandora 120.196.100.99 595 827 200
2121 0a0fe2 kar 120.196.100.99 726 578 200
2131 adfd00fd5 pandora 120.196.100.99 31 501 200
2141 0ad0s7 kar 120.196.100.99 389 925 200
2151 adfd00fd5 pandora 120.196.100.99 136 559 200
2161 00wersa4 pandora 120.196.100.99 3 927 200
2171 adfd00fd5 pandora 120.196.100.99 908 583 200
2181 00wersa4 pandora 120.196.100.99 921 598 200
2191 adfd00fd5 pandora 120.196.100.99 112 956 200
2201 0sfs01 kar 120.196.100.99 115 32 200
2211 a00df6s pandora 120.196.100.99 86 996 200
2221 0ad0s7 pandora 120.196.100.99 790 705 200
2231 0a0fe2 kar 120.196.100.99 24 548 200
2241 a00df6s kar 120.196.100.99 611 58 200
2251 00fdaf3 pandora 120.196.100.99 500 800 200
2261 adfd00fd5 kar 120.196.100.99 852 530 200
2271 00wersa4 pandora 120.196.100.99 208 957 200
2281 0a0fe2 kar 120.196.100.99 900 137 200
2291 adfd00fd5 kar 120.196.100.99 292 550 200
2301 0a0fe2 kar 120.196.100.99 977 983 200
2311 00fdaf3 pandora 120.196.100.99 517 244 200
2321 0a0fe2 pandora 120.196.100.99 419 481 200
2331 00fdaf3 kar 120.196.100.99 117 500 200
2341 adfd00fd5 pandora 120.196.100.99 76 68 200
2351 00wersa4 pandora 120.196.100.99 318 285 200
2361 00wersa4 pandora 120.196.100.99 263 72 200
2371 00fdaf3 kar 120.196.100.99 222 150 200
2381 00wersa4 pandora 120.196.100.99 210 493 200
2391 0sfs01 pandora 120.196.100.99 538 4 200
2401 0ad0s7 kar 120.196.100.99 214 880 200
2411 adfd00fd5 pandora 120.196.100.99 709 369 200
2421 0a0fe2 kar 120.196.100.99 411 383 200
2431 0ad0s7 pandora 120.196.100.99 15 350 200
2441 0a0fe2 kar 120.196.100.99 135 440 200
2451 0sfs01 pandora 120.196.100.99 120 737 200
2461 0a0fe2 kar 120.196.100.99 941 510 200
2471 adfd00fd5 pandora 120.196.100.99 363 263 200
2481 0sfs01 kar 120.196.100.99 928 427 200
2491 0ad0s7 pandora 120.196.100.99 590 635 200
2501 a00df6s pandora 120.196.100.99 335 967 200
2511 0sfs01 pandora 120.196.100.99 214 756 200
2521 0a0fe2 kar 120.196.100.99 208 119 200
2531 adfd00fd5 kar 120.196.100.99 821 579 200
2541 00wersa4 pandora 120.196.100.99 319 262 200
2551 adfd00fd5 pandora 120.196.100.99 310 10 200
2561 00fdaf3 pandora 120.196.100.99 389 103 200
2571 00wersa4 kar 120.196.100.99 772 957 200
2581 0a0fe2 pandora 120.196.100.99 43 359 200
2591 00wersa4 pandora 120.196.100.99 920 4 200
2601 0a0fe2 kar 120.196.100.99 542 391 200
2611 adfd00fd5 kar 120.196.100.99 730 407 200
2621 0a0fe2 pandora 120.196.100.99 682 712 200
2631 0a0fe2 pandora 120.196.100.99 349 304 200
2641 00fdaf3 pandora 120.196.100.99 529 678 200
2651 0sfs01 kar 120.196.100.99 109 100 200
2661 0ad0s7 pandora 120.196.100.99 715 501 200
2671 0sfs01 kar 120.196.100.99 401 41 200
2681 a00df6s kar 120.196.100.99 5 110 200
2691 00wersa4 pandora 120.196.100.99 155 377 200
2701 00fdaf3 kar 120.196.100.99 172 954 200
2711 0a0fe2 kar 120.196.100.99 772 838 200
2721 00wersa4 kar 120.196.100.99 984 397 200
2731 00wersa4 kar 120.196.100.99 803 451 200
2741 0ad0s7 kar 120.196.100.99 636 380 200
2751 0ad0s7 pandora 120.196.100.99 565 288 200
2761 adfd00fd5 kar 120.196.100.99 74 423 200
2771 adfd00fd5 pandora 120.196.100.99 643 734 200
2781 0sfs01 pandora 120.196.100.99 808 847 200
2791 adfd00fd5 kar 120.196.100.99 377 415 200
2801 0a0fe2 pandora 120.196.100.99 263 968 200
2811 0a0fe2 kar 120.196.100.99 258 287 200
2821 0ad0s7 kar 120.196.100.99 32 248 200
2831 0ad0s7 kar 120.196.100.99 343 213 200
2841 00fdaf3 pandora 120.196.100.99 239 62 200
2851 0sfs01 kar 120.196.100.99 327 58 200
2861 a00df6s kar 120.196.100.99 146 691 200
2871 adfd00fd5 pandora 120.196.100.99 158 896 200
2881 00fdaf3 kar 120.196.100.99 897 788 200
2891 0sfs01 pandora 120.196.100.99 904 755 200
2901 0sfs01 pandora 120.196.100.99 661 781 200
2911 00wersa4 kar 120.196.100.99 516 379 200
2921 0a0fe2 kar 120.196.100.99 946 791 200
2931 adfd00fd5 pandora 120.196.100.99 83 272 200
2941 a00df6s pandora 120.196.100.99 350 906 200
2951 0ad0s7 kar 120.196.100.99 711 580 200
2961 0sfs01 pandora 120.196.100.99 162 939 200
2971 00fdaf3 pandora 120.196.100.99 38 956 200
2981 00fdaf3 kar 120.196.100.99 406 567 200
2991 00fdaf3 pandora 120.196.100.99 191 47 200
3001 adfd00fd5 kar 120.196.100.99 78 229 200
3011 0ad0s7 kar 120.196.100.99 122 12 200
3021 adfd00fd5 pandora 120.196.100.99 137 856 200
3031 a00df6s pandora 120.196.100.99 610 0 200
3041 0a0fe2 kar 120.196.100.99 598 722 200
3051 adfd00fd5 kar 120.196.100.99 195 629 200
3061 0a0fe2 kar 120.196.100.99 550 482 200
3071 00fdaf3 kar 120.196.100.99 781 76 200
3081 a00df6s pandora 120.196.100.99 623 78 200
3091 00wersa4 pandora 120.196.100.99 513 628 200
3101 00fdaf3 kar 120.196.100.99 893 809 200
3111 00fdaf3 pandora 120.196.100.99 720 447 200
3121 0a0fe2 kar 120.196.100.99 521 662 200
3131 00wersa4 kar 120.196.100.99 169 291 200
3141 0a0fe2 kar 120.196.100.99 676 930 200
3151 a00df6s kar 120.196.100.99 522 775 200
3161 0ad0s7 kar 120.196.100.99 889 421 200
3171 a00df6s kar 120.196.100.99 299 833 200
3181 0a0fe2 pandora 120.196.100.99 912 209 200
3191 0sfs01 kar 120.196.100.99 856 670 200
3201 0ad0s7 pandora 120.196.100.99 151 929 200
3211 0a0fe2 pandora 120.196.100.99 213 442 200
3221 0ad0s7 pandora 120.196.100.99 857 11 200
3231 0sfs01 pandora 120.196.100.99 361 659 200
3241 0a0fe2 kar 120.196.100.99 953 374 200
3251 adfd00fd5 kar 120.196.100.99 917 594 200
3261 a00df6s pandora 120.196.100.99 909 268 200
3271 0ad0s7 kar 120.196.100.99 828 785 200
3281 a00df6s kar 120.196.100.99 85 953 200
3291 a00df6s pandora 120.196.100.99 715 757 200
3301 00fdaf3 kar 120.196.100.99 82 105 200
3311 00wersa4 kar 120.196.100.99 575 972 200
3321 a00df6s pandora 120.196.100.99 305 573 200
3331 0ad0s7 kar 120.196.100.99 675 64 200
3341 0sfs01 pandora 120.196.100.99 488 38 200
3351 adfd00fd5 pandora 120.196.100.99 553 283 200
3361 adfd00fd5 pandora 120.196.100.99 973 608 200
3371 0a0fe2 pandora 120.196.100.99 166 296 200
3381 00fdaf3 kar 120.196.100.99 156 608 200
3391 00wersa4 kar 120.196.100.99 640 880 200
3401 00wersa4 kar 120.196.100.99 679 598 200
3411 a00df6s kar 120.196.100.99 345 650 200
3421 0a0fe2 kar 120.196.100.99 61 99 200
3431 00fdaf3 pandora 120.196.100.99 139 384 200
3441 0ad0s7 kar 120.196.100.99 901 19 200
3451 00fdaf3 kar 120.196.100.99 379 920 200
3461 0ad0s7 kar 120.196.100.99 153 252 200
3471 a00df6s pandora 120.196.100.99 816 897 200
3481 adfd00fd5 kar 120.196.100.99 126 237 200
3491 a00df6s pandora 120.196.100.99 890 993 200
3501 0ad0s7 kar 120.196.100.99 656 855 200
3511 a00df6s kar 120.196.100.99 21 541 200
3521 a00df6s pandora 120.196.100.99 264 407 200
3531 00fdaf3 pandora 120.196.100.99 749 849 200
3541 0sfs01 kar 120.196.100.99 975 481 200
3551 adfd00fd5 pandora 120.196.100.99 762 107 200
3561 a00df6s kar 120.196.100.99 514 446 200
3571 a00df6s kar 120.196.100.99 123 212 200
3581 00wersa4 kar 120.196.100.99 422 23 200
3591 a00df6s pandora 120.196.100.99 564 638 200
3601 00fdaf3 kar 120.196.100.99 866 543 200
3611 00fdaf3 pandora 120.196.100.99 601 542 200
3621 adfd00fd5 kar 120.196.100.99 321 748 200
3631 a00df6s kar 120.196.100.99 458 323 200
3641 0sfs01 kar 120.196.100.99 503 685 200
3651 0ad0s7 kar 120.196.100.99 551 173 200
3661 00wersa4 pandora 120.196.100.99 833 21 200
3671 0a0fe2 kar 120.196.100.99 171 694 200
3681 0sfs01 pandora 120.196.100.99 943 846 200
3691 a00df6s pandora 120.196.100.99 339 316 200
3701 0a0fe2 kar 120.196.100.99 524 511 200
3711 0ad0s7 pandora 120.196.100.99 438 324 200
3721 adfd00fd5 pandora 120.196.100.99 9 6 200
3731 0ad0s7 kar 120.196.100.99 284 960 200
3741 00fdaf3 pandora 120.196.100.99 191 446 200
3751 0ad0s7 pandora 120.196.100.99 216 618 200
3761 adfd00fd5 pandora 120.196.100.99 812 906 200
3771 a00df6s kar 120.196.100.99 975 630 200
3781 0a0fe2 kar 120.196.100.99 578 47 200
3791 a00df6s pandora 120.196.100.99 448 227 200
3801 0a0fe2 kar 120.196.100.99 965 221 200
3811 a00df6s pandora 120.196.100.99 643 442 200
3821 0a0fe2 pandora 120.196.100.99 819 206 200
3831 a00df6s pandora 120.196.100.99 420 293 200
3841 00wersa4 kar 120.196.100.99 555 765 200
3851 00wersa4 pandora 120.196.100.99 576 817 200
3861 0a0fe2 kar 120.196.100.99 200 552 200
3871 0ad0s7 kar 120.196.100.99 372 212 200
3881 0sfs01 pandora 120.196.100.99 862 471 200
3891 a00df6s pandora 120.196.100.99 139 22 200
3901 0ad0s7 kar 120.196.100.99 849 682 200
3911 0ad0s7 pandora 120.196.100.99 62 884 200
3921 00fdaf3 pandora 120.196.100.99 578 453 200
3931 00wersa4 kar 120.196.100.99 867 536 200
3941 0sfs01 pandora 120.196.100.99 915 631 200
3951 0ad0s7 kar 120.196.100.99 456 449 200
3961 0sfs01 pandora 120.196.100.99 887 462 200
3971 00fdaf3 pandora 120.196.100.99 438 329 200
3981 adfd00fd5 pandora 120.196.100.99 117 956 200
3991 00fdaf3 pandora 120.196.100.99 32 178 200
4001 0a0fe2 pandora 120.196.100.99 140 573 200
4011 a00df6s kar 120.196.100.99 323 800 200
4021 a00df6s pandora 120.196.100.99 94 262 200
4031 0sfs01 kar 120.196.100.99 22 617 200
4041 00wersa4 kar 120.196.100.99 958 223 200
4051 0sfs01 kar 120.196.100.99 993 367 200
4061 0a0fe2 pandora 120.196.100.99 292 665 200
4071 0a0fe2 kar 120.196.100.99 551 684 200
4081 00wersa4 pandora 120.196.100.99 869 137 200
4091 a00df6s kar 120.196.100.99 309 596 200
4101 adfd00fd5 kar 120.196.100.99 157 450 200
4111 0ad0s7 pandora 120.196.100.99 90 86 200
4121 0sfs01 kar 120.196.100.99 231 118 200
4131 a00df6s pandora 120.196.100.99 727 435 200
4141 00wersa4 pandora 120.196.100.99 112 918 200
4151 adfd00fd5 pandora 120.196.100.99 262 61 200
4161 00fdaf3 kar 120.196.100.99 553 164 200
4171 0a0fe2 kar 120.196.100.99 597 321 200
4181 00fdaf3 kar 120.196.100.99 20 464 200
4191 adfd00fd5 kar 120.196.100.99 602 109 200
4201 adfd00fd5 kar 120.196.100.99 508 80 200
4211 0ad0s7 pandora 120.196.100.99 485 513 200
4221 0ad0s7 kar 120.196.100.99 159 237 200
4231 00fdaf3 pandora 120.196.100.99 979 530 200
4241 0sfs01 pandora 120.196.100.99 557 356 200
4251 a00df6s kar 120.196.100.99 502 810 200
4261 0a0fe2 pandora 120.196.100.99 223 642 200
4271 adfd00fd5 kar 120.196.100.99 720 89 200
4281 a00df6s kar 120.196.100.99 538 103 200
4291 0a0fe2 pandora 120.196.100.99 615 415 200
4301 0sfs01 kar 120.196.100.99 676 228 200
4311 0a0fe2 kar 120.196.100.99 231 94 200
4321 0sfs01 kar 120.196.100.99 515 756 200
4331 a00df6s kar 120.196.100.99 885 884 200
4341 adfd00fd5 pandora 120.196.100.99 843 262 200
4351 0a0fe2 kar 120.196.100.99 24 326 200
4361 a00df6s pandora 120.196.100.99 558 904 200
4371 00fdaf3 pandora 120.196.100.99 128 454 200
4381 adfd00fd5 pandora 120.196.100.99 499 501 200
4391 00wersa4 kar 120.196.100.99 49 38 200
4401 adfd00fd5 pandora 120.196.100.99 9 432 200
4411 00wersa4 kar 120.196.100.99 174 138 200
4421 adfd00fd5 kar 120.196.100.99 138 150 200
4431 0a0fe2 pandora 120.196.100.99 151 130 200
4441 a00df6s kar 120.196.100.99 907 896 200
4451 00fdaf3 pandora 120.196.100.99 165 892 200
4461 0sfs01 kar 120.196.100.99 816 326 200
4471 adfd00fd5 kar 120.196.100.99 483 461 200
4481 adfd00fd5 kar 120.196.100.99 810 276 200
4491 00fdaf3 kar 120.196.100.99 253 601 200
4501 0ad0s7 pandora 120.196.100.99 634 176 200
4511 0a0fe2 pandora 120.196.100.99 377 710 200
4521 0a0fe2 kar 120.196.100.99 5 673 200
4531 0a0fe2 pandora 120.196.100.99 290 837 200
4541 0a0fe2 pandora 120.196.100.99 623 465 200
4551 0a0fe2 pandora 120.196.100.99 362 568 200
4561 0a0fe2 kar 120.196.100.99 866 508 200
4571 00fdaf3 kar 120.196.100.99 965 173 200
4581 00wersa4 pandora 120.196.100.99 174 670 200
4591 a00df6s pandora 120.196.100.99 578 377 200
4601 0a0fe2 pandora 120.196.100.99 752 65 200
4611 adfd00fd5 pandora 120.196.100.99 131 564 200
4621 00wersa4 kar 120.196.100.99 349 756 200
4631 00wersa4 pandora 120.196.100.99 33 291 200
4641 0a0fe2 pandora 120.196.100.99 749 316 200
4651 0a0fe2 pandora 120.196.100.99 242 505 200
4661 a00df6s pandora 120.196.100.99 930 486 200
4671 0a0fe2 pandora 120.196.100.99 445 916 200
4681 00wersa4 kar 120.196.100.99 584 154 200
4691 a00df6s pandora 120.196.100.99 35 62 200
4701 a00df6s pandora 120.196.100.99 892 633 200
4711 00wersa4 pandora 120.196.100.99 445 889 200
4721 adfd00fd5 pandora 120.196.100.99 543 664 200
4731 0a0fe2 pandora 120.196.100.99 850 497 200
4741 0sfs01 kar 120.196.100.99 626 691 200
4751 0ad0s7 kar 120.196.100.99 895 886 200
4761 0a0fe2 pandora 120.196.100.99 601 106 200
4771 0a0fe2 pandora 120.196.100.99 718 638 200
4781 a00df6s pandora 120.196.100.99 538 158 200
4791 0ad0s7 kar 120.196.100.99 514 663 200
4801 0ad0s7 pandora 120.196.100.99 605 996 200
4811 0ad0s7 kar 120.196.100.99 695 234 200
4821 0ad0s7 pandora 120.196.100.99 307 784 200
4831 00fdaf3 kar 120.196.100.99 844 128 200
4841 0a0fe2 kar 120.196.100.99 352 963 200
4851 a00df6s pandora 120.196.100.99 690 641 200
4861 0a0fe2 kar 120.196.100.99 690 878 200
4871 00fdaf3 kar 120.196.100.99 179 400 200
4881 0ad0s7 kar 120.196.100.99 593 588 200
4891 0ad0s7 pandora 120.196.100.99 405 888 200
4901 00wersa4 pandora 120.196.100.99 783 611 200
4911 00fdaf3 pandora 120.196.100.99 711 793 200
4921 a00df6s kar 120.196.100.99 95 603 200
4931 0a0fe2 kar 120.196.100.99 145 707 200
4941 0a0fe2 pandora 120.196.100.99 480 597 200
4951 0ad0s7 kar 120.196.100.99 693 288 200
4961 00fdaf3 pandora 120.196.100.99 249 556 200
4971 00wersa4 pandora 120.196.100.99 234 822 200
4981 0a0fe2 kar 120.196.100.99 868 847 200
4991 00wersa4 kar 120.196.100.99 885 707 200
2、编写MapReduce程序
（1）创建SpeakBean对象
package com.lagou.speak;import org.apache.hadoop.io.Writable;import java.io.DataInput;
import java.io.DataOutput;
import java.io.IOException;// 这个类型是map输出kv中value的类型，需要实现writable序列化接口
public class SpeakBean implements Writable {// 定义属性private Long selfDuration; // 自由内容时长private Long thirdPartDuration; // 第三方内容时长private String deviceId; // 设备IDprivate Long sumDuration; // 总时长// 空参构造public SpeakBean() {}// 有参构造public SpeakBean(Long selfDuration, Long thirdPartDuration, String deviceId) {this.selfDuration = selfDuration;this.thirdPartDuration = thirdPartDuration;this.deviceId = deviceId;this.sumDuration = this.selfDuration + this.thirdPartDuration;}// 序列化方法：就是把内容输出到网络或者文本中@Overridepublic void write(DataOutput dataOutput) throws IOException {dataOutput.writeLong(selfDuration);// 自由内容时长dataOutput.writeLong(thirdPartDuration);// 第三方内容时长dataOutput.writeUTF(deviceId);// 设备IDdataOutput.writeLong(sumDuration);// 总时长}// 反序列化方法@Overridepublic void readFields(DataInput dataInput) throws IOException {this.selfDuration = dataInput.readLong();this.thirdPartDuration = dataInput.readLong();this.deviceId = dataInput.readUTF();this.sumDuration = dataInput.readLong();}public Long getSelfDuration() {return selfDuration;}public void setSelfDuration(Long selfDuration) {this.selfDuration = selfDuration;}public Long getThirdPartDuration() {return thirdPartDuration;}public void setThirdPartDuration(Long thirdPartDuration) {this.thirdPartDuration = thirdPartDuration;}public String getDeviceId() {return deviceId;}public void setDeviceId(String deviceId) {this.deviceId = deviceId;}public Long getSumDuration() {return sumDuration;}public void setSumDuration(Long sumDuration) {this.sumDuration = sumDuration;}// 方便观察数据@Overridepublic String toString() {return selfDuration + "\t" + thirdPartDuration + "\t" + deviceId + "\t" + sumDuration;}
}
（2）编写Mapper类
package com.lagou.speak;import org.apache.hadoop.io.*;
import org.apache.hadoop.mapreduce.Mapper;import java.io.IOException;// 四个参数：分为两队KV/*** 第一对kv：map输入参数的kv类型; k-->一行文本偏移量，v-->一行文本内容* 第二队kv：map输出参数kv类型; k-->map输出的key类型，value:map输出的value类型*/
public class SpeakMapper extends Mapper<LongWritable, Text, Text, SpeakBean> {/*** 转换接收到的text数据为String* 按照制表符进行切分：得到自有内容时长，第三方内容时长，设备id，封装为SpeakBean* 直接输出：k-->设备id，value-->speakBane*/Text device_id = new Text();@Overrideprotected void map(LongWritable key, Text value, Mapper<LongWritable, Text, Text, SpeakBean>.Context context) throws IOException, InterruptedException {// 转换接收到的text数据为StringString line = value.toString();String[] fields = line.split("\t");// 自有内容时长String selfDuration = fields[fields.length - 3];// 第三方内容时长String thirdPartDuration = fields[fields.length - 2];// 设备idString deviceId = fields[1];SpeakBean bean = new SpeakBean(Long.parseLong(selfDuration), Long.parseLong(thirdPartDuration), deviceId);// 直接输出: k-->设备id, value -->speakBeandevice_id.set(deviceId);context.write(device_id, bean);}
}
（3）编写Reducer
package com.lagou.speak;import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Reducer;import java.io.IOException;public class SpeakReducer extends Reducer<Text, SpeakBean, Text, SpeakBean> {@Overrideprotected void reduce(Text key, Iterable<SpeakBean> values, Reducer<Text, SpeakBean, Text, SpeakBean>.Context context) throws IOException, InterruptedException {// 定义时长累加的初始值（这是局部变量，只对当前该方法起作用，用完就销毁了）Long self_duration = 0L;Long third_part_duration = 0L;// reduce 方法的key：map输出的是某一个key// reduce方法的value：map输出的kv对中相同key的value组成的一个集合// reduce逻辑：遍历迭代器累加时长即可for (SpeakBean bean : values) {Long selfDuration = bean.getSelfDuration();Long thirdPartDuration = bean.getThirdPartDuration();self_duration += selfDuration;third_part_duration += thirdPartDuration;}// 输出，封装成一个bean对象输出SpeakBean bean = new SpeakBean(self_duration, third_part_duration, key.toString());context.write(key, bean);}
}
（4）编写驱动
package com.lagou.speak;import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.*;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;import java.io.IOException;public class SpeakDriver {public static void main(String[] args) throws IOException, InterruptedException, ClassNotFoundException {Configuration conf = new Configuration();Job job = Job.getInstance(conf, "speakDriver");// 设置jar包本地路径job.setJarByClass(SpeakDriver.class);// 使用mapper和reducerjob.setMapperClass(SpeakMapper.class);job.setReducerClass(SpeakReducer.class);// 设置map输出kv类型job.setMapOutputKeyClass(Text.class);job.setMapOutputValueClass(SpeakBean.class);// 设置reduce输出job.setOutputKeyClass(Text.class);job.setOutputValueClass(SpeakBean.class);// 读取数据路径FileInputFormat.setInputPaths(job, new Path(args[0]));FileOutputFormat.setOutputPath(job, new Path(args[1]));// 提交任务boolean flag = job.waitForCompletion(true);System.exit(flag ? 0 : 1);}
}
注意：本次所用的依赖与上篇博文依赖相同，故省略。
MapReduce编程规范及示例编写_悠然予夏的博客-CSDN博客
序列化Writable接口相关推荐
- mysql xa 序列化_MYSQL 的数据序列化Api接口详情
MYSQL 的数据序列化Api接口详情 发布时间:2020-04-29 17:02:59 来源:亿速云 阅读:158 作者:三月 本文主要给大家介绍MYSQL 的数据序列化Api接口详情,文章内容都是 ...
- 理解Java对象序列化——Serializable接口
概述:当一个类实现了Serializable接口(该接口仅为标记接口,不包含任何方法定义),表示该类可以序列化.序列化的目的是将一个实现了Serializable接口的对象转换成一个字节序列,可以. ...
- 序列化Serializable接口
一.序列化 1.什么是序列化? 序列化就是将对象的状态存储到特定存储介质中的过程,也就是将对象状态转换为可保持或传输格式的过程. 在序列化过程中,会将对象的公有成员.私有成员(包括类名),转换为字节流 ...
- 编程进阶：Java小白的序列化Serializable接口
在之前的学习过程中,我们知道了如何使用FileInputStream输入流和FileOutputStream输出流编写程序读写文件. 下面我们来学习一下如何使用序列化和反序列化读写文件. 一.序列化 ...
- protobuf入门教程(三)：常用序列化/反序列化接口
C数组的序列化和反序列化 #include "addressbook.pb.h" #include <iostream> using namespace std;/* ...
- 数据分析大数据面试题大杂烩01
互联网:通过埋点实时计算用户浏览频次用优惠券等措施吸引用户,通过历史信息用非智能学习的title方式构造用户画像(抖音,京东) 电信,银行统计营收和针对用户的个人画像:处理大量非实时数据 政府:健康码 ...
- Hadoop基础部分-2
第五部分 HDFS分布式文件系统 第一节 HDFS 简介 HDFS (全称:Hadoop Distribute File System,Hadoop 分布式文件系统)是 Hadoop 核心组成,是分布 ...
- MepReduce-开启大数据计算之门
Hadoop MapReduce是一种编程模型,用于大规模数据集(大于1TB)的并行运算.早期的MapReduce(MR)框架简单明了,JobTracker作为MR框架的集中处理点,随着分布式系统集群 ...
- Hadoop详解(三)——MapReduce原理和执行过程，远程Debug，Writable序列化接口，MapReduce程序编写
MapReduce概述 MapReduce是一种分布式计算模型,由Google提出,主要用于搜索领域,解决海量数据的计算问题. MR由两个阶段组成:Map和Reduce,用户只需要实现map()和Re ...
最新文章
- NSwagStudio for Swagger Api
- 分享Kali Linux 2017.1镜像
- Python csv模块的使用
- step4 . day2标准IO和文件IO 小测试demo
- html帧内容,html将内容从一个帧移动到另一个帧
- Web Broadcast Channel
- JQuery中button提交表单报TypeError: elem[type] is not a function jquery
- python lambda表达式及用法_Python：lambda表达式和yield关键字理解与使用讲解
- 前端开发的难点到底在什么地方?
- oracle 11g 备份导入12c,关于12C版本导出导入11G版本的这点事
- C++ 单例模式 模版类(Meyers‘ Singleton)
- cad字体安装_为什么CAD图纸打开后会显示很多问号“？？？”，该怎么解决
- springboot发送qq邮件详细步骤
- 【翻译】LearnYouSomeErlangForGreatGood（一）:导言
- Web 前端必会的 PhotoShop 技能
- ENVOY的首个 NFT 发行版“Decentraboard”数小时内售罄
- Linux0.11内核--进程的调度(就绪态和运行态之间的转换)
- Domain Adaptation for Object Detection using SE Adaptors and Center Loss 论文翻译
- Linux tar命令详解
- Openstack云平台的搭建与部署（具体实验过程截图评论拿）
热门文章
- cad字体 草体_50种免费的草书和手写字体，丰富您的设计
- 发现中国统计年鉴好资源网站
- python3 + selenium 进行强智教务成绩的刷新以及通知成绩（华东理工大学 某ecust）
- C++编程小技巧（纯干货）
- 百度地图可视化之实现散点图
- 金融统计分析python论文_Python量化投资远程班
- 【Vue】疫情实时数据及地图 Vue + Echarts + vxeTable
- msde2000修改服务器名,msde2000数据库服务器
- 【踩坑系列】 SpringBoot ，SpringSecurity，activiti 项目无法正常跳转到登录界面
- python 3 编程实心三角形和空心三角形