kafka集群安装和使用
kafka
(1)kafka是一个分布式的消息缓存系统
(2)kafka集群中的服务器都叫做broker
(3)kafka有两类客户端,一个叫做producer(消息生产者),一类叫做consumer(消息消费者),客户端和broker服务器之间采用TCP协议连接
(4)kafka中的消息可以通过topic进行区分,而且每一个消息topic都会被分区,以分担消息服务器的负载
(5)每一个分区都可以有多个副本,以防止数据的丢失
(6)某一个分区中的数据如果需要更新,都必须通知该分区所有副本中的leader来更新
(7)消费者可以分组,比如有两个消费者组A和B,共同消费一个topic:order_info,A和B所消费的消息不会重复,如order_info中有100个消息,每个消息都有一个id,编号从1-99,那么如果A组消费从0-49,B组消费就从50-99,当然不一定都是连续的
(8)消费者在具体消费某个topic中的消息时,可以制定起始偏移量
集群安装
官网教程
http://kafka.apache.org/22/documentation.html#introduction
1.解压
2.修改server.properties
broker.id=1
zookeeper.connect=hadoop01:2182,hadoop02:2182,hadoop03:2182
3.将zookeeper集群启动
4.在每一台节点上启动broker
bin/kafka-server-start.sh config/server.properties
#5.自带zookeeper,用于单节点,一般集群不用
#bin/zookeeper-server-start.sh config/zookeeper.proterties
开始安装
1.下载解压
[linyouyi@hadoop01 software]$ wget https://mirrors.aliyun.com/apache/kafka/2.2.0/kafka_2.11-2.2.0.tgz [linyouyi@hadoop01 software]$ tar -zxvf kafka_2.11-2.2.0.tgz -C /hadoop/module/
2.修改配置文件
[linyouyi@hadoop01 software]$ cd /hadoop/module/ [linyouyi@hadoop01 module]$ ll total 24 drwxrwxr-x 18 linyouyi linyouyi 4096 Aug 12 21:24 apache-storm-2.0.0 drwxr-xr-x 12 linyouyi linyouyi 4096 Aug 9 22:51 hadoop-2.7.7 drwxrwxr-x 7 linyouyi linyouyi 4096 Aug 11 12:10 hbase-2.0.5 drwxr-xr-x 7 linyouyi linyouyi 4096 Jul 22 2017 jdk1.8.0_144 drwxr-xr-x 6 linyouyi linyouyi 4096 Mar 10 03:47 kafka_2.11-2.2.0 drwxr-xr-x 15 linyouyi linyouyi 4096 Aug 8 11:03 zookeeper-3.4.14 [linyouyi@hadoop01 kafka_2.11-2.2.0]$ vim config/server.properties broker.id=1 log.dirs=/hadoop/kafka_2.11-2.2.0/log/kafka-logs zookeeper.connect=hadoop01:2182,hadoop02:2182,hadoop03:2182
3.拷贝到其他节点
[linyouyi@hadoop01 kafka_2.11-2.2.0]$ cd ../ [linyouyi@hadoop01 module]$ scp -r kafka_2.11-2.2.0/ linyouyi@hadoop02:/hadoop/module/ [linyouyi@hadoop01 module]$ scp -r kafka_2.11-2.2.0/ linyouyi@hadoop03:/hadoop/module/
4.修改另外两台的broker.id
[linyouyi@hadoop02 kafka_2.11-2.2.0]$ vim config/server.properties broker.id=2 [linyouyi@hadoop03 kafka_2.11-2.2.0]$ vim config/server.properties broker.id=3
5.启动kafka
[linyouyi@hadoop01 kafka_2.11-2.2.0]$ bin/kafka-server-start.sh config/server.properties & [linyouyi@hadoop01 kafka_2.11-2.2.0]$ jps 109904 Jps 44057 QuorumPeerMain 109503 Kafka [linyouyi@hadoop02 kafka_2.11-2.2.0]$ bin/kafka-server-start.sh config/server.properties & [linyouyi@hadoop02 kafka_2.11-2.2.0]$ jps 21308 Jps 20925 Kafka 34879 QuorumPeerMain [linyouyi@hadoop03 kafka_2.11-2.2.0]$ bin/kafka-server-start.sh config/server.properties & [linyouyi@hadoop03 kafka_2.11-2.2.0]$ jps 119587 QuorumPeerMain 37651 Jps 37269 Kafka
使用
1.创建话题
//hadoop01:9092,hadoop02:9092,hadoop03:9092都行
[linyouyi@hadoop03 kafka_2.11-2.2.0]$ bin/kafka-topics.sh --create --bootstrap-server hadoop01:9092 --replication-factor 3 --partitions 1 --topic linyouyi [2019-08-17 15:30:34,616] INFO [ReplicaFetcherManager on broker 3] Removed fetcher for partitions Set(linyouyi-0) (kafka.server.ReplicaFetcherManager) [2019-08-17 15:30:34,649] INFO [Log partition=linyouyi-0, dir=/tmp/kafka-logs] Loading producer state till offset 0 with message format version 2 (kafka.log.Log) [2019-08-17 15:30:34,653] INFO [Log partition=linyouyi-0, dir=/tmp/kafka-logs] Completed load of log with 1 segments, log start offset 0 and log end offset 0 in 25 ms (kafka.log.Log) [2019-08-17 15:30:34,655] INFO Created log for partition linyouyi-0 in /tmp/kafka-logs with properties {compression.type -> producer, message.format.version -> 2.2-IV1, file.delete.delay.ms -> 60000, max.message.bytes -> 1000012, min.compaction.lag.ms -> 0, message.timestamp.type -> CreateTime, message.downconversion.enable -> true, min.insync.replicas -> 1, segment.jitter.ms -> 0, preallocate -> false, min.cleanable.dirty.ratio -> 0.5, index.interval.bytes -> 4096, unclean.leader.election.enable -> false, retention.bytes -> -1, delete.retention.ms -> 86400000, cleanup.policy -> [delete], flush.ms -> 9223372036854775807, segment.ms -> 604800000, segment.bytes -> 1073741824, retention.ms -> 604800000, message.timestamp.difference.max.ms -> 9223372036854775807, segment.index.bytes -> 10485760, flush.messages -> 9223372036854775807}. (kafka.log.LogManager) [2019-08-17 15:30:34,656] INFO [Partition linyouyi-0 broker=3] No checkpointed highwatermark is found for partition linyouyi-0 (kafka.cluster.Partition) [2019-08-17 15:30:34,658] INFO Replica loaded for partition linyouyi-0 with initial high watermark 0 (kafka.cluster.Replica) [2019-08-17 15:30:34,658] INFO Replica loaded for partition linyouyi-0 with initial high watermark 0 (kafka.cluster.Replica) [2019-08-17 15:30:34,658] INFO Replica loaded for partition linyouyi-0 with initial high watermark 0 (kafka.cluster.Replica) [2019-08-17 15:30:34,660] INFO [Partition linyouyi-0 broker=3] linyouyi-0 starts at Leader Epoch 0 from offset 0. Previous Leader Epoch was: -1 (kafka.cluster.Partition) [linyouyi@hadoop03 kafka_2.11-2.2.0]$ bin/kafka-topics.sh --list --bootstrap-server localhost:9092 linyouyi [linyouyi@hadoop03 kafka_2.11-2.2.0]$ bin/kafka-topics.sh --list --bootstrap-server hadoop01:9092 linyouyi [linyouyi@hadoop03 kafka_2.11-2.2.0]$ bin/kafka-topics.sh --list --zookeeper hadoop01:2181 linyouyi [linyouyi@hadoop03 kafka_2.11-2.2.0]$ bin/kafka-topics.sh --create --bootstrap-server hadoop01:9092 --replication-factor 3 --partitions 1 --topic youyi [linyouyi@hadoop03 kafka_2.11-2.2.0]$ bin/kafka-topics.sh --list --zookeeper hadoop01:2181 linyouyi youyi
2.生产者往话题里面写消息
[linyouyi@hadoop03 kafka_2.11-2.2.0]$ bin/kafka-console-producer.sh --broker-list localhost:9092 --topic linyouyi>This is a message >This is a message >This is another message
3.消费者消费消息
[linyouyi@hadoop02 kafka_2.11-2.2.0]$ bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic linyouyi --from-beginning This is a message This is another message
4.接着在生产者继续写话题,消费者立马就消费了
5.查看topic linyouyi总体情况
[linyouyi@hadoop01 kafka_2.11-2.2.0]$ bin/kafka-topics.sh --describe --bootstrap-server localhost:9092 --topic linyouyi Topic:linyouyi PartitionCount:1 ReplicationFactor:3 Configs:segment.bytes=1073741824Topic: linyouyi Partition: 0 Leader: 3 Replicas: 3,1,2 Isr: 3,1,2
[linyouyi@hadoop01 kafka_2.11-2.2.0]$ bin/kafka-topics.sh --describe --bootstrap-server localhost:9092 --topic youyi Topic:youyi PartitionCount:1 ReplicationFactor:3 Configs:segment.bytes=1073741824Topic: youyi Partition: 0 Leader: 3 Replicas: 3,1,2 Isr: 3,1,2
转载于:https://www.cnblogs.com/linyouyi/p/11369015.html
kafka集群安装和使用相关推荐
- KafKa集群安装、配置
一.事前准备 1.kafka官网:http://kafka.apache.org/downloads. 2.选择使用版本下载. 3.kafka集群环境准备:(linux) 192.168.145.12 ...
- zookeeper+kafka集群安装之中的一个
zookeeper+kafka集群安装之中的一个 准备3台虚拟机, 系统是RHEL64服务版. 1) 每台机器配置例如以下: $ cat /etc/hosts ... # zookeeper host ...
- Kafka集群安装Version1.0.1(自带Zookeeper)
1.说明 Kafka集群安装,基于版本1.0.1, 使用kafka_2.12-1.0.1.tgz安装包, 其中2.12是编译工具Scala的版本. 而且不需要另外安装Zookeeper服务, 使用Ka ...
- Kafka集群安装Version2.10
Kafka集群安装,基于版本2.10, 使用kafka_2.10-0.10.1.0.tgz安装包. 1.安装规划 Storm集群模式,安装到下面三台机器 IP Hostname 10.43.159.2 ...
- Kafka 集群安装
一.集群准备 node1 192.168.157.128 node1 192.168.157.129 node1 192.168.157.130 二.安装zookeeper集群 Zookeeper 集 ...
- centos7搭建Linux集群,CentOS 7下Kafka集群安装
一.环境操作系统和软件版本介绍 1.环境操作系统为CentOS Linux release 7.2.1511 (Core) 可用cat /etc/RedHat-release查询 2.软件版本 Kaf ...
- Kafka集群安装--测试--关闭
一.前提 1.kafka安装包下载:http://kafka.apache.org/downloads 2.jdk已安装 3.scala已安装 4.zookeeper集群已安装并运行 二.步骤 1.对 ...
- kafka集群安装部署
Apache Kafka 是一个高吞吐量分布式消息系统,由LinkedIn开源.引用官网对kafka的介绍:"Apache Kafka is publish-subscribe messag ...
- 多易教育KAFKA实战(1)-KAFKA集群安装和shell客户端
注意kafka的安装需要依赖Zookeeper集群 ,所以安装kafka之前先安装zookeeper! zookeeper安装 上传安装包 解压 tar -zxvf zookeeper-3.4.6.t ...
- win10下kafka集群安装+集成springboot
kafka安装+集成springboot 记录kafka安装.学习.继承springboot的过程 文章目录 kafka安装+集成springboot 前言 一.kafka + zk的安装 1.zk的 ...
最新文章
- 使用JFreeChart绘制散点图
- linux中的查找文件夹
- KDD 18 论文解读 | GraphWave:一种全新的无监督网络嵌入方法
- 在 Azure VM 上使用 Jitsi 搭建私人视频会议
- mysql 触发器定义变量_MySQL 函数存储过程触发器定义简单示例
- iOS CocoaPods自动管理第三方开源库
- php disable-ipv6,強迫 PHP Curl 使用 IPv4 , 忽略 IPv6
- GeoMAN:多层Attention网络用于地理传感器的时序性预测
- HTML-W3school首页制作
- bat调用ssis package
- 每周论文精读04——A Survey on 3D Hand Pose Estimation: Cameras, Methods, and Datasets
- 31个全网最常用python实现(体系学习,学完显著提高代码复用能力)
- 会议室录播方案及录播设备推荐
- 蜗牛学院:这6种心态,容易毁掉你的职场
- v-if,v-else-if, v-else的实际使用
- NOIP 2012 国王游戏
- Event事件-6:new Event自定义事件 / addEvent绑定自定义 / trigger触发自定义 / removeEvent移除自定义
- 51单片机开发环境搭建 - VS Code 从编写到烧录
- nginx代理离线瓦片,并在leaflet中调用
- Opencv交通标志识别