安装librdkafka和使用C语言操作kafka的范例
C/C++操作kafka
- 一、安装librdkafka
- 二、开启kafka相关服务
- 2.1、启动zookeeper
- 2.2、启动Kafka
- 2.3、创建topic
- 三、c语言操作kafka的范例
- 3.1、消费者
- 3.2、生产者
- 3.3、生产者和消费者的交互
- 总结
- 后言
一、安装librdkafka
git clone https://github.com/edenhill/librdkafka.git
cd librdkafka
git checkout v1.7.0
./configure
make
sudo make install
sudo ldconfig
在librdkafka的examples目录下会有示例程序。比如consumer的启动需要下列参数
% Usage: ./consumer <broker> <group.id> <topic1> <topic2>..
指定broker、group id、topic(可以订阅多个)。示例:
./consumer localhost:9092 0 test
缩略语介绍:
缩略语 | 缩略语全称 | 示例或说明 |
---|---|---|
rd | Rapid Development | rd.h |
rk | RdKafka | |
toppar | Topic Partition | struct rd_kafka_toppar_t { }; |
rep | Reply | struct rd_kafka_t { rd_kafka_q_t *rk_rep }; |
msgq | Message Queue | struct rd_kafka_msgq_t { }; |
rkb | RdKafka Broker | Kafka代理 |
rko | RdKafka Operation | Kafka操作 |
rkm | RdKafka Message | Kafka消息 |
payload | 存在Kafka上的消息(或叫Log) |
二、开启kafka相关服务
2.1、启动zookeeper
启动zookeeper可以通过下面的脚本来启动zookeeper服务,当然,也可以自己独立搭建zookeeper的集群来实现。这里我们直接使用kafka自带的zookeeper。
cd bin/
# 前台运行:
sh zookeeper-server-start.sh ../config/zookeeper.properties# 后台运行:
sh zookeeper-server-start.sh -daemon ../config/zookeeper.properties
可以通过命令lsof -i:2181 查看zookeeper是否启动成功。
$ lsof -i:2181
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
java 74930 fly 96u IPv6 734467 0t0 TCP *:2181 (LISTEN)
2.2、启动Kafka
启动kafka(kafka安装路径的bin目录下执行),默认启动端口9092。
sh kafka-server-start.sh -daemon ../config/server.properties
2.3、创建topic
sh kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test
参数说明:
–create 是创建主题的的动作指令。
–zookeeper 指定kafka所连接的zookeeper服务地址。
–replicator-factor 指定了副本因子(即副本数量); 表示该topic需要在不同的broker中保存几份,这里设置成1,表示在两个broker中保存两份Partitions分区数。
–partitions 指定分区个数;多通道,类似车道。
–topic 指定所要创建主题的名称,比如test。
成功则显示:
Created topic "test".
三、c语言操作kafka的范例
3.1、消费者
在librdkafka\examples下有consumer.c文件,该文件是一个c语言操作kafka的代码范例,内容如下。
/*** Simple high-level balanced Apache Kafka consumer* using the Kafka driver from librdkafka* (https://github.com/edenhill/librdkafka)*/#include <stdio.h>
#include <signal.h>
#include <string.h>
#include <ctype.h>/* Typical include path would be <librdkafka/rdkafka.h>, but this program* is builtin from within the librdkafka source tree and thus differs. */
//#include <librdkafka/rdkafka.h>
#include "rdkafka.h"static volatile sig_atomic_t run = 1;/*** @brief Signal termination of program*/
static void stop (int sig) {run = 0;
}/*** @returns 1 if all bytes are printable, else 0.*/
static int is_printable (const char *buf, size_t size) {size_t i;for (i = 0 ; i < size ; i++)if (!isprint((int)buf[i]))return 0;return 1;
}int main (int argc, char **argv) {rd_kafka_t *rk; /* Consumer instance handle */rd_kafka_conf_t *conf; /* Temporary configuration object */rd_kafka_resp_err_t err; /* librdkafka API error code */char errstr[512]; /* librdkafka API error reporting buffer */const char *brokers; /* Argument: broker list */const char *groupid; /* Argument: Consumer group id */char **topics; /* Argument: list of topics to subscribe to */int topic_cnt; /* Number of topics to subscribe to */rd_kafka_topic_partition_list_t *subscription; /* Subscribed topics */int i;/** Argument validation*/if (argc < 4) {fprintf(stderr,"%% Usage: ""%s <broker> <group.id> <topic1> <topic2>..\n",argv[0]);return 1;}brokers = argv[1];groupid = argv[2];topics = &argv[3];topic_cnt = argc - 3;/** Create Kafka client configuration place-holder*/conf = rd_kafka_conf_new(); // 创建配置文件/* Set bootstrap broker(s) as a comma-separated list of* host or host:port (default port 9092).* librdkafka will use the bootstrap brokers to acquire the full* set of brokers from the cluster. */if (rd_kafka_conf_set(conf, "bootstrap.servers", brokers,errstr, sizeof(errstr)) != RD_KAFKA_CONF_OK) {fprintf(stderr, "%s\n", errstr);rd_kafka_conf_destroy(conf);return 1;}/* Set the consumer group id.* All consumers sharing the same group id will join the same* group, and the subscribed topic' partitions will be assigned* according to the partition.assignment.strategy* (consumer config property) to the consumers in the group. */if (rd_kafka_conf_set(conf, "group.id", groupid,errstr, sizeof(errstr)) != RD_KAFKA_CONF_OK) {fprintf(stderr, "%s\n", errstr);rd_kafka_conf_destroy(conf);return 1;}/* If there is no previously committed offset for a partition* the auto.offset.reset strategy will be used to decide where* in the partition to start fetching messages.* By setting this to earliest the consumer will read all messages* in the partition if there was no previously committed offset. */if (rd_kafka_conf_set(conf, "auto.offset.reset", "earliest",errstr, sizeof(errstr)) != RD_KAFKA_CONF_OK) {fprintf(stderr, "%s\n", errstr);rd_kafka_conf_destroy(conf);return 1;}/** Create consumer instance.** NOTE: rd_kafka_new() takes ownership of the conf object* and the application must not reference it again after* this call.*/// 创建一个kafka消费者rk = rd_kafka_new(RD_KAFKA_CONSUMER, conf, errstr, sizeof(errstr));if (!rk) {fprintf(stderr,"%% Failed to create new consumer: %s\n", errstr);return 1;}conf = NULL; /* Configuration object is now owned, and freed,* by the rd_kafka_t instance. *//* Redirect all messages from per-partition queues to* the main queue so that messages can be consumed with one* call from all assigned partitions.** The alternative is to poll the main queue (for events)* and each partition queue separately, which requires setting* up a rebalance callback and keeping track of the assignment:* but that is more complex and typically not recommended. */rd_kafka_poll_set_consumer(rk);// poll机制,设置消费者实例到poll中/* Convert the list of topics to a format suitable for librdkafka */// 创建主题分区列表subscription = rd_kafka_topic_partition_list_new(topic_cnt);for (i = 0 ; i < topic_cnt ; i++)rd_kafka_topic_partition_list_add(subscription,topics[i],/* the partition is ignored* by subscribe() */RD_KAFKA_PARTITION_UA);/* Subscribe to the list of topics */err = rd_kafka_subscribe(rk, subscription);if (err) {fprintf(stderr,"%% Failed to subscribe to %d topics: %s\n",subscription->cnt, rd_kafka_err2str(err));rd_kafka_topic_partition_list_destroy(subscription);rd_kafka_destroy(rk);return 1;}fprintf(stderr,"%% Subscribed to %d topic(s), ""waiting for rebalance and messages...\n",subscription->cnt);rd_kafka_topic_partition_list_destroy(subscription);/* Signal handler for clean shutdown */signal(SIGINT, stop);/* Subscribing to topics will trigger a group rebalance* which may take some time to finish, but there is no need* for the application to handle this idle period in a special way* since a rebalance may happen at any time.* Start polling for messages. */while (run) {rd_kafka_message_t *rkm;rkm = rd_kafka_consumer_poll(rk, 100);if (!rkm)continue; /* Timeout: no message within 100ms,* try again. This short timeout allows* checking for `run` at frequent intervals.*//* consumer_poll() will return either a proper message* or a consumer error (rkm->err is set). */if (rkm->err) {/* Consumer errors are generally to be considered* informational as the consumer will automatically* try to recover from all types of errors. */fprintf(stderr,"%% Consumer error: %s\n",rd_kafka_message_errstr(rkm));rd_kafka_message_destroy(rkm);continue;}/* Proper message. */printf("Message on %s [%"PRId32"] at offset %"PRId64":\n",rd_kafka_topic_name(rkm->rkt), rkm->partition,rkm->offset);/* Print the message key. */if (rkm->key && is_printable(rkm->key, rkm->key_len))printf(" Key: %.*s\n",(int)rkm->key_len, (const char *)rkm->key);else if (rkm->key)printf(" Key: (%d bytes)\n", (int)rkm->key_len);/* Print the message value/payload. */if (rkm->payload && is_printable(rkm->payload, rkm->len))printf(" Value: %.*s\n",(int)rkm->len, (const char *)rkm->payload);else if (rkm->payload)printf(" Value: (%d bytes)\n", (int)rkm->len);rd_kafka_message_destroy(rkm);}/* Close the consumer: commit final offsets and leave the group. */fprintf(stderr, "%% Closing consumer\n");rd_kafka_consumer_close(rk);/* Destroy the consumer */rd_kafka_destroy(rk);return 0;
}
函数调用 | 含义 |
---|---|
rd_kafka_conf_new(); | 创建配置文件 |
rd_kafka_conf_set(…) | 设置参数。可以设置broker、group id、auto.offset.reset等 |
rd_kafka_new(…) | 创建一个kafka消费者 |
rd_kafka_poll_set_consumer(…) | 设置到poll里面 |
rd_kafka_topic_partition_list_new(…) | 创建主题分区列表 |
rd_kafka_topic_partition_list_add(…) | 将主题添加到列表中,有订阅多个就添加多个 |
rd_kafka_subscribe(…) | 订阅主题 |
rd_kafka_consumer_poll(…) | 轮询数据,可以设置超时 |
3.2、生产者
在librdkafka\examples下有producer.c文件,该文件是一个c语言操作kafka的代码范例,内容如下。
/*** Simple Apache Kafka producer* using the Kafka driver from librdkafka* (https://github.com/edenhill/librdkafka)*/#include <stdio.h>
#include <signal.h>
#include <string.h>/* Typical include path would be <librdkafka/rdkafka.h>, but this program* is builtin from within the librdkafka source tree and thus differs. */
#include "rdkafka.h"static volatile sig_atomic_t run = 1;/*** @brief Signal termination of program*/
static void stop (int sig) {run = 0;fclose(stdin); /* abort fgets() */
}/*** @brief Message delivery report callback.** This callback is called exactly once per message, indicating if* the message was succesfully delivered* (rkmessage->err == RD_KAFKA_RESP_ERR_NO_ERROR) or permanently* failed delivery (rkmessage->err != RD_KAFKA_RESP_ERR_NO_ERROR).** The callback is triggered from rd_kafka_poll() and executes on* the application's thread.*/
static void dr_msg_cb (rd_kafka_t *rk,const rd_kafka_message_t *rkmessage, void *opaque) {if (rkmessage->err)fprintf(stderr, "%% Message delivery failed: %s\n",rd_kafka_err2str(rkmessage->err));elsefprintf(stderr,"%% Message delivered (%zd bytes, ""partition %"PRId32")\n",rkmessage->len, rkmessage->partition);/* The rkmessage is destroyed automatically by librdkafka */
}int main (int argc, char **argv) {rd_kafka_t *rk; /* Producer instance handle */rd_kafka_conf_t *conf; /* Temporary configuration object */char errstr[512]; /* librdkafka API error reporting buffer */char buf[512]; /* Message value temporary buffer */const char *brokers; /* Argument: broker list */const char *topic; /* Argument: topic to produce to *//** Argument validation*/if (argc != 3) {fprintf(stderr, "%% Usage: %s <broker> <topic>\n", argv[0]);return 1;}brokers = argv[1];topic = argv[2];/** Create Kafka client configuration place-holder*/conf = rd_kafka_conf_new();/* Set bootstrap broker(s) as a comma-separated list of* host or host:port (default port 9092).* librdkafka will use the bootstrap brokers to acquire the full* set of brokers from the cluster. */if (rd_kafka_conf_set(conf, "bootstrap.servers", brokers,errstr, sizeof(errstr)) != RD_KAFKA_CONF_OK) {fprintf(stderr, "%s\n", errstr);return 1;}/* Set the delivery report callback.* This callback will be called once per message to inform* the application if delivery succeeded or failed.* See dr_msg_cb() above.* The callback is only triggered from rd_kafka_poll() and* rd_kafka_flush(). */rd_kafka_conf_set_dr_msg_cb(conf, dr_msg_cb);/** Create producer instance.** NOTE: rd_kafka_new() takes ownership of the conf object* and the application must not reference it again after* this call.*/rk = rd_kafka_new(RD_KAFKA_PRODUCER, conf, errstr, sizeof(errstr));if (!rk) {fprintf(stderr,"%% Failed to create new producer: %s\n", errstr);return 1;}/* Signal handler for clean shutdown */signal(SIGINT, stop);fprintf(stderr,"%% Type some text and hit enter to produce message\n""%% Or just hit enter to only serve delivery reports\n""%% Press Ctrl-C or Ctrl-D to exit\n");while (run && fgets(buf, sizeof(buf), stdin)) {size_t len = strlen(buf);rd_kafka_resp_err_t err;if (buf[len-1] == '\n') /* Remove newline */buf[--len] = '\0';if (len == 0) {/* Empty line: only serve delivery reports */rd_kafka_poll(rk, 0/*non-blocking */);continue;}/** Send/Produce message.* This is an asynchronous call, on success it will only* enqueue the message on the internal producer queue.* The actual delivery attempts to the broker are handled* by background threads.* The previously registered delivery report callback* (dr_msg_cb) is used to signal back to the application* when the message has been delivered (or failed).*/retry:err = rd_kafka_producev(/* Producer handle */rk,/* Topic name */RD_KAFKA_V_TOPIC(topic),/* Make a copy of the payload. */RD_KAFKA_V_MSGFLAGS(RD_KAFKA_MSG_F_COPY),/* Message value and length */RD_KAFKA_V_VALUE(buf, len),/* Per-Message opaque, provided in* delivery report callback as* msg_opaque. */RD_KAFKA_V_OPAQUE(NULL),/* End sentinel */RD_KAFKA_V_END);if (err) {/** Failed to *enqueue* message for producing.*/fprintf(stderr,"%% Failed to produce to topic %s: %s\n",topic, rd_kafka_err2str(err));if (err == RD_KAFKA_RESP_ERR__QUEUE_FULL) {/* If the internal queue is full, wait for* messages to be delivered and then retry.* The internal queue represents both* messages to be sent and messages that have* been sent or failed, awaiting their* delivery report callback to be called.** The internal queue is limited by the* configuration property* queue.buffering.max.messages */rd_kafka_poll(rk, 1000/*block for max 1000ms*/);goto retry;}} else {fprintf(stderr, "%% Enqueued message (%zd bytes) ""for topic %s\n",len, topic);}/* A producer application should continually serve* the delivery report queue by calling rd_kafka_poll()* at frequent intervals.* Either put the poll call in your main loop, or in a* dedicated thread, or call it after every* rd_kafka_produce() call.* Just make sure that rd_kafka_poll() is still called* during periods where you are not producing any messages* to make sure previously produced messages have their* delivery report callback served (and any other callbacks* you register). */rd_kafka_poll(rk, 0/*non-blocking*/);}/* Wait for final messages to be delivered or fail.* rd_kafka_flush() is an abstraction over rd_kafka_poll() which* waits for all messages to be delivered. */fprintf(stderr, "%% Flushing final messages..\n");rd_kafka_flush(rk, 10*1000 /* wait for max 10 seconds */);/* If the output queue is still not empty there is an issue* with producing messages to the clusters. */if (rd_kafka_outq_len(rk) > 0)fprintf(stderr, "%% %d message(s) were not delivered\n",rd_kafka_outq_len(rk));/* Destroy the producer instance */rd_kafka_destroy(rk);return 0;
}
其操作流程和消费者是类似的。
函数调用 | 含义 |
---|---|
rd_kafka_conf_new(); | 创建配置文件 |
rd_kafka_conf_set(…) | 设置参数。设置bootstrap.servers |
rd_kafka_conf_set_dr_msg_cb(…) | 设置交付报告回调 |
rd_kafka_new(…) | 创建一个kafka生产者,RD_KAFKA_PRODUCER |
rd_kafka_producev(…) | 发送数据 |
3.3、生产者和消费者的交互
(1)启动消费者。
./consumer localhost:9092 0 test
显示:
% Subscribed to 1 topic(s), waiting for rebalance and messages...
(2)启动生产者。
./producer localhost:9092 test
显示
% Type some text and hit enter to produce message
% Or just hit enter to only serve delivery reports
% Press Ctrl-C or Ctrl-D to exit
(3)通信过程。
生产者发送hello:
$ ./producer localhost:9092 test
% Type some text and hit enter to produce message
% Or just hit enter to only serve delivery reports
% Press Ctrl-C or Ctrl-D to exit
hello consumer
% Enqueued message (14 bytes) for topic test
消费者接受:
$ ./consumer localhost:9092 0 test
% Subscribed to 1 topic(s), waiting for rebalance and messages...
Message on test [0] at offset 4:Value: hello consumer
总结
- 一个分区只能被一个消费者读取。如果一个topic只有一个分区,多个消费者读取时只有一个消费者能读到数据;单个分区开启多个消费者去读取数据是没有意义的。
- 熟悉消费者概念和C/C++编写生产消费。
后言
本专栏知识点是通过<零声教育>的系统学习,进行梳理总结写下文章,对c/c++linux系统提升感兴趣的读者,可以点击链接,详细查看详细的服务:C/C++服务器课程 。
安装librdkafka和使用C语言操作kafka的范例相关推荐
- linux安装mysql和使用c语言操作数据库的方法 c语言连接mysql
http://www.jb51.net/article/46139.htm 1. MySQL的安装与配置: 在Ubuntu下安装MySQL方法很简单,使用如下命令: 复制代码 代码如下: sudo a ...
- kafka实战教程(python操作kafka),kafka配置文件详解
全栈工程师开发手册 (作者:栾鹏) 架构系列文章 应用往Kafka写数据的原因有很多:用户行为分析.日志存储.异步通信等.多样化的使用场景带来了多样化的需求:消息是否能丢失?是否容忍重复?消息的吞吐量 ...
- R语言操作excel文件的数据
简述 令人惊讶的是,网上的教程关于r语言处理excel数据的都是在说如何读.. 下面以用xlsx这个包为例 安装这个包,并读取数据 [两行命令]R语言读取excel数据 操作数据 其实,导入进来之后, ...
- Python 操作 Kafka --- kafka-python
kafka-python:https://github.com/dpkp/kafka-python kafka-python 文档:https://kafka-python.readthedocs.i ...
- Linux C语言操作MySQL
1.MySQL数据库简介 MySQL是一个开源码的小型关系数据库管理系统,体积小,速度快,总体成本低,开源.MySQL有以下特性: (1) 使用C和C++编写,并使用了多种编译器进行测试,保证源码的可 ...
- SAP手记之六:GUI安装后初始配置(中文语言包安装)
SAP手记之六:GUI安装后初始配置(中文语言包安装) 在初始配置前先要了解的东西,这些东西在我们按照好sap客户端应用程序GUI的时候就可以看到: 元素一:client 如何理解clie ...
- c#连接kafka_c#操作kafka(上)搭建kafka环境
小伙伴们大家好,今天没有概念,也没有理论,仅仅和大家一起快速的在centos上搭建一下kafka的测试环境,测试环境嘛,不涉及集群什么的,仅仅是单节点的kafka,日后可以在这个基础上,进行集群的相关 ...
- Docker安装MoogoDB, 进入容器, mongo shell操作mongoDB
安装MoogoDB, 进入容器, mongo shell操作mongoDB [ 包含 Docker-Compose方式.普通方式 ] 文章目录 安装MoogoDB, 进入容器, mongo shell ...
- go 操作 kafka 实现发送和订阅
kafka 消息队列 kafka架构 安装kafka kafak依赖zookeeper 需要先启动zk(集群) zookeeper 启动 单节点启动kafka kafka配置文件 config/ser ...
最新文章
- 关于ios中编译ffmpeg0.9.2库
- 035_Breadcrumb面包屑
- [Qt教程] 第44篇 进阶(四)信号和槽
- 在C#中使用代理的方式触发事件 的简单习作
- Ball Dropping
- linux中时间片,能讲一下在Linux系统中时间片是怎么分配的还有优先级的具体算法是...
- 2个基础操作案例带你入门MySQL
- 对话MySQL之父:代码一次性完成才是优秀程序员
- Spring 创建代理类流程跟踪
- 丁可以组什么词_“一”可以组什么词?落语读书会学期总结
- linux修改vcf编码格式,飞翔vcf文件编码转换
- ArcGIS——计算几何——面积/周长禁用
- 关于面向接口编程,你真的弄懂了吗?
- 宿主机上docker0 Linux 网桥设备是怎么来的?
- Synaptics 蠕虫病毒
- 金融科技成为服贸会热议话题:数字化转型中如何保障金融安全
- 两张MD5值一样但实际不一样的图片
- hdu-4747(线段树)
- 第11节:Docker基本操做
- Universal Style Transfer via Feature Transforms (WCT,风格迁移,NIPS2017)
热门文章
- 小米5 android 耗电严重,小米MIUI 9.5曝耗电严重、闹钟失灵问题
- aspx文件的Page指令参数
- 太卷了, 某公司把自家运营多年的核心系统完全开源了....
- 同花顺python_焦炭期货上市_python同花顺股票实时数据
- 2020 Android Studio安装教程
- cpu使用率 htop显示_最全最强的htop使用详解
- 大数据CDH安装详细教程
- OPPO图片缩放效果
- 8.2 贪心策略例题---快速渡河问题(没看懂!)
- 一般的在线教育平台需具备哪些功能?