Flink：将Kafka中的数据作为Flink的数据源

最新推荐文章于 2024-05-23 10:57:27 发布

程序员无羡

最新推荐文章于 2024-05-23 10:57:27 发布

阅读量234

点赞数

CC 4.0 BY-SA版权

本文链接：https://blog.csdn.net/weixin_45427648/article/details/130181653

Java 同时被 3 个专栏收录

89 篇文章

订阅专栏

Kafka

56 篇文章

订阅专栏

Flink

27 篇文章

订阅专栏

文章介绍了如何启动Kafka服务器、创建主题及生产者，然后展示了如何使用Flink从Kafka读取数据进行流处理，最后将处理结果写回Kafka。主要涉及的工具有Flink、Kafka以及相关配置和Java代码示例。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

启动kafka

./kafka-server-start.sh -daemon ../config/server.properties

创建topic

bin/kafka-topics.sh --create --zookeepper 192.168.25.129:2181 --replication-factor 1 --partitions 1 --topic test

启动控制台kafka生产者

bin/kafka-console-producer.sh --broker-list 192.168.25.129:9092 --topic test

依赖

<!--flink核心包-->
<dependency>
    <groupId>org.apache.flink</groupId>
    <artifactId>flink-java</artifactId>
    <version>1.7.2</version>
</dependency>
<!--flink流处理包-->
<dependency>
    <groupId>org.apache.flink</groupId>
    <artifactId>flink-streaming-java_2.12</artifactId>
    <version>1.7.2</version>
    <scope>provided</scope>
</dependency>	
<dependency>
    <groupId>org.apache.flink</groupId>
    <artifactId>flink-connector-kafka_2.11</artifactId>
    <version>1.7.2</version>
</dependency>

代码

import org.apache.flink.api.common.functions.FlatMapFunction;
import org.apache.flink.api.common.serialization.SimpleStringSchema;
import org.apache.flink.api.java.tuple.Tuple2;
import org.apache.flink.streaming.api.datastream.DataStreamSource;
import org.apache.flink.streaming.api.datastream.SingleOutputStreamOperator;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
import org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumer;
import org.apache.flink.util.Collector;
 
import java.util.Properties;
 
public class StreamFromKafka {
    public static void main(String[] args) throws Exception {
 
        StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
        Properties properties = new Properties();
 
        properties.setProperty("bootstrap.servers","192.168.25.129:9092");
        // 参数一：主题名
        // 参数二：反序列化约束，以便于Flink决定如何反序列化从Kafka获得的数据
        // 参数三：消费者的一些配置信息
        FlinkKafkaConsumer<String> consumer = new FlinkKafkaConsumer<String>("test", new SimpleStringSchema(), properties);
        
        DataStreamSource<String> data = env.addSource(consumer);
        SingleOutputStreamOperator<Tuple2<String, Integer>> wordAndOne = data.flatMap(new FlatMapFunction<String, Tuple2<String, Integer>>() {
            public void flatMap(String s, Collector<Tuple2<String, Integer>> collector) throws Exception {
                for (String word : s.split(" ")) {
                    collector.collect(Tuple2.of(word, 1));
                }
            }
        });
        
        SingleOutputStreamOperator<Tuple2<String, Integer>> result = wordAndOne.keyBy(0).sum(1);
        result.print();
        env.execute();
    }
}

Flink：将Flink中的数据下沉到Kafka