大数据实验 实验七:Flink初级编程实践
实验环境:Windows 10 Oracle VM VirtualBox
虚拟机:cnetos 7
Hadoop 3.3
实验内容与完成情况:
1. 使用IntelliJ IDEA工具开发WordCount程序
在Linux操作系统中安装IntelliJ IDEA,然后使用IntelliJ IDEA工具开发WordCount程序,并打包成JAR包,提交到Flink中运行。
下载flink安装包后解压
启动flink,
程序原码
WordCountData
package WordCount; import org.apache.flink.api.java.DataSet; import org.apache.flink.api.java.ExecutionEnvironment; public class WordCountData { public static final String[] WORDS = new String[]{"To be, or not to be,--that is the question:--", "Whether \'tis nobler in the mind to suffer", "The slings and arrows of outrageous fortune", "Or to take arms against a sea of troubles,", "And by opposing end them?--To die,--to sleep,--", "No more; and by a sleep to say we end", "The heartache, and the thousand natural shocks", "That flesh is heir to,--\'tis a consummation", "Devoutly to be wish\'d. To die,--to sleep;--", "To sleep! perchance to dream:--ay, there\'s the rub;", "For in that sleep of death what dreams may come,", "When we have shuffled off this mortal coil,", "Must give us pause: there\'s the respect", "That makes calamity of so long life;", "For who would bear the whips and scorns of time,", "The oppressor\'s wrong, the proud man\'s contumely,", "The pangs of despis\'d love, the law\'s delay,", "The insolence of office, and the spurns", "That patient merit of the unworthy takes,", "When he himself might his quietus make", "With a bare bodkin? who would these fardels bear,", "To grunt and sweat under a weary life,", "But that the dread of something after death,--", "The undiscover\'d country, from whose bourn", "No traveller returns,--puzzles the will,", "And makes us rather bear those ills we have", "Than fly to others that we know not of?", "Thus conscience does make cowards of us all;", "And thus the native hue of resolution", "Is sicklied o\'er with the pale cast of thought;", "And enterprises of great pith and moment,", "With this regard, their currents turn awry,", "And lose the name of action.--Soft you now!", "The fair Ophelia!--Nymph, in thy orisons", "Be all my sins remember\'d."}; public WordCountData() { } public static DataSetgetDefaultTextLineDataset(ExecutionEnvironment env) { return env.fromElements(WORDS); } }
WordCountTokenizer
package WordCount; import org.apache.flink.api.common.functions.FlatMapFunction; import org.apache.flink.api.java.tuple.Tuple2; import org.apache.flink.util.Collector; public class WordCountTokenizer implements FlatMapFunction> { public void flatMap(String value, Collector > out) throws Exception { String[] tokens = value.toLowerCase().split("\\W+"); int len = tokens.length; for (int i = 0; i < len; i++) { String tmp = tokens[i]; if (tmp.length() > 0) { out.collect(new Tuple2 (tmp, Integer.valueOf(1))); } } } }
WordCount
package WordCount; import org.apache.flink.api.java.DataSet; import org.apache.flink.api.java.ExecutionEnvironment; import org.apache.flink.api.java.operators.AggregateOperator; import org.apache.flink.api.java.utils.ParameterTool; public class WordCount { public WordCount() { } public static void main(String[] args) throws Exception { ParameterTool params = ParameterTool.fromArgs(args); ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment(); env.getConfig().setGlobalJobParameters(params); Object text; //如果没有指定输入路径,则默认使用WordCountData中提供的数据 if (params.has("input")) { text = env.readTextFile(params.get("input")); } else { System.out.println("Executing WordCount example with default input data set."); System.out.println("Use -- input to specify file input."); text = WordCountData.getDefaultTextLineDataset(env); } AggregateOperator counts = ((DataSet) text).flatMap(new WordCountTokenizer()).groupBy(new int[]{0}).sum(1); //如果没有指定输出,则默认打印到控制台 if (params.has("output")) { counts.writeAsCsv(params.get("output"), "\n", " "); env.execute(); } else { System.out.println("Printing result to stdout. Use --output to specify output path."); counts.print(); } } }
运行成功
2. 数据流词频统计
使用Linux操作系统自带的NC程序模拟生成数据流,不断产生单词并发送出去。编写Fink程序对NC程序发来的单词进行实时处理,计算词频,并输出词频统计结果。要求首先在IntelliJ IDEA中开发和调试程序,然后打包成JAR包部署到Flink中运行。
编写程序
package WordCount; import org.apache.flink.api.common.functions.FlatMapFunction; import org.apache.flink.api.java.utils.ParameterTool; import org.apache.flink.streaming.api.datastream.DataStream; import org.apache.flink.streaming.api.datastream.DataStreamSource; import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment; import org.apache.flink.streaming.api.windowing.time.Time; import org.apache.flink.util.Collector; public class WordCount { public static void main(String[] args) throws Exception { //定义socket的端口号 int port; try { ParameterTool parameterTool = ParameterTool.fromArgs(args); port = parameterTool.getInt("port"); } catch (Exception e) { System.err.println("指定port参数,默认值为9000"); port = 9000; } //获取运行环境 StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment(); //连接socket获取输入的数据 DataStreamSourcetext = env.socketTextStream("127.0.0.1", port, "\n"); //计算数据 DataStream windowCount = text.flatMap(new FlatMapFunction () { public void flatMap(String value, Collector out) throws Exception { String[] splits = value.split("\\s"); for (String word : splits) { out.collect(new WordWithCount(word, 1L)); } } })//打平操作,把每行的单词转为 类型的数据 .keyBy("word")//针对相同的word数据进行分组 .timeWindow(Time.seconds(2), Time.seconds(1))//指定计算数据的窗口大小和滑动窗口大小 .sum("count"); //把数据打印到控制台 windowCount.print() .setParallelism(1);//使用一个并行度 //注意:因为flink是懒加载的,所以必须调用execute方法,上面的代码才会执行 env.execute("streaming word count"); } /** * 主要为了存储单词以及单词出现的次数 */ public static class WordWithCount { public String word; public long count; public WordWithCount() { } public WordWithCount(String word, long count) { this.word = word; this.count = count; } @Override public String toString() { return "WordWithCount{" + "word='" + word + '\'' + ", count=" + count + '}'; } }
运行包
向nc中输入数据
读取到数据
出现的问题
问题一
Flink进程已经启动但是用浏览器访问8081失败
问题二
运行打包的程序出现报错
问题三
使用linux自带的nc出现问题
没有找到命令
问题四
运行时出现找不到包的错误
问题五
运行时报错,缺少方法
问题六
在9000端口传入数据后,flink监听会报错
解决方案
问题一
配置的地址是localhost
将文档中的数据改为192.168.118.128:8081
问题解决
问题二
在maven项目中添加
reference.conf
问题解决
问题三
yum install -y nc 安装nc
问题解决
问题四
在打包时将flink/lib下的文件也打入包中
问题解决
问题五
使用flink的依赖包时需要严格保证包的版本号与启动的flink完全一致
不然就会出现缺少方法的报错
启动成功问题解决
问题六
在使用老版的窗口时,未指定时间语义,导致报错.
需要设置时间语义
问题解决
猜你喜欢
- 14天前(希尔顿2021活动)希尔顿集团618盛夏大促开启
- 14天前(零碳中国·绿色投资蓝皮书)中国"零碳"差旅之路暨"绿色低碳酒店"标准研究项目成果发布会召开
- 14天前(三亚海棠湾君悦度假酒店)三亚海棠湾君悦酒店暑期夏令营悦趣海岛游招募中
- 14天前(瑞士大酒店-自助餐怎么样)瑞意心旅,以食为先 瑞士酒店开启全新"瑞士早餐计划"
- 14天前(七尚酒店百度百科)Lohkah七尚酒店首度开创充满新知的闽地研学旅程
- 14天前(甘州区文化旅游局)2025甘津文旅资源对接推介会在兰州举办
- 14天前(2025年安徽省“百场黄梅唱响百家景区”示范展示启动)2025年安徽省“百场黄梅唱响百家景区”示范展示启动
- 14天前(辽宁新增6个国家4a级旅游景区有哪些)辽宁新增6个国家4A级旅游景区
- 14天前(“三天跨两城”催生租车新需求,神州租车清明跨城订单同比增长416%)“三天跨两城”催生租车新需求,神州租车清明跨城订单同比增长416%
- 14天前(阿斯塔纳航空属于哪个联盟)阿斯塔纳航空荣获Skytrax世界航空公司大奖,将继续助力中哈交流往来
网友评论
- 搜索
- 最新文章
- (2020广州车展哈弗)你的猛龙 独一无二 哈弗猛龙广州车展闪耀登场
- (哈弗新能源suv2019款)智能科技颠覆出行体验 哈弗重塑新能源越野SUV价值认知
- (2021款全新哈弗h5自动四驱报价)新哈弗H5再赴保障之旅,无惧冰雪护航哈弗全民电四驱挑战赛
- (海南航空现况怎样)用一场直播找到市场扩张新渠道,海南航空做对了什么?
- (visa jcb 日本)优惠面面俱到 JCB信用卡邀您畅玩日本冰雪季
- (第三届“堡里有年味·回村过大年”民俗花灯会活动)第三届“堡里有年味·回村过大年”民俗花灯会活动
- (展示非遗魅力 长安启源助力铜梁龙舞出征)展示非遗魅力 长安启源助力铜梁龙舞出征
- (阿斯塔纳航空公司)阿斯塔纳航空机队飞机数量增至50架
- (北京香港航班动态查询)香港快运航空北京大兴新航线今日首航
- (我在港航“呵护”飞机 每一次安全着陆就是最好的荣誉)我在港航“呵护”飞机 每一次安全着陆就是最好的荣誉
- 热门文章