idea用maven开发spark程序的pom.xml
2024-07-01 14:53:45
我们用maven构建spark项目，需要使用maven来管理各个组件jar
需要找依赖的话，地址：https://mvnrepository.com/
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd"><modelVersion>4.0.0</modelVersion><groupId>com.cfqq</groupId><artifactId>SparkProject</artifactId><version>1.0</version><inceptionYear>2008</inceptionYear><properties><project.build.sourceEncoding>UTF-8</project.build.sourceEncoding><project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding><!-- 配置以下可以解决 在jdk1.8环境下打包时报错 “-source 1.5 中不支持 lambda 表达式” --><maven.compiler.source>1.8</maven.compiler.source><maven.compiler.target>1.8</maven.compiler.target><scala.binary.version>2.11</scala.binary.version><PermGen>64m</PermGen><MaxPermGen>512m</MaxPermGen><scala.version>2.11.8</scala.version><spark.version>2.3.0</spark.version><hadoop.version>2.6.0</hadoop.version><scala.compat.version>2.11</scala.compat.version><hbase.version>1.3.1</hbase.version></properties><repositories><repository><id>scala-tools.org</id><name>Scala-Tools Maven2 Repository</name><url>http://scala-tools.org/repo-releases</url></repository></repositories><pluginRepositories><pluginRepository><id>scala-tools.org</id><name>Scala-Tools Maven2 Repository</name><url>http://scala-tools.org/repo-releases</url></pluginRepository></pluginRepositories><dependencies><!--①添加scala --><dependency><groupId>org.scala-lang</groupId><artifactId>scala-library</artifactId><version>${scala.version}</version></dependency><dependency><groupId>org.scala-lang</groupId><artifactId>scala-reflect</artifactId><version>${scala.version}</version></dependency><dependency><groupId>org.scala-lang</groupId><artifactId>scala-compiler</artifactId><version>${scala.version}</version></dependency><!-- Spark-core --><dependency><groupId>org.apache.spark</groupId><artifactId>spark-core_${scala.compat.version}</artifactId><version>${spark.version}</version></dependency><!--③添加sparkStreaming --><dependency><groupId>org.apache.spark</groupId><artifactId>spark-streaming_${scala.compat.version}</artifactId><version>${spark.version}</version></dependency><!--③添加sparkStreaming消费kafka数据版本 --><dependency><groupId>org.apache.spark</groupId><!--kafka0.10版本--><artifactId>spark-streaming-kafka-0-10_${scala.compat.version}</artifactId><!--kafka0.8版本--><!--<artifactId>spark-streaming-kafka-0-8_${scala.compat.version}</artifactId>--><version>${spark.version}</version></dependency><!-- 向kafka 生产数据需要包 --><dependency><groupId>org.apache.kafka</groupId><artifactId>kafka-clients</artifactId><version>0.10.0.0</version></dependency><!-- SparkSQL --><dependency><groupId>org.apache.spark</groupId><artifactId>spark-sql_${scala.compat.version}</artifactId><version>${spark.version}</version></dependency><!-- SparkSQL ON Hive--><dependency><groupId>org.apache.spark</groupId><artifactId>spark-hive_${scala.compat.version}</artifactId><version>${spark.version}</version></dependency><!-- sparkMLib --><dependency><groupId>org.apache.spark</groupId><artifactId>spark-mllib_${scala.compat.version}</artifactId><version>${spark.version}</version></dependency><!--连接 Redis 需要的包--><dependency><groupId>redis.clients</groupId><artifactId>jedis</artifactId><version>${spark.version}</version></dependency><!--添加阿里格式化json --><dependency><groupId>com.alibaba</groupId><artifactId>fastjson</artifactId><version>1.2.17</version></dependency><!--添加mysql驱动包 --><dependency><groupId>mysql</groupId><artifactId>mysql-connector-java</artifactId><version>5.1.31</version></dependency><dependency><groupId>org.scalatest</groupId><artifactId>scalatest_2.11</artifactId><version>3.2.0-SNAP5</version><scope>test</scope></dependency><!-- https://mvnrepository.com/artifact/junit/junit --><dependency><groupId>junit</groupId><artifactId>junit</artifactId><version>4.12</version><scope>test</scope></dependency><!-- 添加hadoop 依赖--><!-- https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-client --><dependency><groupId>org.apache.hadoop</groupId><artifactId>hadoop-client</artifactId><version>${hadoop.version}</version></dependency><dependency><groupId>org.apache.hadoop</groupId><artifactId>hadoop-common</artifactId><version>${hadoop.version}</version></dependency><dependency><groupId>org.apache.hadoop</groupId><artifactId>hadoop-hdfs</artifactId><version>${hadoop.version}</version></dependency><!-- hbase 依赖--><dependency><groupId>org.apache.hbase</groupId><artifactId>hbase-server</artifactId><version>${hbase.version}</version></dependency><dependency><groupId>org.apache.hbase</groupId><artifactId>hbase-client</artifactId><version>${hbase.version}</version><exclusions><exclusion><groupId>log4j</groupId><artifactId>log4j</artifactId></exclusion><exclusion><groupId>org.apache.thrift</groupId><artifactId>thrift</artifactId></exclusion><exclusion><groupId>org.jruby</groupId><artifactId>jruby-complete</artifactId></exclusion><exclusion><groupId>org.slf4j</groupId><artifactId>slf4j-log4j12</artifactId></exclusion><exclusion><groupId>org.mortbay.jetty</groupId><artifactId>jsp-2.1</artifactId></exclusion><exclusion><groupId>org.mortbay.jetty</groupId><artifactId>jsp-api-2.1</artifactId></exclusion><exclusion><groupId>org.mortbay.jetty</groupId><artifactId>servlet-api-2.5</artifactId></exclusion><exclusion><groupId>com.sun.jersey</groupId><artifactId>jersey-core</artifactId></exclusion><exclusion><groupId>com.sun.jersey</groupId><artifactId>jersey-json</artifactId></exclusion><exclusion><groupId>com.sun.jersey</groupId><artifactId>jersey-server</artifactId></exclusion><exclusion><groupId>org.mortbay.jetty</groupId><artifactId>jetty</artifactId></exclusion><exclusion><groupId>org.mortbay.jetty</groupId><artifactId>jetty-util</artifactId></exclusion><exclusion><groupId>tomcat</groupId><artifactId>jasper-runtime</artifactId></exclusion><exclusion><groupId>tomcat</groupId><artifactId>jasper-compiler</artifactId></exclusion><exclusion><groupId>org.jruby</groupId><artifactId>jruby-complete</artifactId></exclusion><exclusion><groupId>org.jboss.netty</groupId><artifactId>netty</artifactId></exclusion><exclusion><groupId>io.netty</groupId><artifactId>netty</artifactId></exclusion></exclusions></dependency><dependency><groupId>org.apache.hbase</groupId><artifactId>hbase-protocol</artifactId><version>1.3.1</version></dependency><dependency><groupId>org.apache.hbase</groupId><artifactId>hbase-annotations</artifactId><version>${hbase.version}</version><type>test-jar</type><scope>test</scope></dependency><dependency><groupId>org.apache.hbase</groupId><artifactId>hbase-hadoop-compat</artifactId><version>${hbase.version}</version><scope>test</scope><type>test-jar</type><exclusions><exclusion><groupId>log4j</groupId><artifactId>log4j</artifactId></exclusion><exclusion><groupId>org.apache.thrift</groupId><artifactId>thrift</artifactId></exclusion><exclusion><groupId>org.jruby</groupId><artifactId>jruby-complete</artifactId></exclusion><exclusion><groupId>org.slf4j</groupId><artifactId>slf4j-log4j12</artifactId></exclusion><exclusion><groupId>org.mortbay.jetty</groupId><artifactId>jsp-2.1</artifactId></exclusion><exclusion><groupId>org.mortbay.jetty</groupId><artifactId>jsp-api-2.1</artifactId></exclusion><exclusion><groupId>org.mortbay.jetty</groupId><artifactId>servlet-api-2.5</artifactId></exclusion><exclusion><groupId>com.sun.jersey</groupId><artifactId>jersey-core</artifactId></exclusion><exclusion><groupId>com.sun.jersey</groupId><artifactId>jersey-json</artifactId></exclusion><exclusion><groupId>com.sun.jersey</groupId><artifactId>jersey-server</artifactId></exclusion><exclusion><groupId>org.mortbay.jetty</groupId><artifactId>jetty</artifactId></exclusion><exclusion><groupId>org.mortbay.jetty</groupId><artifactId>jetty-util</artifactId></exclusion><exclusion><groupId>tomcat</groupId><artifactId>jasper-runtime</artifactId></exclusion><exclusion><groupId>tomcat</groupId><artifactId>jasper-compiler</artifactId></exclusion><exclusion><groupId>org.jruby</groupId><artifactId>jruby-complete</artifactId></exclusion><exclusion><groupId>org.jboss.netty</groupId><artifactId>netty</artifactId></exclusion><exclusion><groupId>io.netty</groupId><artifactId>netty</artifactId></exclusion></exclusions></dependency><dependency><groupId>org.apache.hbase</groupId><artifactId>hbase-hadoop2-compat</artifactId><version>${hbase.version}</version><scope>test</scope><type>test-jar</type><exclusions><exclusion><groupId>log4j</groupId><artifactId>log4j</artifactId></exclusion><exclusion><groupId>org.apache.thrift</groupId><artifactId>thrift</artifactId></exclusion><exclusion><groupId>org.jruby</groupId><artifactId>jruby-complete</artifactId></exclusion><exclusion><groupId>org.slf4j</groupId><artifactId>slf4j-log4j12</artifactId></exclusion><exclusion><groupId>org.mortbay.jetty</groupId><artifactId>jsp-2.1</artifactId></exclusion><exclusion><groupId>org.mortbay.jetty</groupId><artifactId>jsp-api-2.1</artifactId></exclusion><exclusion><groupId>org.mortbay.jetty</groupId><artifactId>servlet-api-2.5</artifactId></exclusion><exclusion><groupId>com.sun.jersey</groupId><artifactId>jersey-core</artifactId></exclusion><exclusion><groupId>com.sun.jersey</groupId><artifactId>jersey-json</artifactId></exclusion><exclusion><groupId>com.sun.jersey</groupId><artifactId>jersey-server</artifactId></exclusion><exclusion><groupId>org.mortbay.jetty</groupId><artifactId>jetty</artifactId></exclusion><exclusion><groupId>org.mortbay.jetty</groupId><artifactId>jetty-util</artifactId></exclusion><exclusion><groupId>tomcat</groupId><artifactId>jasper-runtime</artifactId></exclusion><exclusion><groupId>tomcat</groupId><artifactId>jasper-compiler</artifactId></exclusion><exclusion><groupId>org.jruby</groupId><artifactId>jruby-complete</artifactId></exclusion><exclusion><groupId>org.jboss.netty</groupId><artifactId>netty</artifactId></exclusion><exclusion><groupId>io.netty</groupId><artifactId>netty</artifactId></exclusion></exclusions></dependency></dependencies><build><sourceDirectory>src/main/scala</sourceDirectory><testSourceDirectory>src/test/scala</testSourceDirectory><plugins><!-- 在maven项目中既有java又有scala代码时配置 maven-scala-plugin 插件打包时可以将两类代码一起打包 --><plugin><groupId>org.scala-tools</groupId><artifactId>maven-scala-plugin</artifactId><executions><execution><goals><goal>compile</goal><goal>testCompile</goal></goals></execution></executions><configuration><scalaVersion>${scala.version}</scalaVersion><args><arg>-target:jvm-1.5</arg></args></configuration></plugin><plugin><groupId>org.apache.maven.plugins</groupId><artifactId>maven-eclipse-plugin</artifactId><configuration><downloadSources>true</downloadSources><buildcommands><buildcommand>ch.epfl.lamp.sdt.core.scalabuilder</buildcommand></buildcommands><additionalProjectnatures><projectnature>ch.epfl.lamp.sdt.core.scalanature</projectnature></additionalProjectnatures><classpathContainers><classpathContainer>org.eclipse.jdt.launching.JRE_CONTAINER</classpathContainer><classpathContainer>ch.epfl.lamp.sdt.launching.SCALA_CONTAINER</classpathContainer></classpathContainers></configuration></plugin></plugins></build><reporting><plugins><plugin><groupId>org.scala-tools</groupId><artifactId>maven-scala-plugin</artifactId><configuration><scalaVersion>${scala.version}</scalaVersion></configuration></plugin></plugins></reporting>
</project>
idea用maven开发spark程序的pom.xml相关推荐
- java+maven开发spark demo程序
在完成spark环境安装部署之后,部署过程这里略过,就可以通过java开发spark程序了. 1. 添加程序依赖 <dependencies><!-- spark-core spar ...
- 第10课：底实战详解使用Java开发Spark程序学习笔记
第10课:底实战详解使用Java开发Spark程序学习笔记 本期内容: 1. 为什么要使用Java? 2. 使用Java开发Spark实战 3. 使用Java开发Spark的Local和Cluster ...
- 第8课：彻底实战详解使用IDE开发Spark程序
第8课:彻底实战详解使用IDE开发Spark程序 1.下载安装windows下的scala-2.10.4. 2.打开eclipse,新建scala project: WordCount 3.修改依赖的 ...
- 第8课：彻底实战详解使用IDE开发Spark程序--集群模式运行
第8课:彻底实战详解使用IDE开发Spark程序--集群模式运行 拷贝WordCount.scala生成WordCountCluster.scala. 1. 将object WordCount改为ob ...
- Spark API编程动手实战-08-基于IDEA使用Spark API开发Spark程序-01
2019独角兽企业重金招聘Python工程师标准>>> 创建一个Scala IDEA工程: 点击"Next": 点击"Finish"完成工程的 ...
- spark java pom.xml_使用maven方式创建spark项目(配置pom.xml文件)
创建maven项目后,在pom.xml文件中添加如下配置项: 1.8 1.8 2.10.5 1.6.2 2.6.4 UTF-8 org.apache.spark spark-core_2.11 ${s ...
- 3000门徒内部训练绝密视频（泄密版）第8课：彻底实战详解使用IDE开发Spark程序
彻底实战详解使用IDE开发Spark程序 使用IDE开发Spark分析 使用IDE开发Spark实战 使用IDE开发Spark的Local和Cluster 开发两种选择:IDEA.Eclipse 下载 ...
- IDEA创建maven项目报错解决：Failed to create a Maven project: 'C:/Users/../IdeaProjects/../pom.xml' already e
此文首发于我的个人博客:IDEA创建maven项目报错解决 Failed to create a Maven project 'C:/Users/-/IdeaProjects/-/pom.xml' a ...
- maven文件报错（pom.xml或者jar包缺失）解决方法
maven文件报错(pom.xml或者jar包缺失)解决方法 参考文章: (1)maven文件报错(pom.xml或者jar包缺失)解决方法 (2)https://www.cnblogs.com/wa ...
最新文章
- python 安装easy_install和pip
- VS代码提示不出现或者提示变成英文或者各种奇葩问题的解决
- hyperopt中文文档:Scipy2013
- windbg调试实例（4）--句柄泄露
- java 实现 常见排序算法（四）基数排序
- 2021中国新物业服务发展白皮书
- 本土黑马谈车载语音识别 - 汽车电子 - 电子发烧友网
- kafka面试题整合列表
- 前端PDF文件转图片方法
- Office 2010如何打开MDI文件
- 无法安装冰点还原_教程｜有备有还，再备不难，如何实现整机还原？
- 马哥SRE第八周课程作业
- 【读万卷书】《挪威的森林》
- 用MFC做漂亮界面之美化对话框
- 解决sublime汉化后部分中文显示为繁体字
- Java二维码的一些简介、功能
- React 使用Markdown 更改样式
- 国产化适配之人大金仓数据库（二）数据迁移
- 升级 GCC 支持C++11 或 configure: error: *** A compiler with support for C++11 language features is requir
- pgsql按varchar字段排序