zoukankan      html  css  js  c++  java
  • spark学习10(win下利用Intellij IDEA搭建spark开发环境)

    第一步:启动IntelliJ IDEA,选择Create New Project,然后选择Scala,点击下一步,输入项目名称wujiadong.spark继续下一步

    image

    image

    image

    image

    第二步:导入spark-assembly-1.5.1-hadoop2.6.0.jar包

    File——Project Structure——Libraries——点+号——点java——选择下载好的spark-assembly-1.5.1-hadoop2.6.0.jar包——点ok
    image

    image

    image

    第三步:创建WordCount类编写代码

    image

    image

    image

    第四步:导出jar包

    依次选择“File”–> “Project Structure” –> “Artifact”,选择“+”–> “Jar” –> “From Modules with dependencies”,选择main函数,并在弹出框中选择输出jar位置,并选择“OK”。
    最后依次选择“Build”–> “Build Artifact”编译生成jar包。具体如下图所示。

    image

    image

    image

    第五步:spark-submit提交运行

    hadoop@master:~/wujiadong$ spark-submit --class wujiadong.spark.WordCount  --executor-memory 500m --total-executor-cores 2 /home/hadoop/wujiadong/wujiadong.spark.jar hdfs://master:9000/wordcount.txt 
    17/02/02 20:27:34 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
    17/02/02 20:27:37 INFO Slf4jLogger: Slf4jLogger started
    17/02/02 20:27:37 INFO Remoting: Starting remoting
    17/02/02 20:27:37 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriver@192.168.1.131:52310]
    17/02/02 20:27:41 WARN MetricsSystem: Using default name DAGScheduler for source because spark.app.id is not set.
    17/02/02 20:27:44 INFO FileInputFormat: Total input paths to process : 1
    17/02/02 20:27:51 INFO deprecation: mapred.tip.id is deprecated. Instead, use mapreduce.task.id
    17/02/02 20:27:51 INFO deprecation: mapred.task.id is deprecated. Instead, use mapreduce.task.attempt.id
    17/02/02 20:27:51 INFO deprecation: mapred.task.is.map is deprecated. Instead, use mapreduce.task.ismap
    17/02/02 20:27:51 INFO deprecation: mapred.task.partition is deprecated. Instead, use mapreduce.task.partition
    17/02/02 20:27:51 INFO deprecation: mapred.job.id is deprecated. Instead, use mapreduce.job.id
    (spark,1)
    (wujiadong,1)
    (hadoop,1)
    (python,1)
    (hello,4)
    17/02/02 20:27:52 INFO RemoteActorRefProvider$RemotingTerminator: Shutting down remote daemon.
    17/02/02 20:27:52 INFO RemoteActorRefProvider$RemotingTerminator: Remote daemon shut down; proceeding with flushing remote transports.
    17/02/02 20:27:52 INFO RemoteActorRefProvider$RemotingTerminator: Remoting shut down.
    
    
    

    参考资料1

    参考资料2

  • 相关阅读:
    Hi35XXX海思媒体处理平台架构介绍(转)
    STM32F103步进电机梯形匀加速算法(转)
    STM32+ESP8266通过AT指令WIFI连接阿里云MQTT服务器(转)
    ESP8266 WIFI串口通信模块使用详解(转)
    UCGUI字体研究(转)
    用STM32制作汽车蓝牙OBD转速/车速表(带LED转速指示)(转)
    用STM32制作汽车蓝牙OBD转速/车速表(带LED转速指示)(转)
    接口测试框架接入性能测试实践分享
    超全Python IDE武器库大总结,优缺点一目了然!
    推荐一款万能抓包神器:Fiddler Everywhere
  • 原文地址:https://www.cnblogs.com/wujiadong2014/p/6361584.html
Copyright © 2011-2022 走看看