zoukankan      html  css  js  c++  java
  • spark学习

    一。读取文件并输出到控制台

    person.json

    {"name":"json","age":23,"hobby":"running"}
    {"name":"charles","age":32,"hobby":"basketball"}
    {"name":"tom","age":28,"hobby":"football"}
    {"name":"lili","age":24,"hobby":"running"}
    {"name":"bob","age":20,"hobby":"swimming"}
    

    读取文件

    StructType structType  = new StructType().add("name","string").add("age","integer").add("hobby","string");
    Dataset<Row> dr = spark.readStream()
    .schema(structType)
    .json("/Users/huyanxia/Documents/ceshi");//文件夹而不是文件名称

    输出

    dr.map(new MapFunction<Row, String>(){
    
                           @Override
                           public String call(Row row) throws Exception {
                               System.out.println("fdsg");
                               HashMap hashMap =new HashMap();
                               //这是一个遍历操作,row即表示为当前行数据,get(i)表示当前行的第几列
                               hashMap.put(row.get(0),row.get(1));
                               return return;
                           }
                       },Encoders.javaSerialization(Map.class)
                ).writeStream()
                        .outputMode("append")
                        .format("console")//输出到控制台
                        .start()
                        .awaitTermination();
    

      二。使用list构建Dataset

    List<String> list = new ArrayList<>();
                list.add("fawf");
                list.add("feaf");
                list.add("jfki urer");
                Dataset<String> ds  = spark.createDataset(list, Encoders.STRING());
                ds.first().substring(1,2);
                ds.show();
                Dataset<String> f = ds.filter(value->value.contains("a"));
                f.show();
                ds.map(new MapFunction<String, String>(){
    
                    @Override
                    public String call(String venderCode) throws Exception {
                        System.out.println("fdsg");
                        return venderCode+"fedfef";
                    }
                }, Encoders.STRING()).show();
    

      

  • 相关阅读:
    Qt:The CDB Process Terminated!调试失败
    关于【error C3646: 未知重写说明符】的若干种可能性
    QT:圆角设置
    Qt修改图片的背景色及设置背景色为透明的方法
    index首页加载数据库数据方法
    jsp+jstl实现登录验证
    java 分页
    初始Ajax学习笔记
    python wechat
    python json dumps load 区别
  • 原文地址:https://www.cnblogs.com/zhima-hu/p/13658372.html
Copyright © 2011-2022 走看看