zoukankan      html  css  js  c++  java
  • 1、NASA Super Cloud Library(SCL)

    Empowering Data Management, Diagnosis, and Visualization of Cloud-Resolving Models (CRM) by Cloud Library upon Spark and Hadoop

    使用 Spark and Hadoop建立数据管理、诊断、可视化的一套云判识模型(CRM)

    主要有用的有以下几块:
    1、Develop Super Cloud Library (SCL) supporting Cloud Resolving Model using Spark on Hadoop.
    • Create cloud data files
    • Develop data model and Hadoop format transformer
    • Develop dynamic transfer tool to Hadoop
    • Develop subset and visualization APIs (Application Programming Interfaces)
    • Develop Web User Interface




    2、数据模型
    将NU-WRF的netcdf数据和GCE的二进制数据输出为CSV格式。
    例如:
    NU-WRF (1 km and 48 h simulation): IPHEX; 10 output variables. Hourly output in a text (CSV) format has 465GB
    GIGALES, 4096x4096x104, 2-3 day simulation: Single output data size ~2.5TB; Total output data size ~125TB
    3、IDL+Hadoop 
    48个1GB大小的WRF模拟数据的动画生成,能够从原来的17分钟改进为14分钟。
    Developed an animation/movie of 48 text files from NU-WRF Pilot simulations. Each file has ~1GB
    – 14 minutes for IDL with Hadoop streaming
    • NCCS Hadoop cluster has 34 nodes
    • IDL is on one Hadoop node and can run with multiple instances simultaneously
    • Files stored in Hadoop file system (HDFS)
    • 15 seconds for reading each file out of HDFS
    – 17 minutes for IDL with NCCS Discover GPFS file system
    • IDL is on one NCCS Dali compute node
    • Files stored in NCCS GPFS file system





    4、基于Hadoop的动态读取系统。

    Job trackers and task trackers run on the master node, and task trackers and map tasks run on slave nodes. Virtual blocks are managed in the virtual block table in namenode. A task tracker initiates the PFS prefetcher thread to get data directly from a remote PFS. Map tasks initiate the PFS reader threads to import data directly from the remote PFS.

    5、 各个技术使用场景的统计





  • 相关阅读:
    主流液晶显示器尺寸参数
    不能访问网络位置的解决方法(转)
    打开Word提示你正试图运行的函数包含有宏或需要宏语言支持的内容
    教你如何防“蹭网”
    ASA数据库瘦身(原创)
    多种解决:“Word无法启动转换器mswrd632.wpc”方法
    百兆线与千兆线网线制作方法
    linux常用命令
    DefaultIfEmpty
    实现手机发送验证码 进行验证
  • 原文地址:https://www.cnblogs.com/zhangshucheng/p/5127761.html
Copyright © 2011-2022 走看看