zoukankan      html  css  js  c++  java
  • 英特尔深度学习框架BigDL——a distributed deep learning library for Apache Spark

    BigDL: Distributed Deep Learning on Apache Spark

    What is BigDL?

    BigDL is a distributed deep learning library for Apache Spark; with BigDL, users can write their deep learning applications as standard Spark programs, which can directly run on top of existing Spark or Hadoop clusters.

    • Rich deep learning support. Modeled after Torch, BigDL provides comprehensive support for deep learning, including numeric computing (via Tensor) and high level neural networks; in addition, users can load pre-trained Caffe or Torchmodels into Spark programs using BigDL.

    • Extremely high performance. To achieve high performance, BigDL uses Intel MKL and multi-threaded programming in each Spark task. Consequently, it is orders of magnitude faster than out-of-box open source CaffeTorch or TensorFlowon a single-node Xeon (i.e., comparable with mainstream GPU).

    • Efficiently scale-out. BigDL can efficiently scale out to perform data analytics at "Big Data scale", by leveraging Apache Spark (a lightning fast distributed data processing framework), as well as efficient implementations of synchronous SGD and all-reduce communications on Spark.

    Why BigDL?

    You may want to write your deep learning programs using BigDL if:

    • You want to analyze a large amount of data on the same Big Data (Hadoop/Spark) cluster where the data are stored (in, say, HDFS, HBase, Hive, etc.).

    • You want to add deep learning functionalities (either training or prediction) to your Big Data (Spark) programs and/or workflow.

    • You want to leverage existing Hadoop/Spark clusters to run your deep learning applications, which can be then dynamically shared with other workloads (e.g., ETL, data warehouse, feature engineering, classical machine learning, graph analytics, etc.)

    How to use BigDL?

  • 相关阅读:
    2020/11/15助教一周小结(第十一周)
    神经网络--理解
    案例一:鸢尾花数据的分类
    TF2基础知识
    软件工程助教工作总结
    软工课程改进建议
    2020-12-27 助教一周小结(第十七周)
    2020-12-20 助教一周小结(第十六周)
    2020-12-13 助教一周小结(第十五周)
    2020-12-06 助教一周小结(第十四周)
  • 原文地址:https://www.cnblogs.com/bonelee/p/7349523.html
Copyright © 2011-2022 走看看