zoukankan      html  css  js  c++  java
  • StreamSets 设计Edge pipeline

    edge pipeline 运行在edge 执行模式,我们可以使用 data collector UI 进行edge pipeline 设计,
    设计完成之后,你可以部署对应的pipeline到edge 设备

    可以设计的edge pipeline

    • edge 发送pipeline
    edge 发送pipeline 使用特定的orgin读取edge设备上的数据,这个pipeline 可以在将数据发送到data collector 之前进行数据的处理
    • edge 接收pipeline
    接收pipeline可以接收来自edge 设备或者 data collector pipeline的数据

    orgin 组件

    • Dev Random Record Source
    • Dev Raw Data Source
    • Directory Edge pipelines do not support multithreaded processing.
      In an edge pipeline, the Directory origin always creates a single thread to read the files even if you configure it to use multiple threads.
    • File Tail
      In edge pipelines, the File Tail origin can read a single set of files.
      If you configure multiple sets of files for the origin, the origin reads only the files configured in the first set.
    • HTTP Client
      In edge pipelines, the HTTP Client origin does not support batch processing mode, pagination, or OAuth2 authorization.
    • HTTP Server Edge pipelines do not support multithreaded processing.
      In an edge pipeline, the HTTP Server origin always creates a single thread to read the files even if you configure it to use multiple threads.
    • MQTT Subscriber Edge pipelines that use MQTT stages require using an intermediary MQTT broker.
      For example, an edge sending pipeline uses an MQTT Publisher destination to write to an MQTT broker. The MQTT broker temporarily stores the data until the MQTT Subscriber origin in the edge receiving pipeline reads the data.
    • Sensor Reader
    • System Metrics
    • WebSocket Client
    • Windows Event Log

    processsor 组件

    • Delay
    • Dev Identity
    • Expression Evaluator
    • Field Remover
    • JavaScript Evaluator In edge pipelines, the JavaScript Evaluator processor does not support the sdcFunctions scripting object.
    • Stream Selector

    destinations 组件

    • CoAP Client
    • HTTP Client
    • Kafka Producer
    • MQTT Publisher Edge pipelines that use MQTT stages require using an intermediary MQTT broker.
      For example, an edge sending pipeline uses an MQTT Publisher destination to write to an MQTT broker. The MQTT broker temporarily stores the data until the MQTT Subscriber origin in the Data Collector receiving pipeline reads the data.
    • Trash
    • WebSocket Client

    错误记录处理

    • Discard 丢踢
      The pipeline discards the record.
    • Write to File 写到文件
      The pipeline writes error records and related details to a local directory on the edge device. Create another edge pipeline with a Directory origin to process the error records written to the file.
    • Write to MQTT 写到mqtt
      The pipeline publishes error records and related details to a topic on an MQTT broker. Create another edge or standalone Data Collector pipeline with an MQTT Subscriber origin to process the error records published to the broker.

    支持的数据格式

    • json
    • text

    限制

    • Email and webhook notifications cannot be sent by edge pipelines.
    • Rules and alerts cannot be defined for edge pipelines.
    • Edge pipelines support a limited number of record, math, pipeline, and string functions.
    • Edge pipelines do not support dataflow triggers.
    • Edge pipelines do not support multithreaded processing.
    • You cannot capture snapshots for edge pipelines.

    参考资料

    https://streamsets.com/documentation/datacollector/latest/help/datacollector/UserGuide/Edge_Mode/EdgePipelineTypes.html#concept_c14_m4r_4bb

  • 相关阅读:
    安装部署Python开发环境
    CentOS系统常见优化
    chm文件打开无法显示
    数据库恢复技术
    视图的认识
    存储过程的认识
    error C2471: 无法更新程序数据库 ,fatal error C1083: 无法打开程序数据库文件
    ubuntu下使用aptget install下载安装文件管理
    转:[译文] 程序员的禅修之路
    数据库的两段锁协议
  • 原文地址:https://www.cnblogs.com/rongfengliang/p/9515774.html
Copyright © 2011-2022 走看看