zoukankan      html  css  js  c++  java
  • flink time and watermark

    流处理中时间本质上就是一个普通的递增字段(long型,自1970年算起的微秒数),不一定真的表示时间。

    watermark只是应对乱序的办法之一,大多是启发式的,在延迟和完整性之间抉择。(如果没有延迟,就不够完整;如果有延迟,极端情况就是批处理,当然完整性足够高)

    org.apache.flink.streaming.api.watermark
    Class Watermark
      java.lang.Object
      org.apache.flink.streaming.runtime.streamrecord.StreamElement
      org.apache.flink.streaming.api.watermark.Watermark

    @PublicEvolving
    public final class Watermark extends StreamElement

      A Watermark tells operators that no elements with a timestamp older or equal to the watermark timestamp should arrive at the operator. Watermarks are emitted at the sources and propagate through the operators of the topology. Operators must themselves emit watermarks to downstream operators using Output.emitWatermark(Watermark). Operators that do not internally buffer elements can always forward the watermark that they receive. Operators that buffer elements, such as window operators, must forward a watermark after emission of elements that is triggered by the arriving watermark.
      In some cases a watermark is only a heuristic and operators should be able to deal with late elements. They can either discard those or update the result and emit updates/retractions to downstream operations.
      When a source closes it will emit a final watermark with timestamp Long.MAX_VALUE. When an operator receives this it will know that no more input will be arriving in the future.

    Modifier and Type     Field and Description
    static Watermark      MAX_WATERMARK
                 The watermark that signifies end-of-event-time.

    time和watermark的本质:

    1、流处理中的时间本质是一个普通的递增字段,不一定真的表示时间

    2、watermark只是应对乱序的办法之一,大多是启发式的,在延迟和完整性之间抉择

    reference:

    https://www.bilibili.com/video/av53193640/

    https://ci.apache.org/projects/flink/flink-docs-release-1.9/api/java/

  • 相关阅读:
    SQL Server复制情况下的高可用方案(一)镜像+复制
    sqlserver数据库镜像运行模式
    普通PC机支持内存128G,单条32G内存
    ICSharpCode.SharpZipLib 压缩、解压文件 附源码
    利用SharpZipLib进行字符串的压缩和解压缩
    SQL Server常见数据类型介绍
    .net中压缩和解压缩的处理
    solr字段压缩属性compressed新版本已经移除
    solr schema.xml Field属性详解
    Altmetric
  • 原文地址:https://www.cnblogs.com/zgq25302111/p/12077853.html
Copyright © 2011-2022 走看看