zoukankan      html  css  js  c++  java
  • IDL double与float的精度检验

    pro test
      compile_opt idl2
      a = 100.000000002
      b = 100.000000000
      help, a
      help, b
      if a gt b then $
        print, '100.000000002 gt 100.000000000'
      if a eq b then $
        print, '100.000000002 eq 100.000000000'
      if a lt b then $
        print, '100.000000002 lt 100.000000000'
      a = 100.000000002d
      b = 100.000000000d
      help, a
      help, b
      if a gt b then $
        print, '100.000000002d gt 100.000000000d'
      if a eq b then $
        print, '100.000000002d eq 100.000000000d'
      if a lt b then $
        print, '100.000000002d lt 100.000000000d'
      print, ''
      a = double(100.000000002)
      b = double(100.000000000)
      help, a
      help, b
      if a gt b then $
        print, 'double(100.000000002) gt double(100.000000000)'
      if a eq b then $
        print, 'double(100.000000002) eq double(100.000000000)'
      if a lt b then $
        print, 'double(100.000000002) lt double(100.000000000)'
      print, ''
      a = 100.000000002*1.0d
      b = 100.000000000*1.0d
      help, a
      help, b
      if a gt b then $
        print, '100.000000002*1.0d gt 100.000000000*1.0d'
      if a eq b then $
        print, '100.000000002*1.0d eq 100.000000000*1.0d'
      if a lt b then $
        print, '100.000000002*1.0d lt 100.000000000*1.0d'
      print, ''
     
      a = double('100.1234567892')
      b = double('100.1234567890')
      help, a
      help, b
      if a gt b then $
        print, '100.1234567892*1.0d gt 100.1234567890*1.0d'
      if a eq b then $
        print, '100.1234567892*1.0d eq 100.1234567890*1.0d'
      if a lt b then $
        print, '100.1234567892*1.0d lt 100.1234567890*1.0d'
      print, ''
    end
     
    运行结果
    A               FLOAT     =       100.000
    B               FLOAT     =       100.000
    100.000000002 eq 100.000000000
     
    A               DOUBLE    =        100.00000
    B               DOUBLE    =        100.00000
    100.000000002d gt 100.000000000d
     
    A               DOUBLE    =        100.00000
    B               DOUBLE    =        100.00000
    double(100.000000002) eq double(100.000000000)
     
    A               DOUBLE    =        100.00000
    B               DOUBLE    =        100.00000
    100.000000002*1.0d eq 100.000000000*1.0d
     
    A               DOUBLE    =        100.12346
    B               DOUBLE    =        100.12346
    str100.1234567892 gt str100.1234567890
     
    初始的数据如果不是double型的,后面转为double型是没有用的,因为精度已经丢失
    从a=100.000000002 可知,虽然是小数点后9位, 但IDL认为是float型
    a=100.000000002d 可知,加上d之后,IDL认为是double型
    从a = double('100.1234567892') 可知,读入一个位数很多的字符串,将其转为double,可以最大限度的保持后续计算的精度
     
     
     
  • 相关阅读:
    问题描述:判断一个整数 n 是否为 2 的幂次方
    C#的关键字Explicit 和 Implicit
    .NET写入文件操作
    C# Main函数详解
    SpringBoot增加过滤XSS脚本攻击
    Hutool工具包导出Excel文件异常 You need to add dependency of poi-ooxml to your project
    微信H5表单点击输入框提示防欺诈盗号,请勿支付或输入qq密码
    RedisTemplate执行lua脚本在Redis集群模式下报错EvalSha is not supported in cluster environment.
    SpringBoot使用RedisTemplate+Lua脚本实现Redis分布式锁
    SpringBoot使用Thymeleaf打成jar包部署找不到页面
  • 原文地址:https://www.cnblogs.com/lqqgis/p/13626557.html
Copyright © 2011-2022 走看看