zoukankan      html  css  js  c++  java
  • IDL double与float的精度检验

    pro test
      compile_opt idl2
      a = 100.000000002
      b = 100.000000000
      help, a
      help, b
      if a gt b then $
        print, '100.000000002 gt 100.000000000'
      if a eq b then $
        print, '100.000000002 eq 100.000000000'
      if a lt b then $
        print, '100.000000002 lt 100.000000000'
      a = 100.000000002d
      b = 100.000000000d
      help, a
      help, b
      if a gt b then $
        print, '100.000000002d gt 100.000000000d'
      if a eq b then $
        print, '100.000000002d eq 100.000000000d'
      if a lt b then $
        print, '100.000000002d lt 100.000000000d'
      print, ''
      a = double(100.000000002)
      b = double(100.000000000)
      help, a
      help, b
      if a gt b then $
        print, 'double(100.000000002) gt double(100.000000000)'
      if a eq b then $
        print, 'double(100.000000002) eq double(100.000000000)'
      if a lt b then $
        print, 'double(100.000000002) lt double(100.000000000)'
      print, ''
      a = 100.000000002*1.0d
      b = 100.000000000*1.0d
      help, a
      help, b
      if a gt b then $
        print, '100.000000002*1.0d gt 100.000000000*1.0d'
      if a eq b then $
        print, '100.000000002*1.0d eq 100.000000000*1.0d'
      if a lt b then $
        print, '100.000000002*1.0d lt 100.000000000*1.0d'
      print, ''
     
      a = double('100.1234567892')
      b = double('100.1234567890')
      help, a
      help, b
      if a gt b then $
        print, '100.1234567892*1.0d gt 100.1234567890*1.0d'
      if a eq b then $
        print, '100.1234567892*1.0d eq 100.1234567890*1.0d'
      if a lt b then $
        print, '100.1234567892*1.0d lt 100.1234567890*1.0d'
      print, ''
    end
     
    运行结果
    A               FLOAT     =       100.000
    B               FLOAT     =       100.000
    100.000000002 eq 100.000000000
     
    A               DOUBLE    =        100.00000
    B               DOUBLE    =        100.00000
    100.000000002d gt 100.000000000d
     
    A               DOUBLE    =        100.00000
    B               DOUBLE    =        100.00000
    double(100.000000002) eq double(100.000000000)
     
    A               DOUBLE    =        100.00000
    B               DOUBLE    =        100.00000
    100.000000002*1.0d eq 100.000000000*1.0d
     
    A               DOUBLE    =        100.12346
    B               DOUBLE    =        100.12346
    str100.1234567892 gt str100.1234567890
     
    初始的数据如果不是double型的,后面转为double型是没有用的,因为精度已经丢失
    从a=100.000000002 可知,虽然是小数点后9位, 但IDL认为是float型
    a=100.000000002d 可知,加上d之后,IDL认为是double型
    从a = double('100.1234567892') 可知,读入一个位数很多的字符串,将其转为double,可以最大限度的保持后续计算的精度
     
     
     
  • 相关阅读:
    C# Dictionary几种遍历方式
    Android 代码中文字在手机上显示乱码问题解决方法
    【Android学习5】Clean 之后R文件丢失
    【android学习4】Eclipse中Clean作用
    Java基础语法(练习)
    Java基础语法(自定义类、ArrayList集合)
    Java基础语法(方法)
    Java基础语法(数组)
    Java基础(Scanner、Random、流程控制语句)
    Java基础(变量、运算符)
  • 原文地址:https://www.cnblogs.com/lqqgis/p/13626557.html
Copyright © 2011-2022 走看看