zoukankan      html  css  js  c++  java
  • IDL double与float的精度检验

    pro test
      compile_opt idl2
      a = 100.000000002
      b = 100.000000000
      help, a
      help, b
      if a gt b then $
        print, '100.000000002 gt 100.000000000'
      if a eq b then $
        print, '100.000000002 eq 100.000000000'
      if a lt b then $
        print, '100.000000002 lt 100.000000000'
      a = 100.000000002d
      b = 100.000000000d
      help, a
      help, b
      if a gt b then $
        print, '100.000000002d gt 100.000000000d'
      if a eq b then $
        print, '100.000000002d eq 100.000000000d'
      if a lt b then $
        print, '100.000000002d lt 100.000000000d'
      print, ''
      a = double(100.000000002)
      b = double(100.000000000)
      help, a
      help, b
      if a gt b then $
        print, 'double(100.000000002) gt double(100.000000000)'
      if a eq b then $
        print, 'double(100.000000002) eq double(100.000000000)'
      if a lt b then $
        print, 'double(100.000000002) lt double(100.000000000)'
      print, ''
      a = 100.000000002*1.0d
      b = 100.000000000*1.0d
      help, a
      help, b
      if a gt b then $
        print, '100.000000002*1.0d gt 100.000000000*1.0d'
      if a eq b then $
        print, '100.000000002*1.0d eq 100.000000000*1.0d'
      if a lt b then $
        print, '100.000000002*1.0d lt 100.000000000*1.0d'
      print, ''
     
      a = double('100.1234567892')
      b = double('100.1234567890')
      help, a
      help, b
      if a gt b then $
        print, '100.1234567892*1.0d gt 100.1234567890*1.0d'
      if a eq b then $
        print, '100.1234567892*1.0d eq 100.1234567890*1.0d'
      if a lt b then $
        print, '100.1234567892*1.0d lt 100.1234567890*1.0d'
      print, ''
    end
     
    运行结果
    A               FLOAT     =       100.000
    B               FLOAT     =       100.000
    100.000000002 eq 100.000000000
     
    A               DOUBLE    =        100.00000
    B               DOUBLE    =        100.00000
    100.000000002d gt 100.000000000d
     
    A               DOUBLE    =        100.00000
    B               DOUBLE    =        100.00000
    double(100.000000002) eq double(100.000000000)
     
    A               DOUBLE    =        100.00000
    B               DOUBLE    =        100.00000
    100.000000002*1.0d eq 100.000000000*1.0d
     
    A               DOUBLE    =        100.12346
    B               DOUBLE    =        100.12346
    str100.1234567892 gt str100.1234567890
     
    初始的数据如果不是double型的,后面转为double型是没有用的,因为精度已经丢失
    从a=100.000000002 可知,虽然是小数点后9位, 但IDL认为是float型
    a=100.000000002d 可知,加上d之后,IDL认为是double型
    从a = double('100.1234567892') 可知,读入一个位数很多的字符串,将其转为double,可以最大限度的保持后续计算的精度
     
     
     
  • 相关阅读:
    .NET下的加密解密大全(1): 哈希加密
    orm fluentdata使用相关文章
    xml处理相关文章收藏
    Salty Fish(区间和)
    Fennec VS. Snuke
    Splitting Pile
    ST表(离线RMQ)
    Exponentiation(高精度大数)
    高斯消元(模板)
    Online Judge(字符串-格式)
  • 原文地址:https://www.cnblogs.com/lqqgis/p/13626557.html
Copyright © 2011-2022 走看看