zoukankan      html  css  js  c++  java
  • php导出大数据scv

    1.我系统要导出30w的数据,刚开始我使用的是:

    ini_set('memory_limit', '1024M');
    set_time_limit(0);
    header("Content-type:text/csv");
    header("Content-Disposition:attachment;filename=收款明细.csv");
    header('Cache-Control:must-revalidate,post-check=0,pre-check=0');
    header('Expires:0');
    header('Pragma:public');
    (1) $str= "订单编号,来源店铺,付款ID,付款日期,付款金额,付款方式,付款币种,单据类型,记账币种,所属法人,凭证编码
    ";
    (2) $result = 查询30w的数据。
    foreach ($result as $item) {
    $str .= $item['order_no'] . ",";
    $str .= $item['order_from'] . ",";
    $str .= $item['payment_id'] . ",";
    $str .= date('Y-m-d ', strtotime($item['payment_time'])) . ",";
    $str .= $item['payment_amount'] . ",";
    $str .= $item['payment_method'] . ",";
    $str .= $item['payment_currency'] . ",";
    $str .= $documentType[$item['document_type']] . ",";
    $str .= $item['account_currency'] . ",";
    $str .= $item['corporate_sn'] . ",";
    $str .= $item['certificate_code'] . ",";
    $str .= " ";
    }
    echo mb_convert_encoding($str, "CP936", "UTF-8");
    unset($str);
    return ;
    描述:我要导出30w的大数据,(2) 一下子查询30w的数据可能会导致查询连接数据库超时 (1) 如果查询30w的数据查出来了,然后拼接到一个变量上,变量也可能会导致字符串太长而报错。

    所以我们想到查询的时候进行切割,分页去查询。在字符串拼接的时候,可以转换字符串然后echo,优化代码如下:


    ini_set('memory_limit', '1024M');
    set_time_limit(0);
    header("Content-type:text/csv");
    header("Content-Disposition:attachment;filename=收款明细.csv");
    header('Cache-Control:must-revalidate,post-check=0,pre-check=0');
    header('Expires:0');
    header('Pragma:public');
    $top= "订单编号,来源店铺,付款ID,付款日期,付款金额,付款方式,付款币种,单据类型,记账币种,所属法人,凭证编码
    ";
    echo mb_convert_encoding($top, "CP936", "UTF-8");
    $count = 查询总数的sql语句。

    $pre_count=50000;
    //循环获取
    for($i=0;$i<intval($count/$pre_count)+1;$i++){

    $result = 分页查询返回结果。
    foreach ($result as $item) {
    $str .= $item['order_no'] . ",";
    $str .= $item['order_from'] . ",";
    $str .= $item['payment_id'] . ",";
    $str .= date('Y-m-d ', strtotime($item['payment_time'])) . ",";
    $str .= $item['payment_amount'] . ",";
    $str .= $item['payment_method'] . ",";
    $str .= $item['payment_currency'] . ",";
    $str .= $documentType[$item['document_type']] . ",";
    $str .= $item['account_currency'] . ",";
    $str .= $item['corporate_sn'] . ",";
    $str .= $item['certificate_code'] . ",";
    $str .= " ";
    echo mb_convert_encoding($str, "CP936", "UTF-8");
    unset($str);
    }
    }
    return ;



  • 相关阅读:
    spring cloud配置中心
    网关中自定义登陆验证过滤器
    spring cloud网关
    Hystrix断路器 熔断器Hystrix的在Fegin的集成
    Hystrix断路器 熔断器Hystrix的在Ribbon的集成
    负载均衡二Feign
    Eureka负载均衡Ribbon
    Eureka高可用注册中心(解决单点故障)
    Eureka多服务调用
    input错误提示,点击提交,提示有未填项,屏幕滑到input未填项的位置
  • 原文地址:https://www.cnblogs.com/kobigood/p/5584140.html
Copyright © 2011-2022 走看看