zoukankan      html  css  js  c++  java
  • 使用activerecordimport 批量导入数据至mysql on rails enhanced activerecord

    Requiring

    There are essentially two ways to require activerecord-import:

    • Using Bundler (default Rails 3 app uses this)
    • Not using Bundler

    These instructions only work for activerecord-import 0.2.0 or higher. If you have a version before 0.2.0 then upgrade.

    Using Bundler

    There are two ways you can use Bundler. First, the Rails 3 way.

    The Rails 3 Way – autoloading gem dependencies

    Rails 3 by default autoloads your gem dependencies (thanks to config/application.rb). When this happens you only need to make sure activerecord-import is specified as a gem dependency in your Gemfile.

      gem "activerecord-import", ">= 0.2.0"
    

    Not autoloading

    If your gem dependencies aren’t autoloaded then simply require activerecord-import after activerecord has been loaded, ie:

    require 'active_record'
    require 'activerecord-import'
    

    That’s it.

     

     

     

    https://github.com/zdennis/activerecord-import

     

     

     

     

    activerecord-import

    activerecord-import is a library for bulk inserting data using ActiveRecord.

    Why activerecord-import?

    Because plain-vanilla, out-of-the-box ActiveRecord doesn’t provide support for inserting large amounts of data efficiently. With vanilla ActiveRecord you would have to perform individual save operations on each model:

    10.times do |i|
      Book.create! :name => "book #{i}"
    end
    

    This may work fine if all you have is 10 records, but if you have hundreds, thousands, or millions of records it can turn into a nightmare. This is where activerecord-import comes into play.

    An Introductory Example

    To run examples on this page you’ll need to first require activerecord-import, for that please take a brief look at Requiring.

    Here’s an example with equivalent behaviour using the #import method:

    books = []
    10.times do |i| 
      books << Book.new(:name => "book #{i}")
    end
    Book.import books
    

    This call to import does whatever is most efficient for the underlying database adapter. Pretty slick, eh?

    Features

    Here’s a list of some of the high-level features that activerecord-import provides:

    • activerecord-import can work with raw columns and arrays of values (fastest)
    • activerecord-import works with model objects (faster)
    • activerecord-import can perform validations (fast)
    • activerecord-import can perform on duplicate key updates (requires mysql)

    Upgrading from ar-extensions

    activerecord-import replaces the ar-extensions library and is compatible with Rails 3. It provides the exact same API for importing data, but it does not include any additional ar-extensions functionality.]

     

     

     

     

     

     

    desc "导入mail"

    task :batch_import_email => :environment do

        require 'active_record'

        require 'activerecord-import'

        count=0

        count2=0

            r=[]

        File.open("/home/mlzboy/my/b2c2/doc/zzzzzzz.txt","r") do |file|

            file.each_line do |line|

                count2+=1

                begin

                    if line.index("@").nil? ==false and line.index("zzzzzzz").nil?

                        email=line.chomp

                        #unless Subscription.exists?(:email=>email)

                            #Subscription.create(:email=>email)

                          r << Subscription.new(:email=>email)

                            count+=1

                            puts count

                        #end

                    end

                rescue

                    puts "==================="

                    puts line

                    puts count2

                end

                if r.size > 0 and r.size % 10000 == 0

                    Subscription.import r

                    r=[]

                end

            end

            if r.size!=0

                Subscription.import r

            end

        end

        put "finished!"

    end

     

  • 相关阅读:
    三、瀑布流中的ajax请求详解 Confi
    显示GridControl的横向滚动条
    耳目一新的感觉(转)
    JavaScript学习笔记——变量,作用域和内存管理
    Servlet编程笔记
    JavaScript,应该选择内嵌形式还是外链形式(Inline Code Versus External Files)
    JavaScript学习笔记——Javascript基本语法
    为主机空间添加一个phpmyadmin
    关于Post中明文传送密码以及其他
    Maven原理和Maven2新特性
  • 原文地址:https://www.cnblogs.com/lexus/p/1913214.html
Copyright © 2011-2022 走看看