zoukankan      html  css  js  c++  java
  • C# .NET

    SqlBulkCopy.WriteToServer has 4 overloads:
    SqlBulkCopy.WriteToServer (DataRow[])
        Copies all rows from the supplied DataRow array to a destination table specified by the 
        DestinationTableName property of the SqlBulkCopy object. 
    SqlBulkCopy.WriteToServer (DataTable)
        Copies all rows in the supplied DataTable to a destination table specified by the 
        DestinationTableName property of the SqlBulkCopy object. 
    SqlBulkCopy.WriteToServer (IDataReader)
        Copies all rows in the supplied IDataReader to a destination table specified by the 
        DestinationTableName property of the SqlBulkCopy object. 
    SqlBulkCopy.WriteToServer (DataTable, DataRowState)
        Copies only rows that match the supplied row state in the supplied DataTable to a 
        destination table specified by the DestinationTableName property of the SqlBulkCopy object. 

    When importing text files with this method you have to create a DataTable first, import the text file 
    to the created DataTable and then write this DataTable to server.

    With this we're acctually performing 2 tasks in .net:
    1. Fill data from text file to DataTable in memory
    2. Fill data from DataTable in memory to SQL server

    Compared to SQL servers native bulk import methods where we just import the text file directly.

    I used the same file and the same table structure as in previous bulk import methods described in Last
    The time it took to complete the whole process was around 30 seconds.

    This is the code i used for import:

    private void StartImport()
    {
        Stopwatch sw = new Stopwatch();
        sw.Start();
        SqlBulkCopy bulkCopy = new SqlBulkCopy("Server=ServerName;Database=test;Trusted_Connection=True;", 
            SqlBulkCopyOptions.TableLock);
        bulkCopy.DestinationTableName = "dbo.testSQLBulkCopy";
        bulkCopy.WriteToServer(CreateDataTableFromFile());
        sw.Stop();
        txtResult.Text = (sw.ElapsedMilliseconds/1000.00).ToString();
    }
    private DataTable CreateDataTableFromFile()
    {
        DataTable dt = new DataTable();
        DataColumn dc;
        DataRow dr;
    
        dc = new DataColumn();
        dc.DataType = System.Type.GetType("System.Int32");
        dc.ColumnName = "c1";
        dc.Unique = false;
        dt.Columns.Add(dc);
        dc = new DataColumn();
        dc.DataType = System.Type.GetType("System.Int32");
        dc.ColumnName = "c2";
        dc.Unique = false;
        dt.Columns.Add(dc);
        dc = new DataColumn();
        dc.DataType = System.Type.GetType("System.Int32");
        dc.ColumnName = "c3";
        dc.Unique = false;
        dt.Columns.Add(dc);
        dc = new DataColumn();
        dc.DataType = System.Type.GetType("System.Int32");
        dc.ColumnName = "c4";
        dc.Unique = false;
        dt.Columns.Add(dc);
        StreamReader sr = new StreamReader(@"d:work	est.txt");
        string input;
        while ((input = sr.ReadLine()) != null)
        {
            string[] s = input.Split(new char[] { '|' });
            dr = dt.NewRow();
            dr["c1"] = s[0];
            dr["c2"] = s[1];
            dr["c3"] = s[2];
            dr["c4"] = s[3];
            dt.Rows.Add(dr);
        }
        sr.Close();
        return dt;
    }

    Bulk Import Methods are ad below..:- 


    1. BCP
    2. Bulk Insert
    3. OpenRowset with BULK option
    4. SQL Server Integration Services - SSIS

    I ran each bulk import option 12 times, disregarded best and worst time and averaged the remaining ten times.
    Results are:

    1. SSIS - FastParse ON = 7322 ms
    2. SSIS - FastParse OFF = 8387 ms
    3. Bulk Insert = 10534 ms
    4. OpenRowset = 10687 ms
    5. BCP = 14922 ms

    So speed gain is quite large when using FastParse.
    I was also surprised that SSIS - FastParse OFF method was faster by 20% to Bulk Insert and OpenRowset
    and around 40% faster than BCP.

    Since my desire was to test how much faster is importing flat files when FastParse option is used
    I created a text file containing 4 bigint columns with 1,000,000 rows.

    The script i used to create a sample test file in C#:

    string str;
    StreamWriter sw = new StreamWriter(@"d:work	est.txt");
    for (int i = 1; i <= 1000000; i++)
    {
        str = i.ToString() + "|" + Convert.ToString(i * 2) + "|" + Convert.ToString(i * 3) + "|" + Convert.ToString(i / 2);
        sw.WriteLine(str);
    }
    sw.Close();

    I also created this format file for use with BCP, Bulk Insert and OpenRowset:

    9.0
    4
    1       SQLBIGINT        0       8       "|"   1     c1       ""
    2       SQLBIGINT        0       8       "|"   2     c2       ""
    3       SQLBIGINT        0       8       "|"   3     c3       ""
    4       SQLBIGINT        0       8       "
    "   4     c4       ""
    

    SSIS Package was a very simple one with a Flat File source and SQL server destination objects.

    The sql script i used is:

    create database test
    go
    use test
    go
    -- ran for each SSIS test run
    -- SSIS data type for each column was "eight-byte signed integer [DT_I8]"
    drop table testFastParse
    create table testFastParse(c1 bigint, c2 bigint, c3 bigint, c4 bigint)
    go
    -- insert data using OPENROWSET 
    create table testOpenRowset(c1 bigint, c2 bigint, c3 bigint, c4 bigint)
    go
    DBCC DROPCLEANBUFFERS
    declare @start datetime
    set @start = getdate()
    insert into testOpenRowset(c1, c2, c3, c4)
    SELECT    t1.c1, t1.c2, t1.c3, t1.c4
    FROM    OPENROWSET( BULK 'd:work	est.txt', 
            FORMATFILE = 'd:work	estImport-f-n.Fmt') AS t1(c1, c2, c3, c4);
    select  getdate() - @start as ElapsedTime
    drop table testOpenRowset
    -- insert data using Bulk Insert
    create table testBulkInsert(c1 bigint, c2 bigint, c3 bigint, c4 bigint)
    go
    DBCC DROPCLEANBUFFERS
    declare @start datetime
    set @start = getdate()
    BULK INSERT testBulkInsert
       FROM 'd:work	est.txt'
       WITH (FORMATFILE='d:work	estImport-f-n.Fmt')
    select  getdate() - @start as ElapsedTime
    drop table testBulkInsert
    go
    -- insert data using BCP
    create table testBCP(c1 bigint, c2 bigint, c3 bigint, c4 bigint)
    go
    DBCC DROPCLEANBUFFERS
    exec master..xp_cmdshell 'bcp test.dbo.testBCP in d:work	est.txt -T -b1000000 -fd:work	estImport-f-n.Fmt'
    drop table testBCP
    go
    drop database test
  • 相关阅读:
    结对编程附加题、团队作业2、团队作业3评分标准
    【评分】集美大学软件工程1413班工程项目管理结对编程1-模块化
    【评分】集美大学软件工程1413班工程项目管理第0次作业——读后感
    第13周-网络
    第12周-流与文件
    第11周-多线程
    第10周-异常与多线程
    第九周-异常
    第08周-集合与泛型
    第7周-集合
  • 原文地址:https://www.cnblogs.com/itelite/p/4042259.html
Copyright © 2011-2022 走看看