日志分析工具

这是我早先从项目中整理出来的内容，希望对感兴趣的人有所帮助。

1、AWStats
AWStats is a free powerful and featureful tool that generates advanced web, streaming, ftp or mail server statistics, graphically. This log analyzer works as a CGI or from command line and shows you all possible information your log contains, in few graphical web pages. It uses a partial information file to be able to process large log files, often and quickly. It can analyze log files from all major server tools like Apache log files (NCSA combined/XLF/ELF log format or common/CLF log format), WebStar, IIS (W3C log format) and a lot of other web, proxy, wap, streaming servers, mail servers and some ftp servers.

目前我们大部分情况下是使用OpenSource项目-AWStats来进行日志文件的分析，便于我们对系统的访问情况做分析，而不需要自己开发相应的日志分析工具。
1.1、Configuration
主要有以下几项：
1、 LogFile="F:\Logfiles\W3SVC1131665977\ex%YY-24%MM-24%DD-24.log"
是设置需要分析的日志文件
2、 LogType=W
是设置日志文件的类型
3、
LogFormat="%time2 %other %method %url %query %other %host %ua %other %referer %code %bytesd %other"
是设置日志文件中的字段信息，需要与分析的日志文件字段信息一致才可以
4、 SiteDomain="www.mywebsite.com.cn"
是设置域名，这样才可以区分Referer的情况
5、 SkipFiles=""
设置需要忽略的一些URLs
6、 NotPageList="css js class gif jpg jpeg png bmp ico swf xml dat vbs"
# Add here a list of kind of url (file extension) that must be counted as
# "Hit only" and not as a "Hit" and "Page/Download".
7、 URLNotCaseSensitive=1
8、 URLWithQuery=1
9、 URLWithQueryWithOnlyFollowingParameters="tabid"
10、URLReferrerWithQuery=1
1.2、Plugins
http://awstats.sourceforge.net/docs/awstats_contrib.html#PLUGINS
主要有：
LoadPlugin="timezone +8"
# Allow AWStats to correct a bad timezone for user of some IIS that use
# GMT date in its log instead of local server time.
1.3、Extra Sections
The AWStats ExtraSection features are powerfull setup options to allow you to add your own report not provided by default with AWStats. You can use it to build special reports, like number of sales for a particular product, marketing reports, counting for a particular user or agent, etc...

如果想用好AWStats，就必须认真研究ExtraSection，扩展出自定义的报表内容，这样才会对于你的项目很会有帮助。

注：原先的例子没找到，只好粘贴一份别人的例子出来给大家看看：

# To do: Ideally parameterize from not page list.
ExtraSectionName15="Downloads (diff,doc,pdf,rtf,sh,tgz,zip) - Top 10"
ExtraSectionCodeFilter15="200 304"
ExtraSectionCondition15="URL,(.*((\.diff)|(\.doc)|(\.pdf)|(\.rtf)|(\.sh)|(\.tgz)|(\.zip)))"
ExtraSectionFirstColumnTitle15="Download"
ExtraSectionFirstColumnValues15="URL,(.*)"
ExtraSectionFirstColumnFormat15="%s"
ExtraSectionStatTypes15=HBL
ExtraSectionAddAverageRow15=0
ExtraSectionAddSumRow15=1
MaxNbOfExtra15=10
MinHitExtra15=1

Anteztea AWStats resources: how to configure and use

1.4、Remark
1、有时我们需要对日志文件进行批处理，需要对整个目录或*.log的日志进行分析，会用到AWStats提供的logresolvemerge.pl，所以要在系统路径下设置logresolvemerge.pl所在目录。
例如：
set path=%path%;D:\Tools\AWStats\tools;
awstats.pl -update -config=mywebsite.wap -logfile="logresolvemerge.pl wap\1\*.log wap\2\*.log |"
它可以分析wap\1与wap\2两个目录下所有的*.log文件。
2、默认情况下AWStats出的是月报，有时间客户会要求出日报，所以需要做些配置。
awstats.pl -update -config=mywebsite.cs -logfile="logresolvemerge.pl ex*.log |" -databasebreak=day

http://localhost/awstats/awstats.pl?day=10&month=01&year=2007&config=mywebsite.cs&framename=index&databasebreak=day

用类似上面的方法才可以运行日报语句，查看日报情况。
可以认真查阅AWStats文档。
http://sourceforge.net/forum/forum.php?forum_id=43428
http://sourceforge.net/tracker/?atid=363764&group_id=13764&func=browse
2、Log Parser
在有些项目的报表中会用到LogParser来统计每个系列产品的PV值。
Log parser is a powerful, versatile tool that provides universal query access to text-based data such as log files, XML files and CSV files, as well as key data sources on the Windows® operating system such as the Event Log, the Registry, the file system, and Active Directory®.
You tell Log Parser what information you need and how you want it processed. The results of your query can be custom-formatted in text based output, or they can be persisted to more specialty targets like SQL, SYSLOG, or a chart.

目前已对LogParser进行了简单的封装，便于在程序中使用。

        /**//// <summary>
        /// Executes a query and returns a LogRecordSet object that can be used to navigate through the query output records. 
        /// </summary>
        /// <param name="query">A string containing the text of the SQL-Like query to be executed.</param>
        /// <returns></returns>
        public int Execute(string query)
        {
            LogQueryClassClass logParser = new LogQueryClassClass();
            COMIISW3CInputContextClassClass iisLog = new COMIISW3CInputContextClassClass();

            ILogRecordset rsLP = logParser.Execute(query, iisLog);
    
            int ret = rsLP.inputUnitsProcessed;
            rsLP.close();
            return ret;
        }
        /**//// <summary>
        /// Executes a query and writes the output records to an csv output format.
        /// </summary>
        /// <param name="query"></param>
        /// <param name="outfile"></param>
        /// <returns></returns>
        public int Execute(string query, string outfile)
        {
            return (Execute(query, OutputFormat.CSV) ? -1:1);
        }
        /**//// <summary>
        /// Executes a query and writes the output records to an output format. 
        /// But now only support CSV output format.
        /// </summary>
        /// <param name="query">A string containing the text of the SQL-Like query to be executed.</param>
        /// <param name="format">Output Format objects provide programmatic access to the output formats supported by Log Parser.</param>
        /// <returns>A boolean value. Returns TRUE if the query executed with parse errors or warnings; 
        /// FALSE if the query executed without any parse error nor warning.
        /// </returns>
        public bool Execute(string query, OutputFormat format)
        {
            LogQueryClassClass logParser = new LogQueryClassClass();
            COMIISW3CInputContextClassClass iisLog = new COMIISW3CInputContextClassClass();
            // Now only CSV output format
            if(format != OutputFormat.CSV)
                return true;
            
            MSUtil.COMCSVOutputContextClassClass csv = new COMCSVOutputContextClassClass();
            csv.tabs = true;
            
            return logParser.ExecuteBatch(query, iisLog, csv);
        }
        
        public DataSet ParseLog(string query)
        {
            LogQueryClassClass logParser = new LogQueryClassClass();
            COMIISW3CInputContextClassClass iisLog = new COMIISW3CInputContextClassClass();

            ILogRecordset rsLP = null;
            ILogRecord rowLP = null;

            rsLP = logParser.Execute(query, iisLog);

            DataTable tab = new DataTable("Results");

            // copy schema
            for (int i = 0; i < rsLP.getColumnCount(); i++)
            {
                DataColumn col = new DataColumn();
                col.ColumnName = rsLP.getColumnName(i);
                switch (rsLP.getColumnType(i))
                {
                    case 1:
                        col.DataType = Type.GetType("System.Int32");
                        break;
                    case 2:
                        col.DataType = Type.GetType("System.Double");
                        break;
                    case 4:
                        col.DataType = Type.GetType("System.DateTime");
                        break;
                    default:
                        col.DataType = Type.GetType("System.String");
                        break;
                }
                tab.Columns.Add(col);
            }

            // copy data
            while (!rsLP.atEnd())
            {
                rowLP = rsLP.getRecord();
                DataRow row = tab.NewRow();

                for (int i = 0; i < rsLP.getColumnCount(); i++)
                    row[i] = System.Web.HttpUtility.HtmlEncode(Convert.ToString(rowLP.getValue(i)));

                tab.Rows.Add(row);
                rsLP.moveNext();
            }
              rsLP.close();
            DataSet ds = new DataSet();
            ds.Tables.Add(tab);

            return ds;
        }
    }

2.1、Example

<Statistics>
  <BillRun BRI="123">
    <StartDateTime>20060804-150924</StartDateTime>
    <StopDateTime>20060804-153041</StopDateTime>
    <BillrunTotal>
      <TotalDocuments>12976</TotalDocuments>
      <TotalPages>61844</TotalPages>
      <TotalSheets>30922</TotalSheets>
      <TotalPrintedPages>51860</TotalPrintedPages>
    </BillrunTotal>
    <Destinations>
      <Destination Name="Printshop">
        <TotalDocuments>12055</TotalDocuments>
        <TotalPages>57758</TotalPages>
        <TotalSheets>28879</TotalSheets>
        <TotalPrintedPages>48501</TotalPrintedPages>
      </Destination>
      <Destination Name="OnDemand">
        <TotalDocuments>920</TotalDocuments>
        <TotalPages>4078</TotalPages>
        <TotalSheets>2039</TotalSheets>
        <TotalPrintedPages>3352</TotalPrintedPages>
      </Destination>
      <Destination Name="Error">
        <TotalDocuments>1</TotalDocuments>
        <TotalPages>8</TotalPages>
        <TotalSheets>4</TotalSheets>
        <TotalPrintedPages>7</TotalPrintedPages>
      </Destination>
    </Destinations>
  </BillRun>
</Statistics>

SELECT
     TotalDocuments
    , PrintshopDocuments
    , OnDemandDocuments
    , ErrorDocuments
USING
      SUM(CASE Name WHEN 'Printshop' THEN TotalDocuments2 ELSE 0 END) AS PrintshopDocuments
    , SUM(CASE Name WHEN 'OnDemand' THEN TotalDocuments2 ELSE 0 END) AS OnDemandDocuments
    , SUM(CASE Name WHEN 'Error' THEN TotalDocuments2 ELSE 0 END) AS ErrorDocuments
FROM test.xml
GROUP BY TotalDocuments

我们有时间可以研究一下，在查询、分析数据方面会对于项目有所帮助。

可以认真查阅Log Parser文档。
http://www.logparser.com/instantforum33/default.aspx
http://www.logparser.com/instantkb/default.aspx