Advanced Log Parser Charts Part 1 - Working With Configuration Scripts

I recently had a situation where I wanted to customize the chart output from Log Parser, and after a bunch of research I eventually arrived at the conclusion that configuration scripts to create customized charts are probably the least-documented feature of Log Parser. After a lot of experimentation, (and a bit of frustration), I finally managed to achieve the results that I wanted. With that in mind, I thought that it would make a great blog series if I documented some of the settings that I used.

Log Parser and Chart Configuration Scripts

When you look in the Log Parser help file, it makes mention of using configuration scripts to customize charts, and it provides the following small JavaScript sample:

// Add a caption
chartSpace.HasChartSpaceTitle = true;
chartSpace.ChartSpaceTitle.Caption = "Generated by Log Parser 2.2";
chartSpace.ChartSpaceTitle.Font.Size = 6;
chartSpace.ChartSpaceTitle.Position = chartSpace.Constants.chTitlePositionBottom;

// Change the background color
chart.PlotArea.Interior.Color = chartSpace.Constants.chColorNone;

Unfortunately, this sample isn't very useful, although I found dozens of forum posts that quote this sample as a way to do things - but it's the only sample that most people cite. The Log Parser help file mentions looking at the MSDN ChartSpace Object Model documentation, but that documentation is only slightly more useful. These two references are what led me to my earlier conclusion that chart configuration scripts are not well-documented, and especially when you are trying to do something with Log Parser.

What I found to be particularly helpful was to use the Log Parser COM interface and write scripts by using Adersoft's VbsEdit and JsEdit. In case you haven't used either of those applications, they are great IDEs for writing scripts; they both give you a great debugging environment, and they have a great object browser that I used to discover what options were available to me. In the end, these two editors made it possible to create the chart configuration scripts that I will discuss in this blog series.

By the way, chart configuration scripts can be written in VBScript or JavaScript, but for this blog I will use VBScript for the Log Parser COM samples and JavaScript for the configuration script samples. I didn't have to do it that way, but it seemed like a good idea to help differentiate between the samples.

Using COM versus the Command-Line

For the samples in this blog series, I will use Log Parser's COM interface and VBScript to create my charts, but this is not necessary; everything that I am documenting can be done from the command-line version of Log parser, and I'll give you some quick examples to see the differences.

The following examples generate some simple area charts that plot the total number of hits by day, and both examples do exactly the same thing:

Command-Line:

logparser.exe "SELECT Date, COUNT(*) AS Hits INTO HitsByDay.gif FROM *.log GROUP BY Date ORDER BY Date" -i:W3C -fileType:GIF -groupSize:800x600 -chartType:Area -categories:ON -values:ON -legend:OFF

COM Interface:
Option Explicit

Dim objLogQuery, strSQL
Dim objInputW3CFormat, objOutputChartFormat

Set objLogQuery = WScript.CreateObject("MSUtil.LogQuery")
Set objInputW3CFormat = WScript.CreateObject("MSUtil.LogQuery.W3CInputFormat")
Set objOutputChartFormat = WScript.CreateObject("MSUtil.LogQuery.ChartOutputFormat")

strSQL = "SELECT Date, COUNT(*) AS Hits " & _
" INTO HitsByDay.gif " & _
" FROM *.log " & _
" GROUP BY Date " & _
" ORDER BY Date"

objOutputChartFormat.groupSize = "800x600"
objOutputChartFormat.fileType = "GIF"
objOutputChartFormat.chartType = "Area"
objOutputChartFormat.categories = "ON"
objOutputChartFormat.values = "ON"
objOutputChartFormat.legend = "OFF"

objLogQuery.ExecuteBatch strSQL, objInputW3CFormat, objOutputChartFormat
Ugly Charts

Using some of the log files from one of my websites, the above samples created the following basic chart:

Taking a look at this chart makes it easy to see why you would want to customize your output; that light blue is pretty awful, and those values are pretty hard to read.

Specifying Configuration Scripts

If you remember the incredibly basic configuration script from earlier, you only need to add one parameter to each example in order to specify the configuration script:

Command-Line:

logparser.exe "SELECT Date, COUNT(*) AS Hits INTO HitsByDay.gif FROM *.log GROUP BY Date ORDER BY Date" -i:W3C -fileType:GIF -groupSize:800x600 -chartType:Area -categories:ON -values:ON -legend:OFF -config:HitsByDay.js

COM Interface:
Option Explicit

Dim objLogQuery, strSQL
Dim objInputW3CFormat, objOutputChartFormat

Set objLogQuery = WScript.CreateObject("MSUtil.LogQuery")
Set objInputW3CFormat = WScript.CreateObject("MSUtil.LogQuery.W3CInputFormat")
Set objOutputChartFormat = WScript.CreateObject("MSUtil.LogQuery.ChartOutputFormat")

strSQL = "SELECT Date, COUNT(*) AS Hits " & _
" INTO HitsByDay.gif " & _
" FROM *.log " & _
" GROUP BY Date " & _
" ORDER BY Date"

objOutputChartFormat.groupSize = "800x600"
objOutputChartFormat.fileType = "GIF"
objOutputChartFormat.chartType = "Area"
objOutputChartFormat.categories = "ON"
objOutputChartFormat.values = "ON"
objOutputChartFormat.legend = "OFF"
objOutputChartFormat.config = "HitsByDay.js"

objLogQuery.ExecuteBatch strSQL, objInputW3CFormat, objOutputChartFormat
Simple Output

Taking a look at the resulting chart, you can see why I mentioned earlier that the configuration script wasn't very useful; all it does is add a centered title to the bottom of the chart:

Yup - that's a pretty useless sample configuration script for chart customization.

Next...

In my subsequent posts, I'll show how to make this chart (and several other types of charts) look a lot better.


Note: This blog was originally posted at http://blogs.msdn.com/robert_mcmurray/

How to determine if FTP clients are using FTPS

One of my colleagues here at Microsoft, Emmanuel Boersma, just reminded me of an email thread that we had several weeks ago, where a customer had asked him how they could tell if FTPS was being used on their FTP server. He had pointed out that when he looks at his FTP log files, the port number was always 21, so it wasn't as easy as looking at a website's log files and looking for port 80 for HTTP versus port 443 for HTTPS. I had sent him the following notes, and I thought that they might make a good blog. ;-)

As I mentioned earlier, we had discussed the control channel is typically over port 21 for both FTP and FTPS, so you can't rely on the port. But having said that, I mentioned that you will see certain verbs in your FTP logs that will let you know when FTPS is being used, and that’s a reliable way to check.

With that in mind, I suggested the following two methods that you can use to determine if FTPS is being used:

  • If the port number is something other than 990, and you see the following verbs being used (and succeeding), then Explicit FTPS is being used:
    • AUTH
    • PBSZ
    • PROT
  • If the port is 990, then Implicit FTPS is being used. (This means the FTPS is always on.)

For example, see the highlighted data in following FTP log file excerpts:

Explicit FTPS over port 21:

#Fields: date time c-ip cs-username cs-host s-ip s-port cs-method cs-uri-stem sc-status sc-win32-status sc-substatus sc-bytes cs-bytes time-taken
2011-06-30 22:11:24 ::1 - - ::1 21 ControlChannelOpened - - 0 0 0 0 0
2011-06-30 22:11:24 ::1 - - ::1 21 AUTH TLS 234 0 0 31 10 16
2011-06-30 22:11:27 ::1 - - ::1 21 PBSZ 0 200 0 0 69 8 0
2011-06-30 22:11:27 ::1 - - ::1 21 PROT P 200 0 0 69 8 0
2011-06-30 22:11:36 ::1 - - ::1 21 USER robert 331 0 0 69 13 0
2011-06-30 22:11:42 ::1 robert - ::1 21 PASS *** 230 0 0 53 15 2808

Implicit FTPS over port 990:

#Fields: date time c-ip cs-username cs-host s-ip s-port cs-method cs-uri-stem sc-status sc-win32-status sc-substatus sc-bytes cs-bytes time-taken
2011-06-30 22:16:55 ::1 - - ::1 990 ControlChannelOpened - - 0 0 0 0 0
2011-06-30 22:16:58 ::1 - - ::1 990 USER robert 331 0 0 69 13 0
2011-06-30 22:16:58 ::1 robert - ::1 990 PASS *** 230 0 0 53 15 78
2011-06-30 22:16:58 ::1 robert - ::1 990 SYST - 500 5 51 1005 6 0
2011-06-30 22:16:58 ::1 robert - ::1 990 FEAT - 211 0 0 313 6 0
2011-06-30 22:16:58 ::1 robert - ::1 990 OPTS UTF8+ON 200 0 0 85 14 0
2011-06-30 22:16:58 ::1 robert - ::1 990 PBSZ 0 200 0 0 69 8 0
2011-06-30 22:16:58 ::1 robert - ::1 990 PROT P 200 0 0 69 8 0

FWIW – An explanation about Implicit FTPS and Explicit FTPS can be found in the following articles:


Note: This blog was originally posted at http://blogs.msdn.com/robert_mcmurray/

Converting W3C log files to NCSA format

Around a year ago I wrote a blog entry titled "Converting NCSA log files to W3C format", which showed how to use the MSWC.IISLog object to convert log files in the NCSA format back to W3C format. I wrote that blog entry to make up for the fact that the CONVLOG.EXE utility only converts log files to NCSA format, which some older log analysis software packages require. So what happens if you have a bunch of log files in W3C format and you don't have a copy of CONVLOG.EXE on your computer?

This blog entry is something of a reverse direction on my previous post, and shows you how to use the MSUtil.LogQuery object to convert W3C log files to NCSA format. The MSUtil.LogQuery object is shipped with LogParser, which you can download from one of the following URLs:

Once you've downloaded and installed the LogParser package, you will need to manually register the LogParser.dll file in order to use the MSUtil.LogQuery object. Having done so, you can use the Windows Script Host (WSH) code in this blog article to convert a folder filled with W3C log files to NCSA format. 

To use this code, just copy the code into notepad, and save it with a ".vbs" file extension on your system. To run it, copy the script to a folder that contains your W3C log files, (named "ex*.log"), then double-click it.

Option Explicit
Dim objFSO
Dim objFolder
Dim objInputFile
Dim objOutputFile
Dim objLogQuery
Dim objLogRecordSet
Dim objLogRecord
Dim strInputPath
Dim strOutputPath
Dim strLogRecord
Dim strLogTemp

Set objFSO = WScript.CreateObject("Scripting.FileSystemObject")
Set objFolder = objFSO.GetFolder(".")

For Each objInputFile In objFolder.Files
 strInputPath = LCase(objInputFile.Name)
 If Left(strInputPath,2) = "ex" And Right(strInputPath,4) = ".log" Then
  strOutputPath = objFolder.Path & "\" & "nc" & Mid(strInputPath,3)
  strInputPath = objFolder.Path & "\" & strInputPath
  Set objLogQuery = CreateObject("MSUtil.LogQuery")
  Set objLogRecordSet = objLogQuery.Execute("SELECT * FROM " & strInputPath)
  Set objOutputFile = objFSO.CreateTextFile(strOutputPath)
  Do While Not objLogRecordSet.atEnd
  
   Set objLogRecord = objLogRecordSet.getRecord
   strLogRecord = FormatField(objLogRecord.getValue("c-ip"))
   strLogRecord = strLogRecord & " " & FormatField("")
   strLogRecord = strLogRecord & " " & FormatField(objLogRecord.getValue("cs-username"))
   strLogTemp = BuildDateTime(objLogRecord.getValue("date"),objLogRecord.getValue("time"))
   strLogRecord = strLogRecord & " " & FormatField(strLogTemp)
   strLogRecord = strLogRecord & " """ & FormatField(objLogRecord.getValue("cs-method"))
   strLogRecord = strLogRecord & " " & FormatField(objLogRecord.getValue("cs-uri-stem"))
   strLogTemp = FormatField(objLogRecord.getValue("cs-version"))
   If strLogTemp = "-" Then
    strLogRecord = strLogRecord & " HTTP/1.0"""
   Else
    strLogRecord = strLogRecord & " " & strLogTemp & """"
   End If   
   strLogRecord = strLogRecord & " " & FormatField(objLogRecord.getValue("sc-status"))
   strLogRecord = strLogRecord & " " & FormatField(objLogRecord.getValue("sc-bytes"))
    objOutputFile.WriteLine strLogRecord
   objLogRecordSet.moveNext
  Loop
  
  Set objLogQuery = Nothing
  objOutputFile.Close
 
 End If
Next

Function FormatField(tmpField)
 On Error Resume Next
 FormatField = "-"
 If Len(tmpField) > 0 Then FormatField = Trim(tmpField)
End Function

Function BuildDateTime(tmpDate,tmpTime)
 On Error Resume Next
 BuildDateTime = "[" & _
  Right("0" & Day(tmpDate),2) & "/" & _
  Left(MonthName(Month(tmpDate)),3) & "/" & _
  Year(tmpDate) & ":" & _
  Right("0" & Hour(tmpTime),2) & ":" & _
  Right("0" & Minute(tmpTime),2) & ":" & _
  Right("0" & Second(tmpTime),2) & _
  " +0000]"
End Function

I hope this helps!