Geeky Bob

For the next installments in my series about FTP clients, I will be taking a look at two FTP redirectors at the same time. In this specific blog post, I will focus on NetDrive (from Bdrive Inc.), whereas my previous post looked at WebDrive (from South River Technologies).

At the time of this blog's writing, NetDrive is a for-retail FTP client and redirector which is available from the following URL:

http://www.netdrive.net/

For this blog post I will be using NetDrive version 2.3.2.

NetDrive 2.3 Overview

NetDrive is different from many of the other FTP clients that I have reviewed because it is an Internet protocol redirector, meaning that it allows you to map drive letters to a variety of Internet repositories. When you install and open NetDrive, you are presented with the list of supported Internet protocols and repositories which you can use for mapping drives:

As you can see from the illustration above, NetDrive's list of support technologies is quite extensive: DropBox, Box.net, Google Drive, OneDrive, Amazon S3, Openstack Swift, FTP, SFTP, and WebDAV.

When you add a drive or configure the settings for one of the default drives, you are presented with a dialog box where you can enter the settings for the drive connection; note that there are very few settings for FTP connections:

As you add drives, the NetDrive user interface will display the drives and their current connection status:

As an added touch, NetDrive customizes its drive icons in Windows Explorer, so you can easily see the type of mapped drive for each connection:

I would love to take an in-depth look at all of the supported protocols in this review, but this series is about FTP clients, so I'll move on to the FTP-specific features that I normally review.

Using NetDrive 2.3 with FTP over SSL (FTPS)

NetDrive 2.3 has built-in support for FTP over SSL (FTPS), although it only appears to support Explicit FTPS - and it does so in a confusing way. When you are editing the settings for an FTP drive connection, you need to check the box for SSL/TLS in order to enable FTPS. Unfortunately, when you do so, the dialog box will change the port to 990, which is the port number for Implicit FTPS; however, in my testing I could not get Implicit FTPS to work:

With the above information in mind, I needed to manually change the port number back to 21 in order to use Explicit FTPS with NetDrive:

Using NetDrive 2.3 with True FTP Hosts

True FTP hosts are not supported natively by NetDrive 2.3, and there are no settings which allow you to customize the login environment in order to work around this situation.

Using NetDrive 2.3 with Virtual FTP Hosts

NetDrive 2.3's login settings allow you to specify the virtual host name as part of the user credentials by using syntax like "ftp.example.com|username" or "ftp.example.com\username", so you can use virtual FTP hosts with NetDrive 2.3.

Scorecard for NetDrive 2.3

This concludes my quick look at a few of the FTP features that are available with NetDrive 2.3, and here are the scorecard results:

That wraps things up for today's review of NetDrive 2.3. Your key take-aways should be: NetDrive has some nice features, and it supports a wide variety of protocols with a similar user experience; that being said, NetDrive has very few settings for drive connections, so its capabilities are somewhat limited.

Note: This blog was originally posted at http://blogs.msdn.com/robert_mcmurray/

For the next installments in my series about FTP clients, I will be taking a look at two FTP redirectors at the same time. In this specific blog post, I will focus on WebDrive (from South River Technologies), whereas my next post will look at NetDrive (from Bdrive Inc.).

At the time of this blog's writing, WebDrive is a for-retail FTP client and redirector which is available from the following URL:

http://www.webdrive.com/

For this blog post I will be using WebDrive version 12.10.4082.

WebDrive 12 Overview

Before I continue, I would like to begin with some background information: because of my ongoing blog series about FTP clients, one question that I have often been asked is, "Which FTP client do you use?" Usually I have to answer, "That depends." I know that my answer sounds non-committal, but to be honest - I have yet to find an FTP client that does everything that I want, although a few FTP clients have had enough features for me to use them quite often. And with that in mind, I need to point out that I purchased my first license for WebDrive over 12 years ago, and over the years I have periodically renewed my license for later versions. So to partially answer my earlier question - WebDrive is one of the FTP clients that I have used a lot.

That being said, WebDrive is different from many of the other FTP clients that I have reviewed because it is an Internet protocol redirector, meaning that it allows you to map drive letters to a variety of Internet-based repositories. (I'll discuss those various protocols and repositories shortly.)

When you install and open WebDrive, you are presented with a fairly empty user interface:

If you click the App Settings icon, you will be presented with a dialog box that offers dozens of customizable options:

When you click the New icon, you will be presented with a Site Wizard which lists the supported Internet protocols and repositories which you can use for mapping drives:

As you can see from the illustration above, WebDrive's list of support technologies is quite extensive: WebDAV, Secure WebDAV, FTP, Secure FTP, Google Drive, Amazon S3, SFTP, Dropbox, and FrontPage Server Extensions.

When you choose to create an FTP connection, WebDrive launches its Site Wizard, and the initial dialog box is pretty self-explanatory:

However, when you click the Advanced Settings button, you are presented once again with dozens of customizable settings for this specific connection:

As you continue to add sites with WebDrive, their connection types and current statuses are displayed in the user interface:

However, when you view your drives in Windows Explorer, even though network drives which are mapped through WebDrive are displayed with a different icon, you cannot tell the protocol type for mapped drives; this is one of the few times where NetDrive supported a feature that I really missed in WebDrive. (See my next blog entry for more information.)

WebDrive 12 supports command-line scripting, so if you find the features of the built-in Windows FTP client are somewhat limited, you can investigate scripting WebDrive:

WebDrive Command Line Parameters

I would love to take an in-depth look at all of the supported protocols in this review, but this series is about FTP clients, so I'll move on to the FTP-specific features that I normally review.

Using WebDrive 12 with FTP over SSL (FTPS)

WebDrive 12 has built-in support for FTP over SSL (FTPS), and it supports both Explicit and Implicit FTPS. To specify which type of encryption to use for FTPS, you need to choose the appropriate option from the Security Type drop-down menu in the FTP Settings for a site:

Using WebDrive 12 with True FTP Hosts

True FTP hosts are not supported natively by WebDrive 12, and there are no settings that I could find which would allow me to customize the login environment in order to work around this situation.

Using WebDrive 12 with Virtual FTP Hosts

WebDrive 12's login settings allow you to specify the virtual host name as part of the user credentials by using syntax like "ftp.example.com|username" or "ftp.example.com\username", so you can use virtual FTP hosts with WebDrive 12.

Scorecard for WebDrive 12

This concludes my quick look at a few of the FTP features that are available with WebDrive 12, and here are the scorecard results:

That wraps things up for today's review of WebDrive 12. Your key take-aways should be: WebDrive is a powerful redirector with support for a wide variety of protocols. What's more, the WebDrive application and each individual connection contain dozens of options which allow you to customize the environment in hundreds of ways. As is the case with many of my reviews, I have barely presented a fraction of the capabilities that are available in WebDrive 12; you might want to try it out and experiment with all of its possibilities.

Note: This blog was originally posted at http://blogs.msdn.com/robert_mcmurray/

For this next installment in my series about FTP clients, I want to take a look at Globalscape's CuteFTP, which is available from the following URL:

http://www.cuteftp.net/

CuteFTP is a for-retail product that used to be available in several editions - Lite, Home, and Pro - but at the time of this blog CuteFTP was only available in a single edition which combined all of the features. With that in mind, for this blog post I used CuteFTP 9.0.5.

CuteFTP 9.0 Overview

I should start off with a quick side note: it's kind of embarrassing that it has taken me so long to review CuteFTP, because CuteFTP has been my primary FTP client at one time or other over the past 15 years or so. That being said, it has been a few years since I had last used CuteFTP, so I was curious to see what had changed.

Fig. 1 - The Help/About dialog in CuteFTP 9.0.

To start things off, when you first install CuteFTP 9.0, it opens a traditional explorer-style view with the Site Manager displayed.

Fig. 2 - CuteFTP 9.0's Site Manager.

When you click File -> New, you are presented with a variety of connection options: FTP, FTPS, SFTP, HTTP, etc.

Fig. 3 - Creating a new connection.

When the Site Properties dialog is displayed during the creation of a new site connection, you have many of the options that you would expect, including the ability to change the FTP connection type after-the-fact; e.g. FTP, FTPS, SFTP, etc.

Fig. 4 - Site connection properties.

Once an FTP connection has been established, the CuteFTP connection display is pretty much what you would expect in any graphical FTP client.

Fig. 5 - FTP connection established.

A cool feature for me is that CuteFTP 9.0 supports a COM interface, (which is called the Transfer Engine), so you can automate CuteFTP commands through .NET or a scripting language. What was specifically cool about CuteFTP's scripting interface was the inclusion of several practical samples in the help file that is installed with the application.

Fig. 6 - Scripting CuteFTP.

Fig. 7 - Scripting samples in the CuteFTP help file.

Anyone who has read my blogs in the past knows that I am also a big fan of WebDAV, and an interesting feature of CuteFTP is built-in WebDAV integration. Of course, this functionality is a little redundant if you are using any version of Windows starting from Windows XP and later since WebDAV integration is built-in to the operating system via the WebDAV redirector, (which lets you map drive letters to WebDAV-enabled websites). But still - it's cool that CuteFTP is trying to be an all-encompassing transfer client.

Fig. 8 - Creating a WebDAV connection.

One last cool feature that I should call out in the overview is the integrated HTML editor, which is pretty handy. I could see where this might be useful on a system where you use FTP and you don't want to bother installing a separate editor.

Fig. 9 - CuteFTP's Integrated HTML editor.

Using CuteFTP 9.0 with FTP over SSL (FTPS)

CuteFTP 9.0 has built-in support for FTP over SSL (FTPS), and it supports both Explicit and Implicit FTPS. To specify which type of encryption to use for FTPS, you need to choose the appropriate option from the Protocol type drop-down menu in the Site Properties dialog box for an FTP site.

Fig. 10 - Specifying the FTPS encryption.

I was really happy to discover that I could use CuteFTP 9.0 to configure an FTP connection to drop out of FTPS on either the data channel or command channel once a connection is established. This is a very flexible design, because it allows you to configure FTPS for just your user credentials with no data and no post-login commands, or all commands and no data, or all data and all commands, etc.

Fig. 11 - Specifying additional FTPS options.

Using Using CuteFTP 9.0 with True FTP Hosts

CuteFTP 9.0 does not have built-in support for the HOST command that is specified in RFC 7151, nor does CuteFTP have a first-class way to specify pre-login commands for a connection.

But that being said, I was able find a way to configure CuteFTP 9.0 to send a HOST command for a connection by specifying custom advanced proxy commands. Here are the steps to pull this off:

1. Bring up the properties dialog for an FTP site in the CuteFTP Site Manager
2. Click the Options
3. Choose Use site specific option in the drop-down
4. Enter your FTP domain name in the Host name field
5. Click the Advanced button
6. Specify Custom for the Authentication Type
7. Enter the following information:
   
   HOST ftp.example.com
   USER %user%
   PASS %pass%
   
   Where ftp.example.com is your FTP domain name
8. Click OK for all of the open dialog boxes

Fig. 12 - Specifying a true FTP hostname via custom proxy settings.

Note: I could not get this workaround to successfully connect with FTPS sessions; I could only get it to work with regular (non-encrypted) FTP sessions.

Using Using CuteFTP 9.0 with Virtual FTP Hosts

CuteFTP 9.0's login settings allow you to specify the virtual host name as part of the user credentials by using syntax like "ftp.example.com|username" or "ftp.example.com\username". So if you don't want to use the workaround that I listed earlier, or you need to use FTPS, you can use virtual FTP hosts with CuteFTP 9.0.

Fig. 13 - Specifying an FTP virtual host.

Scorecard for CuteFTP 9.0

This concludes my quick look at a few of the FTP features that are available with CuteFTP 9.0, and here are the scorecard results:

That wraps things up for today's review of CuteFTP 9.0. Your key take-aways should be: CuteFTP is good FTP client; it has added some great features over the years, and as with most of the FTP clients that I have reviewed, I am sure that I have barely scratched the surface of its potential.

Note: This blog was originally posted at http://blogs.msdn.com/robert_mcmurray/

I get a lot of question about various configuration settings for the IIS FTP service, and most of the settings that I discuss with people can be configured through the FTP features in the IIS Manager. That being said, there are some useful configuration settings for the FTP service which I periodical send to people that have no user interface for setting them. With that in mind, I thought I would write a quick blog to point out a few of these obscure settings that I personally use the most-often or I send to other people.

Using Domain Name Syntax as an FTP Virtual Hostname

I use this setting on all of my FTP servers because it seems a little more natural to me. Here's the scenario: the IIS FTP service supports two kinds of hostnames:

• "Real" FTP hostnames - these are real FTP hostnames that are specified by using the FTP HOST command (as defined in RFC 7151)
• "Virtual" ftp hostnames - these are FTP hostnames that are specified as part of the FTP USER command

Real FTP hostnames are pretty straight-forward: an FTP client specifies the hostname with a HOST command when a user is connecting to the server. Once the IIS FTP service receives that command, the FTP service routes the FTP session to the correct FTP site.

That being said, the FTP HOST command is still rather new, so only a handful of FTP clients currently support it. Because of that, you can use FTP "virtual" hostnames with the IIS FTP service. By default that syntax uses the "vertical line" or "pipe" character to differentiate between the hostname and user name. For example:

• "ftp.contoso.com|username"
• "ftp.fabrikam.com|username"

When you are specifying your FTP credentials in your FTP client, you would enter your username like the preceding examples. While this syntax is valid for both the IIS FTP service and the underlying FTP protocol, it seems a little odd to most users (including me). With that in mind, we added a configuration setting for the FTP service that will allow you to use the more-familiar domain\username syntax like the following examples:

• "ftp.contoso.com\username"
• "ftp.fabrikam.com\username"

To enable this feature, use the following steps:

1. Open a command prompt as an administrator.
2. Type the following commands:
   cd /d "%SystemRoot%\System32\Inetsrv"
appcmd.exe set config -section:system.ftpServer/serverRuntime /hostNameSupport.useDomainNameAsHostName:"True" /commit:apphost
net.exe stop FTPSVC
net.exe start FTPSVC
3. Close the command prompt.

More information about this feature is available in the IIS configuration reference at the following URL:

• FTP Host Name Support <hostNameSupport>
  http://www.iis.net/ConfigReference/system.ftpServer/serverRuntime/hostNameSupport

FTP Credential Caching

The FTP service caches user credentials for successfully-authenticated user sessions in order to help improve login performance, and I wrote the following detailed blog about this a couple of years ago:

Credential Caching in FTP 7.0 and FTP 7.5

I don't want to re-post an old blog, but I have sent several people to that blog over the years, so I thought that it was worth mentioning here since it seems to be referenced quite often. The problem that people seem to run into the most is that their old password is still valid for FTP after they have changed it, and this is caused by the FTP service caching their user credentials.

This is especially annoying for me personally when I am working on a development computer where I am creating an authentication provider. Unless I disable credential caching on my development computer, I can never seem to get any work done. To resolve this issue, I disable credential caching for the FTP service by using the following steps:

1. Open a command prompt as an administrator.
2. Type the following commands:
   cd /d "%SystemRoot%\System32\Inetsrv"
appcmd.exe set config -section:system.ftpServer/caching /credentialsCache.enabled:"False" /commit:apphost
net.exe stop FTPSVC
net.exe start FTPSVC
3. Close the command prompt.

The blog which I mentioned earlier goes into more detail about setting a custom timeout interval for credential caching instead of disabling the feature entirely, and all of the settings for FTP credential caching are in the IIS configuration reference at the following URLs:

• FTP Caching <caching>
  http://www.iis.net/ConfigReference/system.ftpServer/caching
• FTP Credentials Cache <credentialsCache>
  http://www.iis.net/ConfigReference/system.ftpServer/caching/credentialsCache

FTP Client Certificate Authentication

FTP Client Certificate Authentication is an often-overlooked feature of the IIS FTP service, and I think that this is due to two reasons:

1. There is no user interface to configure the required settings
2. Configuring the required settings is very difficult

My second reason cannot be understated; I usually have to set up FTP Client Certificate Authentication once or twice a year in order to test various scenarios, and each time I do so I am reminded of just how difficult it can be to get everything right, and equally how easy it is to get something wrong.

Fortunately I took the time a couple of years ago to write a blog which documents everything that it takes to configure the FTP service, and I have used my notes in that blog several times. In complement to my blog on the subject, Vivek Kumbhar wrote an excellent blog series with additional steps to configure your Active Directory for certificate authentication. With that in mind, here are all of the requisite blog posts that you would need to set up this feature:

• Configuring FTP Client Certificate Authentication in FTP 7
  Describes how to configure your FTP server for FTP Client Certificate Authentication.
• Configure Client Certificate Mapping in FTP 7.5 - Part 1
  Describes how to set up your Active Directory server, IIS server, and FTP client systems.
• Configure Client Certificate Mapping in FTP 7.5 - Part 2
  Walks you through obtaining and installing a server certificate, which you will use later for your FTP site.
• Configure Client Certificate Mapping in FTP 7.5 - Part 3
  Walks you through setting up an FTP site with SSL.
• Configure Client Certificate Mapping in FTP 7.5 - Part 4
  Describes how to configure Active Directory mapping for a user account.

As I have mentioned before, configuring this feature is not for the faint-of-heart, but it can be very beneficial from a security standpoint.

For more information about the settings that are required for FTP Client Certificate Authentication, see the following articles in the IIS configuration reference:

• FTP Client Certificate Authentication <clientCertAuthentication>
  http://www.iis.net/ConfigReference/system.applicationHost/sites/site/ftpServer/security/authentication/clientCertAuthentication
• FTP over SSL <ssl>
  http://www.iis.net/ConfigReference/system.applicationHost/sites/site/ftpServer/security/ssl
• FTP SSL Client Certificates <sslClientCertificates>
  http://www.iis.net/ConfigReference/system.applicationHost/sites/site/ftpServer/security/sslClientCertificates

That wraps it up for today's post. ;-]

Note: This blog was originally posted at http://blogs.msdn.com/robert_mcmurray/

Shortly after I published my FTP ETW Tracing and IIS 8 blog post, I was using the batch file from that blog to troubleshoot an issue that I was having with a custom FTP provider. One of the columns which I display in my results is Clock-Time, which is obviously a sequential timestamp that is used to indicate the time and order in which the events occurred.

(Click the following image to view it full-size.)

At first glance the Clock-Time values might appear to be a range of useless numbers, but I use Clock-Time values quite often when I import the data from my ETW traces into something like Excel and I need to sort the data by the various columns.

That being said, apart from keeping the trace events in order, Clock-Time isn't a very user-friendly value. However, LogParser has some great built-in functions for crunching date/time values, so I decided to update the script to take advantage of some LogParser coolness and reformat the Clock-Time value into a human-readable Date/Time value.

My first order of business was to figure out how to decode the Clock-Time value; since Clock-Time increases for each event, it is obviously an offset from some constant, and after a bit of searching I found that the Clock-Time value is the offset in 100-nanosecond intervals since midnight on January 1, 1601. (Windows uses that value in a lot of places, not just ETW.) Once I had that information, it was pretty easy to come up with a LogParser formula to convert the Clock-Time value into the local time for my system, which is much easier to read.

With that in mind, here is the modified batch file:

@echo off

rem ======================================================================

rem Clean up old log files
for %%a in (ETL CSV) do if exist "%~n0.%%a" del "%~n0.%%a"

echo Starting the ETW session for full FTP tracing...
LogMan.exe start "%~n0" -p "IIS: Ftp Server" 255 5 -ets
echo.
echo Now reproduce your problem.
echo.
echo After you have reproduced your issue, hit any key to close the FTP
echo tracing session. Your trace events will be displayed automatically.
echo.
pause>nul

rem ======================================================================

echo.
echo Closing the ETW session for full FTP tracing...
LogMan.exe stop "%~n0" -ets

rem ======================================================================

echo.
echo Parsing the results - this may take a long time depending on the size of the trace...
if exist "%~n0.etl" (
   TraceRpt.exe "%~n0.etl" -o "%~n0.csv" -of CSV
   LogParser.exe "SELECT [Clock-Time], TO_LOCALTIME(ADD(TO_TIMESTAMP('1601-01-01 00:00:00', 'yyyy-MM-dd hh:mm:ss'), TO_TIMESTAMP(DIV([Clock-Time],10000000)))) AS [Date/Time], [Event Name], Type, [User Data] FROM '%~n0.csv'" -i:csv -e 2 -o:DATAGRID -rtp 20
)

When you run this new batch file, it will display an additional "Date/Time" column with a more-informative value in local time for the sever where you captured the trace.

(Click the following image to view it full-size.)

The new Date/Time column is considerably more practical, so I'll probably keep it in the batch file that I use when I am troubleshooting. You will also notice that I kept the original Clock-Time column; I chose to do so because I will undoubtedly continue to use that column for sorting when I import the data into something else, but you can safely remove that column if you would prefer to use only the new Date/Time value.

That wraps it up for today's post. :-)

Note: This blog was originally posted at http://blogs.msdn.com/robert_mcmurray/

Shortly after I published my FTP ETW Tracing and IIS 8 blog post, I was using the batch file from that blog to troubleshoot an issue that I was having with a custom FTP provider. One of the columns which I display in my results is Clock-Time, which is obviously a sequential timestamp that is used to indicate the time and order in which the events occurred.

(Click the following image to view it full-size.)

At first glance the Clock-Time values might appear to be a range of useless numbers, but I use Clock-Time values quite often when I import the data from my ETW traces into something like Excel and I need to sort the data by the various columns.

That being said, apart from keeping the trace events in order, Clock-Time isn't a very user-friendly value. However, LogParser has some great built-in functions for crunching date/time values, so I decided to update the script to take advantage of some LogParser coolness and reformat the Clock-Time value into a human-readable Date/Time value.

My first order of business was to figure out how to decode the Clock-Time value; since Clock-Time increases for each event, it is obviously an offset from some constant, and after a bit of searching I found that the Clock-Time value is the offset in 100-nanosecond intervals since midnight on January 1, 1601. (Windows uses that value in a lot of places, not just ETW.) Once I had that information, it was pretty easy to come up with a LogParser formula to convert the Clock-Time value into the local time for my system, which is much easier to read.

With that in mind, here is the modified batch file:

@echo off

rem ======================================================================

rem Clean up old log files
for %%a in (ETL CSV) do if exist "%~n0.%%a" del "%~n0.%%a"

echo Starting the ETW session for full FTP tracing...
LogMan.exe start "%~n0" -p "IIS: Ftp Server" 255 5 -ets
echo.
echo Now reproduce your problem.
echo.
echo After you have reproduced your issue, hit any key to close the FTP
echo tracing session. Your trace events will be displayed automatically.
echo.
pause>nul

rem ======================================================================

echo.
echo Closing the ETW session for full FTP tracing...
LogMan.exe stop "%~n0" -ets

rem ======================================================================

echo.
echo Parsing the results - this may take a long time depending on the size of the trace...
if exist "%~n0.etl" (
   TraceRpt.exe "%~n0.etl" -o "%~n0.csv" -of CSV
   LogParser.exe "SELECT [Clock-Time], TO_LOCALTIME(ADD(TO_TIMESTAMP('1601-01-01 00:00:00', 'yyyy-MM-dd hh:mm:ss'), TO_TIMESTAMP(DIV([Clock-Time],10000000)))) AS [Date/Time], [Event Name], Type, [User Data] FROM '%~n0.csv'" -i:csv -e 2 -o:DATAGRID -rtp 20
)

When you run this new batch file, it will display an additional "Date/Time" column with a more-informative value in local time for the sever where you captured the trace.

(Click the following image to view it full-size.)

The new Date/Time column is considerably more practical, so I'll probably keep it in the batch file that I use when I am troubleshooting. You will also notice that I kept the original Clock-Time column; I chose to do so because I will undoubtedly continue to use that column for sorting when I import the data into something else, but you can safely remove that column if you would prefer to use only the new Date/Time value.

That wraps it up for today's post. :-)

Note: This blog was originally posted at http://blogs.msdn.com/robert_mcmurray/

In the past I have written a couple of blogs about using the FTP service's Event Tracing for Windows (ETW) features to troubleshoot issues; see FTP and ETW Tracing and Troubleshooting Custom FTP Providers with ETW for details. Those blog posts contain batch files which use the built-in Windows LogMan utility to capture an ETW trace, and they use downloadable LogParser utility to parse the results into human-readable form. I use the batch files from those blogs quite often, and I tend to use them a lot when I am developing custom FTP providers which add new functionality to my FTP servers.

Unfortunately, sometime around the release of Windows 8 and Windows Server 2012 I discovered that the ETW format had changed, and the current version of LogParser (version 2.2) cannot read the new ETW files. When you try to use the batch files from my blog with IIS 8, you see the following errors:

Verifying that LogParser.exe is in the path...
Done.

Starting the ETW session for full FTP tracing...
The command completed successfully.

Now reproduce your problem.

After you have reproduced your issue, hit any key to close the FTP tracing session. Your trace events will be displayed automatically.

Closing the ETW session for full FTP tracing...
The command completed successfully.

Parsing the results - this may take a long time depending on the size of the trace...
Task aborted.
Cannot open <from-entity>: Trace file "C:\temp\ftp.etl" has been created on a OS version (6.3) that is not compatible with the current OS version

Statistics:
-----------
Elements processed: 0
Elements output: 0
Execution time: 0.06 seconds

I meant to research a workaround at the time, but one thing led to another and I simply forgot about doing so. But I needed to use ETW the other day when I was developing something, so that seemed like a good time to quit slacking and come up with an answer. :-)

With that in mind, I came up with a very easy workaround, which I will present here. Once again, this batch file has a requirement on LogParser being installed on your system, but for the sake of brevity I have removed the lines from this version of the batch file which check for LogParser. (You can copy those lines from my previous blog posts if you want that functionality restored.)

Here's the way that this workaround is implemented: instead of creating an ETW log and then parsing it directly with LogParser, this new batch file invokes the built-in Windows TraceRpt command to parse the ETW file and save the results as a CSV file, which is then read by LogParser to view the results in a datagrid like the batch files in my previous blogs:

@echo off

rem ======================================================================

rem Clean up old log files
for %%a in (ETL CSV) do if exist "%~n0.%%a" del "%~n0.%%a"

echo Starting the ETW session for full FTP tracing...
LogMan.exe start "%~n0" -p "IIS: Ftp Server" 255 5 -ets
echo.
echo Now reproduce your problem.
echo.
echo After you have reproduced your issue, hit any key to close the FTP
echo tracing session. Your trace events will be displayed automatically.
echo.
pause>nul

rem ======================================================================

echo.
echo Closing the ETW session for full FTP tracing...
LogMan.exe stop "%~n0" -ets

rem ======================================================================

echo.
echo Parsing the results - this may take a long time depending on the size of the trace...
if exist "%~n0.etl" (
   TraceRpt.exe "%~n0.etl" -o "%~n0.csv" -of CSV
   LogParser.exe "SELECT [Clock-Time], [Event Name], Type, [User Data] FROM '%~n0.csv'" -i:csv -e 2 -o:DATAGRID -rtp 20
)

Here's another great thing about this new batch file - it will also work down-level on Windows 7 and Windows Server 2008; so if you have been using my previous batch files with IIS 7 - you can simply replace your old batch file with this new version. You will see a few differences between the results from my old batch files and this new version, namely that I included a couple of extra columns that I like to use for troubleshooting.

(Click the following image to view it full-size.)

There is one last thing which I would like to mention in closing: I realize that it would be much easier on everyone if Microsoft simply released a new version of LogParser which works with the new ETW format, but unfortunately there are no plans at the moment to release a new version of LogParser. And trust me - I'm just as depressed about that fact as anyone else. :-(

Note: This blog was originally posted at http://blogs.msdn.com/robert_mcmurray/

I received an email yesterday from the RFC Editor that a new Request for Comments (RFC) document has just been published, RFC 7151, which adds support for a new "HOST" command to FTP. This new command allows hosting multiple FTP sites on a single IP address, much like what Host Headers provide for HTTP.

Here's the URL to the new RFC on the RFC Editor website:

File Transfer Protocol HOST Command for Virtual Hosts
http://www.ietf.org/rfc/rfc7151.txt

Or you can see the HTML-based version at the following URL:

http://tools.ietf.org/html/rfc7151

One minor point which I would like to clarify is that this adds a new command for FTP to specify which virtual host to connect to. I periodically hear people referring to this as "FTP Host Headers", but that is technically incorrect since FTP does not have request headers like HTTP. Here's a simple example of what the communications flow looks like when the HOST command is used:

I need to make sure that I thank my co-author for this RFC, Paul Hethmon, who has authored other FTP-related RFCs over the years. For example, Paul wrote RFC 3659, and he co-wrote RFC 2389 with Robert Elz. As a result, the Internet community has Paul and Robert to thank for several great FTP command extensions in the past. (e.g. FEAT, OPTS, MDTM, SIZE, REST, MLST, MLSD, etc.) Paul and I co-wrote RFC 7151 over the past several years, and it was great working with him.

Support for the HOST command has been built-in to Microsoft's FTP service since IIS 7.0, but now that the RFC has been officially published I hope that this feature will be adopted by other FTP servers and clients. That being said, IIS is not the only implementation of the FTP HOST command; at the time of this blog post, these are the server and client implementations that I am aware of which already provide support for this new command. (Note: there may be more than I have listed here; these are just the implementations that I currently know about.)

• FTP Servers:
  • FTP Daemon (see http://www.pro-bono-publico.de/projects/ftpd.html)
  • Ipswitch (in the WS_FTP service; see http://www.ipswitchft.com/)
  • Microsoft (in the IIS FTP service; see http://www.iis.net/)
  • RhinoSoft (in the Serv-U File server; see http://www.serv-u.com/)
• FTP Clients:
  • Microsoft (in the Visual Studio 2010/2012/2013 and Expression Web 4 products)
  • Rebex (in the Rebex FTP/SSL for .NET library; see http://www.rebex.net/)
  • RhinoSoft (in the FTP Voyager client; see http://www.rhinosoft.com/)
  • Scooter Software (in the Beyond Compare client; see http://scootersoftware.com/)
  • SmartFTP (in the SmartFTP client; see http://smartftp.com/)

In addition to the clients listed above, if you have been reading my series on FTP clients over the past few years, I have posted details on how to use the FTP HOST command with some other FTP clients which do not provide built-in support. For example, the Core FTP Client allows you to specific pre-login commands as part of an FTP site's connection properties, so you can manually type in the HOST command and save it along the site's settings.

A Little Bit of History

When I joined the feature team which was creating the FTP service for Windows Server 2008, one of the things that bothered us was that there was no way at the protocol level to host multiple FTP sites on the same IP address. There were several ways that FTP server implementations were approximating that sort of functionality, for example the User Isolation features that we ship with FTP for IIS, but each FTP server seemed to be implementing its own workaround and there was no standardization.

Because of this limitation, our team received a lot of requests to add "FTP Host Headers," although as I explained earlier FTP has no concept of request headers. To help address some of the questions which I was often seeing, I explained the lack of hostname support for FTP in detail in the comments section of my FTP User Isolation with Multiple User Accounts blog that I posted back in 2006, which was shortly before we began work on implementing the HOST command. I will paraphrase some of my comments here:

While I realize that the ability host multiple FTP sites on the same IP address and port like HTTP is a desired configuration, the simple answer is that FTP does not currently support this at the protocol level. To put things in perspective, RFC 959 is the governing document for FTP, and that was published in October of 1985. FTP was simply not designed for the Internet as we use and understand it today, even though it is a generally reliable protocol that many people will continue to use for some time. HTTP/1.1 was designed much later and resolved this problem, but only for HTTP requests.

There are three ways that you can create unique bindings for a web or HTTP site: IP address, port, or host header. FTP can create unique bindings by IP address or port, but the FTP protocol does not currently provide support for hostnames.

Here's why: HTTP packets consist of a set of request headers and possibly a block of data. Here's an example of a simple GET request:

GET /default.aspx HTTP/1.0 [CrLf]
Accept: */* [CrLf]
[CrLf]

When HTTP 1.1 was published in RFC 2068 and RFC 2616, it defined a header for specifying a "host" name in a separate name/value pair:

GET /default.aspx HTTP/1.1 [CrLf]
Host: example.com [CrLf]
Accept: */* [CrLf]
[CrLf]

The "Host" header allows multiple HTTP virtual servers ("hosts") on the same IP address and port that are differentiated by host name. While this works great for the HTTP protocol, FTP currently has no comparable functionality. As such, the FTP protocol would have to be updated to allow multiple hosts on the same IP address and port, then FTP servers and clients would need to be updated to accommodate the changes to FTP.

While my explanation may have clarified root cause of the FTP limitation for anyone who was asking about it, I personally thought the situation was unacceptable. This inspired me to research the addition of a new command for FTP which would allow FTP clients to specify hostnames. As I was researching how to propose a new RFC document to the IETF, I discovered that Paul Hethmon had been researching the same problem a few years earlier. I contacted Paul and offered to combine our work, and he agreed. After several years of work and a great deal of supportive assistance from dozens of great people whom I met through the IETF, RFC 7151 has finally been published.

There are a lot of people besides Paul whom I should thank, and we mention them in the acknowledgments section of our RFC, which you can read at the following URL:

http://tools.ietf.org/html/rfc7151#appendix-B

One final note - two of my coworkers, Jaroslav Dunajsky and Wade Hilmo, are mentioned in the acknowledgments section of the RFC. Jaroslav is the developer who implemented the FTP HOST command for IIS, and Wade is a senior developer on the IIS team who graciously allowed me to bounce ideas off him while I was doing my research over the past few years. (I probably I owe him a lunch or two.)

Note: This blog was originally posted at http://blogs.msdn.com/robert_mcmurray/

For this next installment in my series about FTP clients, I want to take a look at WinSCP, which is an open source FTP/SFTP client that is available from the following URL:

http://www.winscp.net/

For this blog post I used WinSCP 5.5.1, and it was available for free when I wrote this blog post. That being said, WinSCP's author (Martin Prikryl) takes donations. (And I think that it's a worthy cause; I like to support independent development work.)

WinSCP 5.5 Overview

When you open WinSCP 5.5, you will first see the Login dialog box, which will be empty until you add some sites to the list. The Login dialog allows you to create folders so you can categorize your sites, and the user interface is comparable to what you would expect in a Site Manager for other FTP clients.

Fig. 1 - The opening Login dialog in WinSCP 5.5.

When you are adding FTP sites, you have three choices for the protocol: FTP, SCP, and SFTP; you also have four choices for encryption: No encryption, TLS/SSL Implicit encryption, TLS Explicit encryption, and SSL Explicit encryption. (I'll discuss those later.)

When you open a site for which you did not save the password, (which I highly recommend), you will be prompted for your password.

Fig. 2 - The WinSCP 5.5 Password dialog.

Once your FTP site is opened, the main application window is displayed, and it resembles a two-column file explorer interface with local and remote folders, which you might expect from a GUI-based FTP client. (Note: WinSCP refers to this as it's "Commander" interface.)

Fig. 3 - Local and Remote Folders.

That being said, if you change your application preferences, you can change the user interface so that it uses a single-column file explorer interface with a folder tree, which might be useful if you would rather use the FTP client as a drag-and-drop repository. (Note: WinSCP refers to this as it's "Explorer" interface.)

Fig. 4 - Remote Folder Tree and Files.

WinSCP 5.5 has support for automation through .NET and COM, and documentation about automating WinSCP 5.5 programmatically is available on the WinSCP website at the following URL:

WinSCP .NET Assembly and COM Library

There are several detailed automation examples on the WinSCP website that are written in C#, VB.NET, PowerShell, JavaScript, VBScript, etc., and the documentation is quite good. If you need to do a lot of FTP scripting and you are looking for a good way to automate your FTP sessions, you might want to consider this FTP client.

If you don't want to write a bunch of code, you can also automate WinSCP from a command line, and the documentation about that is available on the WinSCP website at the following URL:

WinSCP Command-line Options

Another great feature about WinSCP is that it can be downloaded as portable executables, which makes it easy to copy between systems. This is a great feature for me since I like to keep a collection of handy utilities in my SkyDrive/OneDrive folders.

Using WinSCP 5.5 with FTP over SSL (FTPS)

WinSCP 5.5 has built-in support for FTP over SSL (FTPS), and it supports both Explicit and Implicit FTPS. To specify which type of encryption to use for FTPS, you need to choose the appropriate option from the Encryption drop-down menu for an FTP site.

Fig. 5 - Specifying the FTPS encryption.

Once you have established an FTPS connection through WinSCP 5.5, the user experience is the same as it is for a standard FTP connection. That being said, I could not find a way to drop out of FTPS once a connection is established, so FTPS is an all or nothing option for your sessions.

Using Using WinSCP 5.5 with True FTP Hosts

True FTP hosts are not supported natively, and even though WinSCP 5.5 allows you to send post-login commands after an FTP site has been opened, I could not find a way to send a custom command before sending user credentials, so true FTP hosts cannot be used.

Using Using WinSCP 5.5 with Virtual FTP Hosts

WinSCP 5.5's login settings allow you to specify the virtual host name as part of the user credentials by using syntax like "ftp.example.com|username" or "ftp.example.com\username", so you can use virtual FTP hosts with WinSCP 5.5.

Fig. 6 - Specifying an FTP virtual host.

Scorecard for WinSCP 5.5

This concludes my quick look at a few of the FTP features that are available with WinSCP 5.5, and here are the scorecard results:

That wraps things up for today's blog. Your key take-aways should be: WinSCP 5.5 is good FTP client with a lot of options, and it has a very powerful automation story. As I mentioned earlier, if you have to do a lot of FTP automation, you should really take a look at this FTP client.

Note: This blog was originally posted at http://blogs.msdn.com/robert_mcmurray/

I had an interesting question earlier today which I thought was worth sharing. One of my coworkers was trying to use the code sample from my Programmatically Flushing FTP Logs blog, and he was getting the following error:

Unhandled Exception: System.UnauthorizedAccessException: Access is denied. (Exception from HRESULT: 0x80070005 (E_ACCESSDENIED))
   at Microsoft.Web.Administration.Interop.IAppHostMethodInstance.Execute()
   at Sample.Main() in c:\Projects\FtpTests\Program.cs:line 25

I knew that the code sample in my blog worked perfectly when I originally wrote it, so I figured that my coworker must be doing something wrong. (And every developer has said "It works on my computer..." at one time or other.) But I decided to give him the benefit of the doubt, so I copied the source code from my blog into a new Visual Studio project and I ran it.

Much to my surprise, I saw the same error that my coworker was seeing if I didn't step the code through with a debugger.

When I stepped through the code in a debugger, I saw the following error message:

At this point I was thinking, "What the heck? I know this code was working before..." I started to wonder if we had released a breaking change to the FTP service sometime during the past two years, but then it suddenly dawned on me: I hadn't started the FTP service on my computer.

[Duh.]

That was the source of the problem: I usually have the FTP service configured for manual startup on my development computers, but the FTP methods to start and stop FTP sites and flush the FTP logs do not work when the FTP service is not running. Once both of us started the FTP service on each of our systems the problem went away.

I hope this helps. ;-]

Note: This blog was originally posted at http://blogs.msdn.com/robert_mcmurray/