WebTest brings down Visual Studio.

I started a WebTest that fires 200,000 requests to the web server. I left the test running overnight. On getting back to work today, I found that the Visual Studio instance running the test has crashed!! To my dismay, Visual Studio did not even generate a trx file, which would have given me some idea as to how many requests it was able to shoot and what might have caused it to crash.

This experience has made me wiser though. I have now kicked off the webtest using command line.

mstest /testcontainer:testproject2.dll /test:WebTest1Coded /runconfig:localtestrun.Testrunconfig /resultsfile:testResults.trx

The test has completed and I have a 639 MB trx file lying on my disk, which always brings down Visual Studio when I try to load it. So it is now as good as not having the file to analyze. Take this lesson from me (for free) - Do not try to run a WebTest that will generate a trx file of such a size that it cannot be opened later in VS for analysis. I could not find anything on msdn that tells me of the upper limit on size of trx file. Can we have that, Microsoft?

Instead of trying to find a fault with Microsoft, let me see what could have gone wrong in the code. The WebTest code was firing 200,000 requests in a for loop with each request getting a different aspx page. I think I should remove the for loop, parameterize the web page and create a load test based on this webtest. Let me check this. Will update this space in a few hours :)

Load testing Excel Data Transfer to SQL Server 2008

Ever tried to transfer 1 million rows from Excel to SQL Server using the DTS Wizard in SQL Server 2008? Well I did and here is what I observed. My file had 3 columns in it and the total size of the file on disk was 37 MB. After I specified the Excel file as the source of data and clicked the Next button on the DTS Wizard, the memory consumption of DTS Wizard went up and up till it reached 1.5 GB. It was only then that it presented me with the select destination screen and the memory footprint went down. When I was done with all the steps and started the transfer, it again touched 1.5GB at the step named "Source Connection". I waited for it to reach the step named "Copying..." but I lost my patience and clicked on the Stop button. Inspite of that, the DTSWizard continued execution and I ended up killing it from Task Manager. I will try it again some other day when I have enough time to stare at the screen or keep it running overnight. I think it would be better to write my own program that uses SQLBulkCopy with some optimizations.

Just a thought before I hit the sack. Wouldn't it be cool, if an application like DTS Wizard, would tell you "I am going to take this much memory, this much time for completing this task. Are you willing to continue?" :). If I select not to continue, will it present me with a list of alternatives to do the same task and direct me to a site that gets me in touch with the smart minds @ MS. Any takers at Microsoft?

ASP.NET SQLCacheDependency challenging scenarios

There is an ASP.NET application using VirtualPathProvider to return web pages from the database. The VPP uses SQL Server Query Notification (SQLCacheDependency) to get notified if pages change in the database. Hence for every web page request, an entry goes into the database registering w3wp.exe, which handles the page request, as a subsriber to page change notifications.

Lets assume that the application has handled 1 million requests in a day. So there are 1 million subscriptions in the DB registered against the process id of w3wp.exe (pid:1018). Now lets take a look at the list of scenarios which must be considered by the development team.
Case 1: Application pool recycled and then Page is changed in the DB
The subscriber process w3 with pid 1018 disappears. SQL Server will attempt to send a change notification but will fail and log the error message. User is very likely to see an older version of the page. Disk space utilization will increase.

Case 2: SQL Server recycled.
SQL Server will attempt to send out 1 million change notifications to the w3wp.exe process with pid 1018. Memory utilization of SQL Server will shoot up. Users may experience slow applications response.

Case 3: SQL Server and App pool recycled.
SQL Server will error out when trying to send out change notifications to the w3wp.exe process. Memory utilization of SQL Server & disk space utilization will shoot up. Users may experience slow applications response.

All 3 cases present an interesting engineering problem to solve! Will blog more about the possible solutions. Keep watching this space.

IIS LogonMethod property

I am trying to figure out what the LogonMethod Property associated with an Application Pool means(/LM/W3SVC/AppPools, /LM/W3SVC/AppPools/DefaultAppPool/application_pool_name) means. Based on what I have found so far, I am posting some notes.

We associate credentials of a domain or local account with an application pool. Before a w3wp.exe process is started using the credentials of this account, the system calls the LogonUser function. This function accepts a parameter named "dwLogonType". I think that the value of this parameter is set based on the value assigned to the LogonMethod property in IIS Metabase. By default, IIS 6.0 uses NETWORK_CLEARTEXT as the value for the LogonMethod property. If the function succeeds in authenticating the user, it returns a handle to a token that represents the app pool account and a w3wp process is created to run in the context of this account. When using NETWORK_CLEARTEXT as the value for the LogonMethod property, the username and password of the app pool account are preserved in the DLL containing the authentication logic and used for establishing connections to other network resources during the process lifetime.

References:
1. David Wang's Post
2. MS Support article

IIS MaxBandWidth property

IIS MaxBandWidth property can be set to a value specified in bytes per second indicating an upper limit on the amount of bandwidth to be consumed by the service. By setting this value to a rightly calculated value, we can simulate scenarios of broadband or dial up access to the site.

For quick reference, I have reproduced the calculation steps given by Chris. Thanks Chris!
In order to accurately simulate a 56K dial-up connection, we must convert Kbits into Kbytes, remembering also to account for modem sync information. Since there are eight bits in a byte, you’d think we’d simply divide 56 by eight to determine the number of kilobytes to set the throttle to. However, don’t forget that part of the 56K bandwidth is dedicated to keeping the connection up and not used for data. As such, we actually divide 56 by 12 to arrive at the correct number. We then multiply by 1,024 to turn Kbytes into bytes. This results in 4,778 and change. You can keep the change.

Windows Ports and Services

I am engaged in creating a deployment topology for an ASP.NET application. One of the important things to consider is the list of services to run on the servers and what ports need to be kept open without compromising security of the system. I found this page on technet to be a very handy reference.

Adding a few more references:
1. List of IIS 6.0 Metabase properties. You can find the same list on your server in C:\Windows\Help\iismmc.chm.
2. System Definition Model (SDM) types that differ from their counterparts in the IIS metabase
3. How to Modify the Cache-Control HTTP Header When You Use IIS
4. Evolution of IIS architecture

.NET Runtime versions.

Aaron Stebner's blog has a table that lists all .NET versions released so far. This table is very useful if you are creating a Logical Data Center diagram in Visual Studio Team Architect and want to specify the 4 part version number of the Target Framework. Thanks Aaron! :)

Selective proxy setting in WebTests

In developing a WebTest, I came across a scenario where I wanted a set of requests to use a proxy and another set of requests to bypass the proxy. A WebTest has a property named Proxy as shown in the figure below, which will be used by the Visual Studio Test Engine when sending out Http Requests.
In my WebTest, I wanted the 1st two WebRequests to bypass the proxy and the 3rd one to use a Proxy. Here I found it useful to create a WebTestRequestPlugin that is later associated with a WebRequest to set a proxy in the PreRequest method.


Here is the code for the WebTestRequestPlugin.

public class ProxySetter : WebTestRequestPlugin

{

    string proxy = null;

 

    public string Proxy

    {

        get

        {

            return this.proxy;

        }

        set

        {

            this.proxy = value;

        }

    }

 

    public override void PreRequest(object sender, PreRequestEventArgs e)

    {

        if (!String.IsNullOrEmpty(this.proxy))

        {

            e.WebTest.Proxy = this.proxy;

        }

        else

        {

            e.WebTest.Proxy = null;

        }

    }

}

Importance of specifying the right FileAccess option.

When debugging an application that accepts a filename as input, I came across the following exception
System.UnauthorizedAccessException: Access to the path 'C:\test.txt' is denied.

The line at which it failed was
FileStream fs = new FileStream(path, FileMode.Open);

I checked my access permissions on the file and everything was perfect. Except that I noticed that this file test.txt was marked Read Only. When I changed the above line to include the FileAccess option
FileStream fs = new FileStream(path, FileMode.Open, FileAccess.Read);
the code executed successfully!

So moral of this is that wherever possible we should specify the FileAccess option for file operations.

Using .NET AutomationElement classes from Console Application.

When using .NET 3.0 UIAutomation classes in a console application, I was surprised when a call to AutomationElement.FindFirst returned a null value. The same method when called from a .NET WinForm application returned a valid AutomationElement.

If you encounter a similar issue, please bear in mind that UI Automation APIs only work from a STA thread. So if you ever want to get UIAutomation run from the console application, remember to decorate your Main function with the STAThreadAttribute.

Ref: http://forums.microsoft.com/MSDN/ShowPost.aspx?PostID=1706155&SiteID=1

SQL Server Reporting Services support for Firefox.

For a list of Reporting services features not supported in Firefox, check out http://msdn.microsoft.com/en-us/library/ms156511.aspx

Reading a INI file

A win32 function named GetPrivateProfile can be used to read an INI file.

Ref: http://msdn.microsoft.com/en-us/library/ms724353(VS.85).aspx

.NET Colors visual reference

I needed to visualize the different colors available for setting the BackColor and ForeColor properties of some controls. In my search, I came across this page. Thanks to the person putting this up for use of everybody. :)

Better Together Microsoft Project Plan + TFS.

Today, I learnt how to synchronize my MPP with TFS work items. I must confess that it is a delightful feature for any project manager!

You can access this productive feature from the Team menu of a MPP. Select Get Work Items and have your mpp populated with work items created in TFS. If you are not too happy with the field mappings, Microsoft provides a nice command line utility named TFSFieldMapping which resides in "%ProgramFiles%\Microsoft Visual Studio 9.0\Common7\IDE". Follow the steps given here and here to customize the mappings for your project.

Simple :)

Updating TFS Warehouse manually.

To update TFSWarehouse manually, you must access the machine hosting TFS and browse to
http://localhost:8080/Warehouse/v1.0/warehousecontroller.asmx

Then invoke the Run operation. This operation executes asynchronously. To see if the warehouse update is complete, invoke the operation GetWarehouseStatus. Simple!

Improving Website performance

While studying performance considerations for website design, I came across this site

http://developer.yahoo.com/performance/

that has good tips for developers to improve website performance.

Mobile Browsers

While researching on usage of Mobile Browsers, came across two interesting sites
1. Mobile browser statistics
2. Mobility checker

Logging and Performance.

What is the impact of logging on performance? If my ASP.NET website logs information about requests to a log file, the log file will turn out to be a resource that will get concurrently accessed. A lock placed on the log file will significantly impact ability of the server to concurrently process requests. Logging frameworks like log4net handle this very smartly.

Frequently used TFS commands

Some of the TFS commands I use frequently include
1. tfsdeleteproject
2. witimport
3. witexport
4. tf workspace
5. tf status: To check who has checked out a file Default.aspx, run the command
tf status Default.aspx /user:*

SqlBulkCopy vs SSIS Bulk Insert

I recently wrote a program to move data from a CSV file into a SQL Server table using SqlBulkCopy.
The CSV file had data like
0,0,35.6,-122.5,1,
0,1,35.61188,-122.4438,1,
0,2,35.61979,-122.4062,1,
.
.

Below are some of my observations.

Trailing Delimiter Handling
With SqlBulkCopy, the data was successfully moved to SQL Server.

But when I used the SSIS Bulk Insert Task, the following error was reported

Bulk load data conversion error (type mismatch or invalid character for the specified codepage) for row 1, column 5
Bulk load data conversion error (type mismatch or invalid character for the specified codepage) for row 2, column 5
.
.

The fix for this was to remove the trailing ',' character at the end of each line in the CSV file as shown below and then perform SSIS Bulk Insert.
0,0,35.6,-122.5,1
0,1,35.61188,-122.4438,1
0,2,35.61979,-122.4062,1
.
.

Speed
I did not observe any significant difference in the data load speed of SqlBulkCopy & SSIS Bulk Insert task. I will post performance numbers at some later time.