Mapping IIS worker processes to pools

Once in a while you have to diagnose memory or performance issues with .NET applications. The first indication comes from the Windows task manager which may indicate a process hogging the CPU/memory or users may complain of excessive slowness. In my particular case, I recently examined an IIS6 worker process which was spinning at 20% and using 1GB of memory on a 32 bit Windows 2003 Server. Combine with user complaints that the system was extremely slow and unresponsive, it was something to look into. The investigation is still ongoing, but I am documenting the steps taken so far.

Step #1: Find out which application or applications are causing the problem. Using the process ID from the task manager, you can use one of the following methods depending on whether this is IIS6 or IIS7

1. On II7 the IIS Manager gives an option to locate running worker processes, the pools they belong to and the requests they are currently processing right from the GUI. 
2. On II6, this can get a little tricky. A little VB Script can be of help: IISApp.vbs. Since it is typically found in the system folders, you can pretty much run it directly from the command tool. Note that this script won't be found on IIS7 installs.

Step #2: The next step typically is to start tracking performance counters and see which ones may indicate problems. From the command prompt, type "perfmon". Using perfmon, you can turn on different performance counters. The slowness can be due to a variety of reasons. There could be a queue which is getting very long - a processor or network queue for example or system is doing too many context switches or there are too many interrupts or .NET garbage collection could be going on. The important point is to pick a place to start and find patterns. For example, I noticed that the processor queue length was relatively high and the context switches were very high in the system. The next step, it seemed to me was to try and isolate the process threads which were switching in and out and what was causing them to block so the OS would switch the thread out.

Perfmon shows multiple instances of the same process with suffixes like "#1", '#2' etc. This is not very helpful and brings us to Step #3 meanwhile in the digging process:

Step #3 The following article from Microsoft describes how to fix perfmon to print process IDs instead.

http://support.microsoft.com/kb/281884

Setting up Subversion on Windows

There are a dozen ways to set up Subversion on Windows. I have done it at least three or more time and the last two times, I noticed I used different methods. On the server I used last, I am now stuck with an extra Apache instance so I am writing this blog so that other people don't make the same mistake. In addition, once you set up SVN, you should set up nightly backups as well where the repository is backed up on an external drive. That has saved me at least once when the host Windows OS failed on us once crashing the hard disks with it. One day I know I am going to be back on this page looking through my notes setting up SVN for the fourth time.

In our company we use Windows predominantly as there are many more people who are familiar with managing it, backing it. There is more software which can be re-used as well inside the organization. If I used Linux, which I am obviously more familiar with in terms of using SVN or as a product development box, I am sure the installation, backup and maintenance will be a lot simpler but then I am going to be stuck with it :) 

The first step to installing SVN on Windows is to find an idle server or install one from scratch. A Windows 2008 Server, 32 bit would enough. Make sure you put on some partitions on it to store the repositories since these can become huge. I would say put 250-400 GB in a separate partition or drive and use RAID on this server. In my case when the server crashed, it took down the RAID controller with it making the hard drives unusable. So you must still do backups. 

Second step is to make sure there is no IIS running on this box, since we will use the port 80 for SVN. Also make sure other software which grinds the disk is kept to a minimum (like SQL Server etc). You will possibly install MySQL on this server if you keep reading my blog.

Now you can download SVN binaries from one of the partners - Collabnet or WANDisco. You will need to register on their site before you download the binaries and make sure you dont download a "bundle" which includes other software like Apache. We will install that separately, or at least I did. By downloading it separately, you can get the latest Apache version which will work for you. Dont try to set up SVN as a service or anything since we will be using it with Apache.

On my machine for example, SVN got installed at: C:\Program Files\WANdisco\Subversion. Just note down where it gets installed on your system.

Copy Paste not working inside Windows RDP?

You know the solution to the problem! You remote into the Windows server and paste the script, excited. Then you paste it again and then one more time only to find out nothing is getting transferred. Then you go back and try it again. It is frustrating isn't it. Especially since at the same time everyone else is having no issues doing the same thing. Well, next time you face this problem, launch the Task Manager and look for a process called "rdpclip.exe"running under your user id. Kill it and then restart it using the Run prompt. Things should magically start working. 

You should know that this process only runs if someone else is logged onto your computer using remote desktop. It is a server side process. This process has some known issues as well like sometimes it can eat up too much memory and CPU. When you exit out of a remote desktop session with something stuck in your clipboard then you run the chance of confusing this process. In any case it is safe to kill at the end of the remote session before logging off.

How to run command on one windows PC from another

If you don't know about it yet, psexec is a nice utility which allows you to run a windows command on one PC from another. Here is an example:

psexec \\remote-machine C:\RunBatchFile.bat

Run a Unix command remotely from Windows

After helping a customer numerous times by diagnosing a failed cron job, I was looked up how to call the job from Windows. This turned out to be interesting given it was a command which could only be run from sudo. Here are the details:

1. First you need a program called plink which is part of the putty distribution. You can download putty from  http://the.earth.li/~sgtatham/putty/latest/x86/putty-0.60-installer.exe
2. Create a text file with your commands somewhere on the PC. For example:
1. /usr/bin/sudo su -
   
3. Either set up your path environment variable on Windows or simply go the putty install folder (e.g C:\Program Files\Putty).
4. plink -t zz000412@PRODDOCS-VORA -m

psexec is another command to run from windows to call a script on another windows box.

Mediawiki + Bugzilla + SVN + SCMBug + Tweaks = Software Manager's recipe

It was fun trying to integrate SVN, Bugzilla and Mediawiki using SCMBug and BugzillaReports. The integrated system is really an efficiency booster and considering how easy it was to set up the value/effort ratio was well worth it. I am listing the steps needed to set this system up on Windows here which makes things very interesting sometimes. Linux install is similar and simpler.

1. The first step was installing Apache, PHP and Active Perl on Windows. This is straightforward and well documented.
2. Install MySQL
3. Install Bugzilla. This is straightforward as well. Bugzilla uses Perl.
4. Install Mediawiki.
1. Turn off caching in Mediawiki. With caching on, you get stale data on bugs which can lead you to make wrong decisions.
5. Install the Mediawiki - BugzillaReports Addon. This gives the ability to write short scripts in mediawiki which query bugzilla and present data on a web page as a reports. This itself takes you from zero to fifty.
6. Set up backup scripts for MySQL.
7. Install SVN
1. Collabnet worked out pretty well.
2. Set up Backup scripts for SVN
8. Create custom fields in Bugzilla for recording code change list
9. Install SCMBug
10. Customize code in SCMBug. This will help:
1. Format the regular expression for parsing out the bug ID from the comments.
2. Change code to post changes to custom field instead of comments. SCMBug by default posts changes to Bugzilla comments which can be very annoying to users.

Getting G.729 to work on Windows in the cheapest and most effective way

This is a very common question people ask for G.729: G.729 is the most common codec supported by a variety of VoIP providers and gateways. Though there are lots of better and free codecs around, G.729 is still the most preferred (and expensive) codecs around. The G.729 IPR billing is controlled by a company called SIPRO and if your product is sold with the G.729 codec embedded in it (even if it never gets turned on), you must pay the order of $5-$10 bucks per port to the consortium. SIPRO takes care of disbursing the payments around to the IPR owners. However, this is a well hidden secret for some reason. The common path taken by a lot of product vendors for supporting G.729 is that they license a complete media package from companies like VoiceAge. The media package bundles a voice processing layer which integrates G.729. The cost of the package is in the tens of thousands of $$. Here is an option which is much cheaper and works as good or better since it gives you full control of what is going on at least on Windows:

1) Get the Intel IPP development package from any of the resellers. This costs a few hundered bucks. The IPP is basically a development package which takes the original G.729 code published in the ITU standard and makes it work on Windows with some mods and optimizations.

2) As and when you sell licenses of your product you get a deal done with SIPRO and pay few bucks on a per seat basis. This avoids a lot of fees and proves to be much cheaper.

DI almost forgot to add that its as good voice quality as well!

SIPX on Windows

SIPX was one of the first open source SIP stacks we played around with on Windows. SIPX is supported by a strong community of developers and the code baseline came from Pingtel. There is a company called SIPEZ started by Daniel Petrie, a well known SIP Guru which supports SIPX. SIPX is used in a wide variety of commercial applications and is licensed under LGPL which is very attractive. I knew Daniel from IETF and often contacted and got his help during our evaluation phase. Based on what we found, SIPX is a great choice if you want to build a softphone very quickly since it has a lot of applications built into it and a lot of third party software hooks as well. However, we couldn't hire SIPX developers fast enough to support customer requirements and it proved harder to change and support in a cost effective way. Our next stop was brief and we looked at reSIProcate. reSIProcate, while supported on Windows is really meant for server side applications - at least that is the impression I got by talking to the SIP community at large. My team never really played around with it too much and we moved on to something called PJSIP. PJSIP has a very restrictive license (licensed under GPL) compared with SIPX and reSIProcate. That was one reason it was way down in our list though my gut feeling always was that it would have been a quicker fit (that's a lesson there for me - always feel out your guts properly in this phase :). PJSIP was supported by a much smaller community of developers and was not that widely known or deployed. One of my friends, Dr Henry Sinnreich (now at Adobe) actually introduced PJSIP to me. Overall we found PJSIP to be much more malleable and suitable to what we wanted to do both from a tactical and long term perspective. It also has a much smaller footprint compared to SIPX. However applications are much fewer. Its much more suitable for embedded systems.

Making SIP work on Windows

Few months back I did some research on open source Windows based SIP stacks and the sheer number of options available excited me. We explored several stacks: SIPX, PJSIP, ReSIProcate. In this article and few others I will put down my insight into what we learned during an ordeal which lasted for six months, talking to numerous people, vendors and component providers. Our goal was to have a stable, clean and extendable SIP API in our application well integrated with some good quality sound processing software. That was just 50% of the application. The other 50% was to do with rich GUI and data processing functionality for supporting a SaaS application on Windows. Nothing like this existed in the market. Our application was also a smart SIP client which implemented some new Peer-2-Peer techniques to avoid expensive network components which were inscalable and hard to manage (and thus supported the SaaS based model very well).