Galin Iliev's blog

Software Architecture & Development

Finally up & running

As you might noticed this week the site was down. This was due to hosting upgrade to favorite IIS7 and some issues with dasBlog and one of it’s controls. Luckily I was able to fix it and also upgraded to dasBlog v2.2.

Hopefully there won’t be any downtime next… 10 years (next upgrade should be handled nicely :) ) Not like I am planning uptime 5 nines – 99.99999% :)

Now seriously:

If you planning your servers’ availability you might want to check how Microsoft.com Ops are doing it by taking a consultation from them for free. Why?! Because these are the results (as of 2005) – just imagine what are now:

*.Microsoft.com:

  • 3 Data Centers
  • 1606 Data Center Servers
  • 506 Servers in Labs
  • 111 Web Sites
  • 1069 Databases
  • 1000’s of Web Applications
  • 80+Gbit/sec Network Traffic

WWW.Microsoft.com

  • 13 million unique users/day
  • 70 million page views per day
  • 10,000 requests/sec, 300 concurrent connection on 80 servers
  • 350 Vroots

Windows Update/Download:

  • 150 million unique client scans/day
  • 12,000 ASP.NET requests/sec
  • 500K concurrent connections
  • 1 Billion Downloads and 750K client installs in 2 weeks (April 2006)
  • 4Gbit/sec Web Site Egress (Web Pages Only…No Downloads)
  • 20+ Billion Downloads in 2005…Routinely 150M+/Day

IIS 7.5 features lists

Internet Information Services 7.5 (IIS)

The great Internet Information Services (IIS) has it’s v7.5 with following features in Windows Server 2008 R2

  • PowerShell provider for IIS 7 has more than 50 new cmdlets
  • Administration Pack extensions: Database Manager (SQL Server management within IIS Manager), Configuration Editor (generate scripts with a GUI to automate administrative tasks), IIS Reports, Request Filtering (HTTP filtering, URL rewriting, etc.)
  • One-click publishing in Visual Studio 10
  • Web Deployment Tool (formerly MS Deploy): Deployment, management, and migration of Web applications, sites, and entire servers
  • Configuration Tracing: track configuration changes to IIS and applications
  • New performance counters
  • .NET support for Server Core
  • WebDav integration (was available before as a separate extension)
  • URLScan 3.0 integration: restricts the types of HTTP requests (was available before as a separate extension)
  • FTP server services: integrated in the IIS administration interface; new .NET XML-based *.config format; virtual host names for FTP sites; improved logging
  • Integrated extensions: new kind of extensions that appear to be an integral part of IIS

(via 4sysops.com article)

Time for reinstall… .NET Framework

I had interesting issue today: I was on urge to complete a task when I saw a strange exception – you know – the Murphy's law. It hit at the most inconvenient time. To be even more interesting I’ve deployed my work on two different servers and worked fine.

But on the most important one I got this exception when I tried to hit WCF service:

System.MissingMethodException: Method not found: 'Void System.ServiceModel.Diagnostics.EventLogger.UnsafeLogEvent(System.Diagnostics.TraceEventType, System.ServiceModel.Diagnostics.EventLogCategory, System.ServiceModel.Diagnostics.EventLogEventId, Boolean, System.String[])'.
   at System.Runtime.CompilerServices.RuntimeHelpers.PrepareDelegate(Delegate d)
   at System.AppDomain.add_UnhandledException(UnhandledExceptionEventHandler value)
   at System.ServiceModel.ServiceHostingEnvironment.EnsureInitialized()
   at System.ServiceModel.PartialTrustHelpers.PartialTrustInvoke(ContextCallback callback, Object state)
   at System.ServiceModel.Activation.HttpModule.ProcessRequest(Object sender, EventArgs e)
   at System.Web.HttpApplication.SyncEventExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()
   at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously)

Very strange… and search engines doesn’t know much about the problem…

Luckily this was not used heavily and I could reinstall .NET Framework 3.5. And this solved the problem.. Now back to more productive things…

VS2008 Web Setup Project and Win2008

I had interesting experience today. I tried to create a web setup project for one of my recent projects. As you know it is pretty straightforward: From Visual Studio 2008 File –> Add –> New Project –> Select Setup wizard.

image  

and just add output from existing project.
Then press Ctrl+Shift+B (Build ) and you’re ready… But not in Windows Server 2008. When I decided to test this newly made installation package I hit the ground with single dialog showing this message:

"The installer was interrupted before ApplicationName could be installed. You need to restart the installer to try again.

Click "Close" to exit."

 

Being experienced installer package developer I knew what I had to do: run the installer with verbose logging.

by executing this line:

   1: msiexec /i Installer.msi /lv detail.log

And I had nice 57 KB  file to read in wonderful notepad. And the problem action quickly appeared:

   1: Action start 16:22:53: WEBCA_SetTARGETSITE.
   2: MSI (c) (B4:4C) [16:22:53:082]: Note: 1: 2235 2:  3: ExtendedType 4: SELECT `Action`,`Type`,`Source`,`Target`, NULL, `ExtendedType` FROM `CustomAction` WHERE `Action` = 'WEBCA_SetTARGETSITE' 
   3: MSI (c) (B4:48) [16:22:53:082]: Invoking remote custom action. DLL: C:\Users\ADMINI~1\AppData\Local\Temp\MSIFB61.tmp, Entrypoint: SetTARGETSITE
   4: INFO   : [11/11/2008 16:22:53:097] [SetTARGETSITE                           ]: Custom Action is starting...
   5: INFO   : [11/11/2008 16:22:53:097] [SetTARGETSITE                           ]: CoInitializeEx - COM initialization Apartment Threaded...
   6: ERROR  : [11/11/2008 16:22:53:097] [SetTARGETSITE                           ]: FAILED:    -2147221164
   7: ERROR  : [11/11/2008 16:22:53:097] [SetTARGETSITE                           ]: Custom Action failed with code: '340'
   8: INFO   : [11/11/2008 16:22:53:097] [SetTARGETSITE                           ]: Custom Action completed with return code: '340'
   9: Action ended 16:22:53: WEBCA_SetTARGETSITE. Return value 3.
  10: MSI (c) (B4:4C) [16:22:53:097]: Doing action: FatalErrorForm
  11: Action start 16:22:53: FatalErrorForm.

Seeing WEBCA_SetTARGETSITE means that the installer was trying to set the destination. Having a web setup this means IIS was asked for “Default Web Site” and this call must be the one that fails. But why!? I have Web Server Role installed on the machine:

image

I and remembered from my MS DevDays 2008 IIS talk  – you still can use old (pre v7) IIS management tools with IIS7 as long as you have “IIS 6 Metabase Compatibility” role service installed.

image

I installed this role and … Voila!!! it works!!!

Virtual Machine's Network Adapter Hangs

I recently moved Galcho.com (and this blog) on a new Virtual Machine kindly provided by my friend Nanio Nanev and his system administration company PrimaNet Consult LTD.

The VM has Win2003 Web edition SP2 and it is very fast ( as it is hosted on monster hosting server ) but there is one nasty issue we are fighting with: The network adapter that is connected to WAN - external network and has real static IP address - hangs once in a while.

How is possible Intel 21140-Based PCI Fast Ethernet Adapter (Generic) Network adapter on Virtual machine to hangs?!?!

I was able to connect using internal network adapter and after disable and re-enable WAN it was fine for another 3-4 hours.

I've found a way to this by script - by using DevCon - command-line utility functions as an alternative to Device Manager (direct download link).

Using it this simple script does the job:

C:\Install\devcon disable PCI\VEN_1011&DEV_0009&SUBSYS_21140A00&REV_20\3&267A616A&0&50
C:\Install\devcon enable PCI\VEN_1011&DEV_0009&SUBSYS_21140A00&REV_20\3&267A616A&0&50


Note that device class can differ so the question "How did you get these?" comes naturally. Here is how you can list all devices from setup class:

c:\install\devcon listclass net


And here is the result in my case:

image

So doing this reset on certain period helps now but this is not the smartest solution. Does anyone have another idea?

Maintain Database Indexes

It is wide known that creating index on table column can speed up queries that has this column in it's Where clause. Table indexes are binary trees in most cases and they are stored in pages similar to stored data itself. Over time data changes which cause index changes and it require some sort of maintenance to keep database optimized and running as fast as possible.

Over time these modifications can cause the information in the index to become scattered in the database (fragmented). Fragmentation exists when indexes have pages in which the logical ordering, based on the key value, does not match the physical ordering inside the data file. Heavily fragmented indexes can degrade query performance and cause your application to respond slowly.*

This can be fixed by either rebuilding index (by dropping existing and create new one) or reorganize it (or defrag it).

Rebuild indexes

Rebuilding indexes can be done by either one of these

I won't cover the details as you can look them up on MSDN. Just my favorite way is like following:

ALTER INDEX ALL ON Person.Address REBUILD WITH (ONLINE=ON, FILLFACTOR = 80, SORT_IN_TEMPDB = ON, STATISTICS_NORECOMPUTE = OFF);

 

The advantage is this operation is online - meaning you can query table during index rebuild.

Reorganizing indexes

Reorganizing an index defragments the leaf level of clustered and nonclustered indexes on tables and views by physically reordering the leaf-level pages to match the logical order (left to right) of the leaf nodes. Having the pages in order improves index-scanning performance. The index is reorganized within the existing pages allocated to it; no new pages are allocated. If an index spans more than one file, the files are reorganized one at a time. Pages do not migrate between files.

Reorganizing also compacts the index pages. Any empty pages created by this compaction are removed providing additional available disk space.*

 

There are two ways to perform index reorganization:

Again my preferable is this:

ALTER INDEX ALL ON Person.Address REORGANIZE;


This is also online operation.

Note: Although both operation stated above should be online I've applied it on big tables (above 140M records on ~60 GB in two tables) and of course it was pretty I/O intensive which caused some delays in performed queries. Having in mind that default CommandTimeout in .NET Class Library is 30 seconds and application writing at least once per minute creates very challenging DB to maintain. Possible solution would be using MS SQL Server 2008 Resource Governor. Unfortunately the server was MS SQL 2005...

How to detect fragmentation

In order to apply techniques described above fragmentation should be detected. For this comes a new DMV (Dynamic Management View) sys.dm_db_index_physical_stats - that gives us fragmentation in percent (avg_fragmentation_in_percent).  

These are the recommendations depending on returned value in avg_fragmentation_in_percent column:

avg_fragmentation_in_percent value

Corrective statement

> 5% and < = 30%

ALTER INDEX REORGANIZE

> 30%

ALTER INDEX REBUILD WITH (ONLINE = ON)*

 

Using sys.dm_db_index_physical_stats could not be very useful when used by itself so I prefer using it together with system tables sys.tables and sys.indexes:

---=== get index fragmentation
SELECT a.index_id, t.name as TableName, i.name as IndexName, avg_fragmentation_in_percent
FROM sys.dm_db_index_physical_stats(NULL,NULL,NULL,NULL,NULL) AS a
INNER JOIN sys.indexes AS i ON a.object_id = i.object_id AND a.index_id = i.index_id
join  sys.tables t on t.object_id=i.object_id
ORDER BY avg_fragmentation_in_percent DESC

 

The result is like this (executed in AdventureWorks):

image  

 sys.dm_db_index_physical_stats can take time to execute so if you want to view all indexes with the table name this can be used:

select t.object_id, t.name as TableName, i.name as IndexName, i.type_desc as IndexType  
from sys.indexes i 
join  sys.tables t on t.object_id=i.object_id
where i.object_id >1000 
order by t.create_date asc


Which return following result:

image

And it can be used to generate detailed T-SQL queries for reorganizing indexes one at time:

select  'ALTER INDEX ' + i.name + ' ON ' + t.name + ' REORGANIZE;'
from sys.indexes i 
join  sys.tables t on t.object_id=i.object_id
where i.object_id >100 
order by t.create_date desc


producing

image

Summary

So far we took a look at following

  • Detect index fragmentation
  • Rebuild indexes
  • Reorganize indexes
  • Use T-SQL to generate T-SQL to maintain indexes.

I hope this helps.

* quoted from MSDN article Reorganizing and Rebuilding Indexes.

Microsoft.com on IIS7 performance data

You know www.microsoft.com, right? :) This the corporate web site of the biggest software company and the very wanted target of every hacker (or wanna-be hacker). When this website (or some other Microsoft websites like www.msdn.com) is down or show an unexpected error there are screenshots on the web (and blog posts) and this become a news of the day in software world :) - or at least on web dev world.

So now imagine of you are decision maker for hosting platform!? or hardware behind it?! or setting a bandwidth :)?! There is very little room for mistakes, huh?

And still www.microsoft.com is hosted on IIS7 .. since Beta3 (post is from June 15th, 2007). When Microsoft trust enough on IIS7 and host such important site on it, why you can't?

There is no doubt that the configuration behind Microsoft.com is interesting so here is it:

=============== Microsoft.com configuration ====================

Hardware:

  Model: HP DL585 G1 (4 dual-core CPUs)

  RAM: 32GB 

OS:

  Windows Server 2008 RTM (Build: 6.0.6001.18000) Enterprise version x6

Cluster:
  Number of clusters: 4 (in multiple datacenters)
  Machines in each cluster: 20
  Total machines: 80
Load Balancing:
  Hardware load balancing solution is used. The load balancing algorithm we are using is based on “Least Current Client Connections” to each load balanced member server of the cluster (not round robin, or other any other load balancing algorithms). The hardware load balancer will maintain the same number of current client connections to each member of the cluster. So if a W2K8 server is completing web requests faster than a W2K3 server, the load balancer will send more traffic to the server W2K8 RTM server.

======================================================

Recently some performance data has been released on TechNet and here is what it says:

  • Win2008/IIS7 process more Requests per second(RPS) than Win2003/IIS6.
  • Due to #1 Win2008's CPU is more utilized.
  • As Win2008/IIS7 is performing better the load balancer send more requests to it.

IIS7-vs-IIS6

Server Efficiency (RPS/ CPU %) – Efficiency of serving live web platform traffic

W2K3 SP2 4.36 “requests per CPU cycle”

W2K8 RTM 4.84 ~ 10.9% increased efficiency

CPU Utilization (%)

W2K3 SP2 44.8%

W2K8 RTM 52.8% ~ 17.9% degradation (This is impacted by the increased RPS the W2K8 servers are handling)

Web Service – Total Methods Requests/Sec (RPS)

W2K3 SP2 194

W2K8 RTM 255 ~ 31.4% more traffic is being sent to the W2K8 RTM servers

Web Service – Current Connections

W2K3 SP2 280

W2K8 RTM 294 ~ 5% increase

Load Balancing – Current Client Connections

W2K3 SP2 116

W2K8 RTM 116 Equal – as the hardware load balancer maintains the same amount of outstanding open client connections.

.NET CLR Memory – % Time in GC

W2K3 SP2 1.1%

W2K8 RTM 2.5% No significant degradation in “Time in GC

 

Source: Microsoft.com Operations blog post on TechNet.

IIS7 is really next generation web platform...

ASP.NET Configuration @ Microsoft.com

I came across an article which covers some details of how ASP.NET web sites hosted by Microsoft are configured:

Key things are:

  1. Set the Compilation Switch Appropriately
  2. Use Medium Trust in ASP.NET 2.0
  3. Restrict Download of Specified File Types
  4. Be Careful When Adding Assembly References
  5. Remove Manually Set MaxConnection Values
  6. Beware of Unhandled Exceptions
  7. Ensure Proper Proxy Server Configuration
  8. Do Not Display Custom Errors to Everyone
  9. Know When to Enable Tracing
  10. Disable Session State Web Farms

Read full article for detailed explanations.

My Sessions at Microsoft Days 2008 in Sofia, Bulgaria

MS Days 2008 in Bulgaria is in history now and I could say I had a nice two days. There were many lecturers (about 50) and 72 sessions in 6 tracks.

For those who missed my talks or are interested in slides here are summary of the sessions:

LINQ to XML - Data Access Technologies

This session was focused on the new API from XML team for .NET languages. I gave a side by side comparison between traditional DOM vs. LINQ to XML regarding those most common actions:

  • Create XML
  • Traverse XML
  • Transform XML

I covered also VB9 Literals. At the moment I started talking about VB I was thinking people would throw rocks at me (and some really considered that option:) ). But at the moment when repeated some of demos with VB9 code the audience was very impressed and they forgot about those rocks in their pockets. Even there were initial brainstorming whether same things can be implemented in C# with custom code.(Unfortunately this is a compiler feature and we cannot do it very easily).

Another thing I mentioned was LINQ to XSD.

I've decided that people will understand my points better if I write code in front of them instead of just explaining it. This is also more challenging :). I think it went well...

Here are the downloads:

IIS7 for IT Pros

IIS7 is the most interesting feature in Windows Server 2008 and I already had some talks about it. In this talk I covered (from administration perspective) following key topics:

  • What is missing in IIS 6.0
  • IIS7 module architecture and it's benefits
  • New .NET-like configuration files and metadata
  • Delegated Administration
  • Shared Configuration
  • Tracing and Diagnostics

The things I've demonstrated are:

  1. New tools - new management console as well as APPCMD command-line tool
  2. Richness of new error pages and generated trace file - it is whole HTML+JS application built with XML & XSLT with incredible amount of information.
  3. WCAT stress test with view of live requests on the server.
  4. Analyze server and site load using IIS7 Admin Pack features.

And here is the presentation: MS PowerPoint 2007 format (0.98 MB)

Any feedback is very welcome.