Subscribe to
Posts
Comments

TL;DR The Asp.net session ID in the database uses the Site ID from IIS as part of a composite key. Ensure the IIS Site ID is consistent in a web farm.

The website I work on needed to use a RadCaptcha recently on a form. Although it was configured as per the Telerik article to use out of process session state (SQL for us), it would occasionally show a grey box instead of the captcha.

After some investigation with Fiddler we found that one of the three web servers couldn’t share session state with the other two. I.e. a capture generated on server 1 couldn’t be read by server 2, but could be read by server 3. And the inverse of this was also true. Machine keys were already being shared between the servers so it wasn’t a decryption problem.

Monitoring the AspSession database watching session creation we found that server 1 and 3 were creating the same session ID whilst server 2 was creating a different but very similar ID. We saw the following IDs in the ASPStateTempSessions table in ASPState database:

3jt3wvhazn22rcliw1vyij3h2d3aafb7
3jt3wvhazn22rcliw1vyij3h2d3aafb5

After a bit of investigation I found this article which describes how the SessionID is made up of the Session ID + Application ID. The Application ID is a hash based on the AppName which is based on the metabase path of your IIS site.

As our servers run multiple sites and we’d brought these sites online in a different order, the Site IDs had got out of sync. We found the following values.

server 1 => /lm/w3svc/7/root
server 2 => /lm/w3svc/5/root
server 3 => /lm/w3svc/7/root

Server 1 and 3 AppName’s hashed to the same value, but server 2 didn’t and couldn’t therefore find the session data for the RadCaptcha.

I changed the IIS Site ID using IIS admin under Site => Advanced Settings as below and it fixed the problem. Note: this will recycle your app pool.

The site I work on fetches significant amounts of data on-demand from a remote data centre via a JSON feed. In order to provide resilience against the site being unavailable I implemented Redis on Windows as a read-through cache. (None Windows operating systems were not permitted at the time). I also implemented a refresh-ahead cache service which would ensure ‘hot’ content was always fresh.

Although the solution worked I was recently asked to look at replacements for several reasons

  • Redis on Windows wasn’t a supported platform
  • We couldn’t cluster it trivially as members within a distributed Redis cluster were not equal so some would need to be read slaves and some write masters. This would introduce more complexity to our code, deployment and could have introduced a single point of failure.
  • When a Redis process reached it’s 32-bit Windows memory limit of 2GB it would crash. We had deployed 4 instances to each server and we distributed the read/writes across these instances.

I chose AppFabric Caching Services from Microsoft’s for a few reasons, and within an hour or so had replaced Redis with AppFabric.

  • It supports equal cluster members, so all members are read/write
  • Native 64-bit support meant no 2GB process limit
  • Active support and community
  • Like Redis, it’s free

Having deployed AppFabric in a production environment for several months now, I thought i’d write a few findings:

  • Installation is not entirely scriptable. Despite efforts by Microsoft to make everything scriptable with PowerShell the very first installation and subsequent upgrades had to be done via the Wizard. After that, the PowerShell commandlets were available for use.
  • There can be long waits (5 minutes or so) during startup when using a distributed cache. It makes sense that the various nodes require time to synchronise, but the error messages that are reported are cryptic.
  • It’s rock solid once you get it running. We’ve used it now for about 3 months in a 3 server cluster storing items in the cache of upto 50MB and it’s fast.
  • You can’t enumerate the cache keys. With Redis our refresh-ahead service would iterate over the keys and freshen content if necessary, we’ve had to drop this functionality and take a small performance hit with AppFabric.
  • Upgrading isn’t entirely trivial. I upgraded our servers from v1.1 to v1.2 and although it’s designed to have heterogenous nodes in the cluster with differing versions, some of the security settings appeared to have changed between versions meaning that our website ‘client’ was not permitted access to the cluster. Luckily we still had Redis running in the background and a config switch directed the website at that temporarily.
  • Changing cache settings requires you to delete the cache. Carefully consider the largest size object you want in the cache when you create the cache – if you want to change it in future, you’ll need to either start a new cache and update your config, or take the site off-line whilst you upgrade your cache.
  • You need to specify the CacheItemVersion when deleting items. Use the following to remove from the cache. When I tried Remove(key) I always got a false returned and the item remained. This makes sense that in a distributed cache it needs to know explicitly which version to remove, but seemed to only occur when we had a cache cluster.

    public bool Remove(string key)
    {
    var dataCacheItem = _cache.Get(key);
    return _cache.Remove(key, dataCacheItem.Version);
    }

Overall I’m pretty happy with AppFabric, but some of these gotchas, the general lack of enthusiasm on forums etc. concerns me slightly about the future. But hey, it didn’t take long to write the concrete implementation, tests and config switches needed to get it into the codebase, so it won’t take long to replace it if it does get dropped.

In Visual Studio, choose Tools | Macro, Macros IDE, create a new module and drop this in. I bind it to Ctrl-Alt-1 for quick access.

    Sub AttachToW3WP()
        Dim attached As Boolean = False
        Dim proc As EnvDTE.Process

        For Each proc In DTE.Debugger.LocalProcesses
            If (Right(proc.Name, 8 ) = "w3wp.exe") Then
                proc.Attach()
                attached = True
            End If
        Next

        If attached = False Then
            MsgBox("w3wp.exe is not running")
        End If

    End Sub

AppFabric Cache has a max buffer size of 8MB. If you’re using the SQL Provider, you need to export and re-import the Xml configuration file to modify this. Here’s a powershell script to do it for you, building on the blog post from Javi.

Save this to UpdateAppFabricCacheBufferSize.ps1

Param([int]$maxBufferSize)

Import-Module DistributedCacheAdministration, DistributedCacheConfiguration

Function UpdateBufferSizeInConfig ([string]$configFilename, [int]$maxBufferSize)
{
	$xml = New-Object XML
	$xml.Load($configFilename)
	$transportProperties = $xml.configuration.dataCache.advancedProperties.transportProperties
	if ($transportProperties -eq $NULL) {
	  $transportProperties = $xml.CreateElement("transportProperties")
	}
	$transportProperties.SetAttribute("maxBufferSize", "$maxBufferSize")
	$xml.configuration.dataCache.advancedProperties.appendChild($transportProperties)
	$xml.Save($configFilename)
}

$tempConfigLocation = "c:\config.xml"

Use-CacheCluster
Export-CacheClusterConfig -File $tempConfigLocation

UpdateBufferSizeInConfig $tempConfigLocation $maxBufferSize

Stop-CacheCluster
Import-CacheClusterConfig -File $tempConfigLocation -Force
Start-CacheCluster

To change your buffer size to 15MB:

powershell ./UpdateAppFabricCacheBufferSize.ps1 15000000

Two tips I gave recently to a colleague just setting out with Disqus.

Q. How do I make Disqus comments visible to Google?
A. Use the Javascript code snippet Disqus provide as this fetches the comments asynchronously. On your server implement a background task to fetch and cache recent comments from Disqus using the Disqus API (you could fetch them during page render, but your page load speed will be directly coupled to the response time from Disqus). When the page is rendered embed the cached comments between <noscript> tags. This allows you to use HTML page caching services like Akamai/Varnish whilst still having moderately fresh comments in the page for Google (and non-JS users). Best to only include a few comments to keep page size down and then provide pagination links for the search engines. (This was inspired by http://www.seroundtable.com/disqus-seo-14093.html).

If you are using an ESI caching solution you might be tempted to implement an ESI include to fetch the comments as they are dynamic content. I’d recommend not doing this as you’ll be fetching the comments (from your cache, or Disqus) on every page load which is very unnecessary just for the occasional visit by Google.

Q. What should I use for the disqus_identifier?
A. I recommend using an internal identifier for the piece of content to which the comment is attached prefixed with an environment indicator, e.g. disqus_identifier = ‘live_ 21EC2020-3AEA-1069-A2DD-08002B30309D’. I’d strongly suggest that you do not leave it blank and do not use the page URL. If you leave it blank Disqus will automatically use the URL which may not be permanent, thus when the article title changes (which is regularly included in a URL), the comments will be lost. Prefixing the environment to the identifier mitigates any clash with comments made in your testing environment when you move your CMS data around.

I need this with alarming frequency, so here it is. Accessing a remote FTP site through a proxy.

ftp://remoteuser%40remotehost:remotepassword@proxyhost/folder

  1. Open Google Chrome developer tools.
  2. Click the cog in the bottom right of the window

  3. Choose “Preserve log upon navigation”

This was tested with Google Chrome 14.0.835.163 m on Windows.

Fiddler is a Web Debugging Proxy which logs all HTTP(S) traffic between your computer and the Internet. Fiddler allows you to inspect all HTTP(S) traffic, set breakpoints, and “fiddle” with incoming or outgoing data.

Download Fiddler from http://www.fiddler2.com/, it’s freeware! It runs on Windows, but can debug traffic originating in any operating system (by making that OS point to Fiddler on Windows as a proxy). Before reading this you should read these articles which provide an overview of Fiddler.

Stubbing network responses

During development with a third party it’s often handy to insulate yourself from any downtime/network problems that might affect your testing. Quite often this involves writing a piece of code to simulate network responses and pointing your app to that. Instead of doing this, turn to Fiddler.

Record and replay

Configure your application to use Fiddler as a proxy (see this for .NET apps, use localhost:8888), then hit your third party endpoint with your application. Fiddler will capture the traffic in the session list. Now, click on the Auto Responder tab and enable Automatic Responses. Drag each row from the session list into the Auto Responder list. Now re-run your app, and instead of connecting to the remote machine, Fiddler will auto-respond for you. (If you are using SSL, read how to decrypt SSL traffic and also in .NET you’ll need to suppress the invalid man-in-the-middle cert that Fiddler uses by returning true in the ServerCertificateValidationCallback

From an interface spec

If you have an interface spec but no endpoint to hit, create a file matching the content you expect to be returned, define a match for the URI, and use your sample file as the response content. See AutoResponder reference for more information.

You can use regex pattern matching for the URI, and you can either respond with a local file, or captured session. With a regex to match the entire host you can make all calls to your network resource respond with a HTTP 403 Denied and ensure your app behaves as expected.

Custom rules to show Akamai cached pages

I’ve used Akamai edge caching on a number of sites over the past few years to improve site performance, and it’s always useful to see which pages are being served from cache, and which aren’t. The easiest way I’ve found to do this is to add a custom rule to Fiddler to highlight requests for me. From Fiddler, choose Rules, Customize Rules. In the Javascript that opens, enter the following code:

With the other field definitions…

	public static RulesOption("Highlight Akamai cache Hits")
	var m_HighlightAkamaiHits: boolean = false;

In the “OnBeforeRequest” method…

	if (m_HighlightAkamaiHits) {
		oSession.oRequest.headers.Add("Pragma", "akamai-x-get-cache-key");
		oSession.oRequest.headers.Add("Pragma", "akamai-x-cache-on");
		oSession.oRequest.headers.Add("Pragma", "akamai-x-cache-remote-on");
		oSession.oRequest.headers.Add("Pragma", "akamai-x-get-true-cache-key");
		oSession.oRequest.headers.Add("Pragma", "akamai-x-check-cacheable");
		oSession.oRequest.headers.Add("Pragma", "akamai-x-get-extracted-values");
		oSession.oRequest.headers.Add("Pragma", "akamai-x-get-nonces");
		oSession.oRequest.headers.Add("Pragma", "akamai-x-get-ssl-client-session-id");
		oSession.oRequest.headers.Add("Pragma", "akamai-x-serial-no");
	}

In the “OnBeforeResponse” method…

	if (m_HighlightAkamaiHits) {
		if (oSession.oResponse.headers.ExistsAndContains("X-Cache","TCP_MEM_HIT")) {
			oSession["ui-customcolumn"] = "HIT";
		} else if (oSession.oResponse.headers.ExistsAndContains("X-Cache","TCP_IMS_HIT")) {
			oSession["ui-customcolumn"] = "HIT";
		}
	}

Now close the Javascript file, and go back to Fiddler. If you made any mistakes in the Javascript, Fiddler will tell you immediately. From the Rules menu you now have a new option – “Highlight Akamai cache Hits”. Enable this, and visit http://www.facebook.com/ in your browser. In Fiddler, you should see the word “HIT” for several of the requests in the “custom” column. You can rearrange the column order to move the custom column if you like.

Add request time

This is a simple new rule but surprisingly handy.

With the other field definitions…

	public static RulesOption("Show response time")
	var m_ShowResponseTime: boolean = false;

Add to either “OnBeforeRequest” or “OnBeforeResponse” method…

	if (m_ShowResponseTime) {
		oSession["ui-customcolumn"] = DateTime.Now.ToString();
	}

Remember when using these rules that when you save the Javascript file, the Rules menu will be reset so any previously enabled rules will need re-enabling.

Fiddler also has a nice set of C# APIs which allow you to embed the fiddler engine directly into your test suite, which makes for a really nice set of integration tests (using the AutoResponder) with only a few lines of code. I’ll go into this in a future post.

UPDATED: 7/Feb/2011 with comments from Simon Smith.

I’ve seen a few examples of people trying to mimic the Google Instant search with their own solution. Most of these have just made them “instant” searches by changing

$("#searchButton").click(function(){
  ...perform actual search...
});

to

$("#searchButton").keyup(function(){
  ...perform actual search...
});

My gripe is that Instant doesn’t need to be, and in fact shouldn’t always be, Instant. There are 3 reasons for this

1) A lot of users type looking at their keyboard so Instant just needs to mean “ready when a user looks up from their keyboard to see the result”
2) Browser and network performance can be significantly harmed if you’re issuing complex javascript/ajax/network requests on every single keypress.

The solution? Well, my solution is very simple.

$(document).onready(function(){
   var _timerId = 0;

   $("searchButton").keyup(function(){
      window.clearTimeout(_timerId);
      _timerId = window.setTimeout(function() {
         ...perform actual search...
      }, 170);
   }).keydown(function(){
      window.clearTimeout(_timerId);
   });
});

In this example I set a timer which fires 170ms after the LAST keypress. It’s pretty imperceivable that there’s a delay at all, but it dramatically improves CPU/bandwidth performance of these “Instant” searches, and it still appears to be pretty Instant.

Just been parsing logfiles from our site generated by Akamai with analog. We use COMBINED log format on the Akamai help page, but the default COMBINED log format from analog wasn’t able to parse. Based on info from the analog log format help page the following worked.

LOGFORMAT (%s - - [%d/%M/%Y:%h:%n:%j %j] "%j %r %j" %c %b "%j" "%B" "%j")

Next »