Al Priest’s Blog

Two tips I gave recently to a colleague just setting out with Disqus.

Q. How do I make Disqus comments visible to Google?
A. Use the Javascript code snippet Disqus provide as this fetches the comments asynchronously. On your server implement a background task to fetch and cache recent comments from Disqus using the Disqus API (you could fetch them during page render, but your page load speed will be directly coupled to the response time from Disqus). When the page is rendered embed the cached comments between <noscript> tags. This allows you to use HTML page caching services like Akamai/Varnish whilst still having moderately fresh comments in the page for Google (and non-JS users). Best to only include a few comments to keep page size down and then provide pagination links for the search engines. (This was inspired by http://www.seroundtable.com/disqus-seo-14093.html).

If you are using an ESI caching solution you might be tempted to implement an ESI include to fetch the comments as they are dynamic content. I’d recommend not doing this as you’ll be fetching the comments (from your cache, or Disqus) on every page load which is very unnecessary just for the occasional visit by Google.

Q. What should I use for the disqus_identifier?
A. I recommend using an internal identifier for the piece of content to which the comment is attached prefixed with an environment indicator, e.g. disqus_identifier = ‘live_ 21EC2020-3AEA-1069-A2DD-08002B30309D’. I’d strongly suggest that you do not leave it blank and do not use the page URL. If you leave it blank Disqus will automatically use the URL which may not be permanent, thus when the article title changes (which is regularly included in a URL), the comments will be lost. Prefixing the environment to the identifier mitigates any clash with comments made in your testing environment when you move your CMS data around.

I need this with alarming frequency, so here it is. Accessing a remote FTP site through a proxy.

ftp://remoteuser%40remotehost:remotepassword@proxyhost/folder

1. Open Google Chrome developer tools.
2. Click the cog in the bottom right of the window
   

3. Choose “Preserve log upon navigation”
   

This was tested with Google Chrome 14.0.835.163 m on Windows.

Fiddler is a Web Debugging Proxy which logs all HTTP(S) traffic between your computer and the Internet. Fiddler allows you to inspect all HTTP(S) traffic, set breakpoints, and “fiddle” with incoming or outgoing data.

Download Fiddler from http://www.fiddler2.com/, it’s freeware! It runs on Windows, but can debug traffic originating in any operating system (by making that OS point to Fiddler on Windows as a proxy). Before reading this you should read these articles which provide an overview of Fiddler.

Stubbing network responses

During development with a third party it’s often handy to insulate yourself from any downtime/network problems that might affect your testing. Quite often this involves writing a piece of code to simulate network responses and pointing your app to that. Instead of doing this, turn to Fiddler.

Record and replay

Configure your application to use Fiddler as a proxy (see this for .NET apps, use localhost:8888), then hit your third party endpoint with your application. Fiddler will capture the traffic in the session list. Now, click on the Auto Responder tab and enable Automatic Responses. Drag each row from the session list into the Auto Responder list. Now re-run your app, and instead of connecting to the remote machine, Fiddler will auto-respond for you. (If you are using SSL, read how to decrypt SSL traffic and also in .NET you’ll need to suppress the invalid man-in-the-middle cert that Fiddler uses by returning true in the ServerCertificateValidationCallback

From an interface spec

If you have an interface spec but no endpoint to hit, create a file matching the content you expect to be returned, define a match for the URI, and use your sample file as the response content. See AutoResponder reference for more information.

You can use regex pattern matching for the URI, and you can either respond with a local file, or captured session. With a regex to match the entire host you can make all calls to your network resource respond with a HTTP 403 Denied and ensure your app behaves as expected.

Custom rules to show Akamai cached pages

I’ve used Akamai edge caching on a number of sites over the past few years to improve site performance, and it’s always useful to see which pages are being served from cache, and which aren’t. The easiest way I’ve found to do this is to add a custom rule to Fiddler to highlight requests for me. From Fiddler, choose Rules, Customize Rules. In the Javascript that opens, enter the following code:

With the other field definitions…

	public static RulesOption("Highlight Akamai cache Hits")
	var m_HighlightAkamaiHits: boolean = false;

In the “OnBeforeRequest” method…

	if (m_HighlightAkamaiHits) {
		oSession.oRequest.headers.Add("Pragma", "akamai-x-get-cache-key");
		oSession.oRequest.headers.Add("Pragma", "akamai-x-cache-on");
		oSession.oRequest.headers.Add("Pragma", "akamai-x-cache-remote-on");
		oSession.oRequest.headers.Add("Pragma", "akamai-x-get-true-cache-key");
		oSession.oRequest.headers.Add("Pragma", "akamai-x-check-cacheable");
		oSession.oRequest.headers.Add("Pragma", "akamai-x-get-extracted-values");
		oSession.oRequest.headers.Add("Pragma", "akamai-x-get-nonces");
		oSession.oRequest.headers.Add("Pragma", "akamai-x-get-ssl-client-session-id");
		oSession.oRequest.headers.Add("Pragma", "akamai-x-serial-no");
	}

In the “OnBeforeResponse” method…

	if (m_HighlightAkamaiHits) {
		if (oSession.oResponse.headers.ExistsAndContains("X-Cache","TCP_MEM_HIT")) {
			oSession["ui-customcolumn"] = "HIT";
		} else if (oSession.oResponse.headers.ExistsAndContains("X-Cache","TCP_IMS_HIT")) {
			oSession["ui-customcolumn"] = "HIT";
		}
	}

Now close the Javascript file, and go back to Fiddler. If you made any mistakes in the Javascript, Fiddler will tell you immediately. From the Rules menu you now have a new option – “Highlight Akamai cache Hits”. Enable this, and visit http://www.facebook.com/ in your browser. In Fiddler, you should see the word “HIT” for several of the requests in the “custom” column. You can rearrange the column order to move the custom column if you like.

Add request time

This is a simple new rule but surprisingly handy.

With the other field definitions…

	public static RulesOption("Show response time")
	var m_ShowResponseTime: boolean = false;

Add to either “OnBeforeRequest” or “OnBeforeResponse” method…

	if (m_ShowResponseTime) {
		oSession["ui-customcolumn"] = DateTime.Now.ToString();
	}

Remember when using these rules that when you save the Javascript file, the Rules menu will be reset so any previously enabled rules will need re-enabling.

Fiddler also has a nice set of C# APIs which allow you to embed the fiddler engine directly into your test suite, which makes for a really nice set of integration tests (using the AutoResponder) with only a few lines of code. I’ll go into this in a future post.

UPDATED: 7/Feb/2011 with comments from Simon Smith.

I’ve seen a few examples of people trying to mimic the Google Instant search with their own solution. Most of these have just made them “instant” searches by changing

$("#searchButton").click(function(){
  ...perform actual search...
});

to

$("#searchButton").keyup(function(){
  ...perform actual search...
});

My gripe is that Instant doesn’t need to be, and in fact shouldn’t always be, Instant. There are 3 reasons for this

1) A lot of users type looking at their keyboard so Instant just needs to mean “ready when a user looks up from their keyboard to see the result”
2) Browser and network performance can be significantly harmed if you’re issuing complex javascript/ajax/network requests on every single keypress.

The solution? Well, my solution is very simple.

$(document).onready(function(){
   var _timerId = 0;

   $("searchButton").keyup(function(){
      window.clearTimeout(_timerId);
      _timerId = window.setTimeout(function() {
         ...perform actual search...
      }, 170);
   }).keydown(function(){
      window.clearTimeout(_timerId);
   });
});

In this example I set a timer which fires 170ms after the LAST keypress. It’s pretty imperceivable that there’s a delay at all, but it dramatically improves CPU/bandwidth performance of these “Instant” searches, and it still appears to be pretty Instant.

Just been parsing logfiles from our site generated by Akamai with analog. We use COMBINED log format on the Akamai help page, but the default COMBINED log format from analog wasn’t able to parse. Based on info from the analog log format help page the following worked.

LOGFORMAT (%s - - [%d/%M/%Y:%h:%n:%j %j] "%j %r %j" %c %b "%j" "%B" "%j")

Recently I was asked to investigate a site which wasn’t performing as well as expected. The initial reaction was to look at the code – perhaps our NHibernate mappings were eagerly fetching too much data, perhaps we’re not caching enough, etc.

Then I stopped and remembered the database. I found that the table structure and indexes were sub-optimal and after a quick restructure the performance was boosted 400%. This got me wondering why my initial reaction was to dive into the code rather than consider any other options.

One of the Agile principles is Inspect and Adapt. To provide a regular team cadence this usually takes the form of a retrospective every couple of weeks. These sessions discuss the actions which came out from the previous retrospective and the previous couple of weeks work. The team establishes a few important areas of concern and these are discussed in more detail to establish a course of resolution.

Extending this line of thought I ended up wondering why do we do what we do each day? Is it the path of least resistance? Force of habit? Loyalty?

What might a Life Retrospective uncover?

• Would we discover a new route to work which we’d previously discounted for being too long? A new carriage on the train which doesn’t get as busy?
• A new variety of pasta sauce even tastier than our regular? A supermarket own brand for half the price but the same taste?
• Might we find that every day we live with the coffee table getting in the way when we walk into the lounge, but we never try a new layout?
• How much extra happiness could we gleen from running life retrospectives? Who would run it? How frequently would be useful?

Lots of questions and not many answers I’m afraid. Maybe I need a retrospective of this post.

public static class FileSizeExtensions
	{
		[DllImport("Shlwapi.dll", CharSet = CharSet.Auto)]
		public static extern long StrFormatByteSize(long fileSize,
		                                            [MarshalAs(UnmanagedType.LPTStr)] StringBuilder buffer, int bufferSize);

		public static string DisplayAsByteSize(this long filesize)
		{
			StringBuilder sbBuffer = new StringBuilder(20);
			StrFormatByteSize(filesize, sbBuffer, 20);
			return sbBuffer.ToString();
		}
	}

In 2008 I started running with the BBC Running Club after the squash club at work closed. Late 2009 I found I had a place in the London Marathon, and the training started. Up until that point i’d been measuring my distance in kilometres, my maximum weekly distance was 12km and my longest single run was 10km. I changed my watch to track in miles and started increasing my weekly mileage.

From January through to marathon day I put in 165 miles of training runs, which is a lot less than i’d been recommended (i’d been advised that the 5 longest runs in your training plan should total 100 miles). My lack of training was due to terrible pain in my right calf muscle which I later discovered was due to the wrong pair of trainers. My gait was wrongly analysed in the shop and annoyingly it took me 2 months to associate the pain and the new trainers. Cue me spending over £300 on physiotherapy and then another £100 on new trainers when it was resolved.

Armed with the new trainers in late March I started training in earnest and put in an average of around 20 miles a week including the Silverstone Half marathon. I couldn’t ramp up too much too quickly with marathon day looming so my biggest run was a 15 miler.

The obligatory carb loading started on the Friday and with every huge bowl of pasta I felt increasingly like a goose being fattened for a foie gras fate. On the Saturday I went to a friends house who lived nearer Blackheath. That night when I closed my eyes all my preparation came back to me – the muddy cross country; the street runs in the snow and ice; the canal runs to Little Venice; the pain of my unnecessary transevatherapy; the fantastic words of advice from previous marathon runners; the added variety of NikeTownRunners nights; the Tuesday Trot organised by the BBC running club.

On Sunday morning my friend graciously drove me to within 500m of the start line and with surprisingly few people approaching from the Lewisham side and I got to the red start enclosure around 9am.

The organisation was fantastic and despite the short downpour spirits were high. My iPod was charged, my kit bag was in the truck to be taken to the finish line and I was in starting pen 6 ready to race.

At 9:50:24 I crossed the start line which really surprised me – i’d expected it to take 20-30 minutes following my experience back at the British 10k in July. At that point I was desperate to go for it, the crowd were cheering and I was full of adrenalin.

Within 0.5 mile I needed a Paula Radcliffe break but I was quickly back in the thick of it, and despite being constantly overtaken I was able to stick to my pace target of 10:18/mile (4h30 marathon). Around 15 minutes later I switched the iPod off and let the crowd support carry me instead. I was wary of this as almost all of my previous runs had been with a soundtrack, but my caution was unnecessary as I started hearing my name shouted out spurring me on. (The people with pints and bacon butties in hand at 10am did little to inspire me however!)

45 minutes into the race I took my first gel pack. I had tried and enjoyed SIS energy gels during my training and took 6 carried on my gel belt. (Top tip: at the marathon expo 2010, they were selling the marathon belt including 4 gels and an energy bar for £10). My next gel was at 90 minutes, and the remainder were every 35 minutes. I heartily recommend using these. Also, these specific gels don’t need to be taken with water making them more versatile, some others do so ask!

After 6.5 miles and just over an hour, I reached the Cutty Sark. Since the fire in 2007 she’s covered in scaffolding so the first highlight of the route was unfortunately not to be enjoyed. Another 6.5 miles, and around 2hrs 20 into the race I turned onto Tower Bridge. I’d been warned not to speed up with the huge crowds that line the road. With the volume of people and the fact i’d just run 13 miles, this wasn’t difficult advice to take.

Over the bridge the course turns right and the road splits down the middle, with mile 23 of the course coming in the opposite direction. This served to reinforce in me two things – 1) i’m not a ‘runner’ i’m a jogger, and 2) i’m only half way.

The route then wound around the Isle Of Dogs for several miles and I lost track of how many miles I’d completed after mile 17. I was pretty sure i’d passed the mile 18 marker and so was setting a faster-than-required pace when I saw in the distance the ‘mile 19′ marker. This marker turned out to be for mile 18 which was desperately disappointing to see.

At mile 18 the route goes underground for a few hundred metres and whilst out of sight of the spectators an increasing number of people opted to walk. Rather annoyingly people weren’t walking on the sides of the road so whilst I was attempting to run around them, I did end up bouncing into others like the ball in a pinball machine. Sorry about that.

I had completely lost my bearings whilst around Canary Wharf but when I came out at mile 20 I knew what I had left in store – 3 miles of the course i’d run in the opposite direction already, and the remaining 3 i’d done on the British 10k.

They say that a Marathon doesn’t follow the conventional rules of mathematics. That is, the first half of the 26.2 mile course is 21 miles. Perhaps it was the gels i’d been taking, or perhaps it was my water schedule – 2 or 3 sips at every water station, then ditch the bottle – but I didn’t find this to be the case. Judging by the increasing numbers of people walking i’m sure it’s more luck than training.

All along the route strangers were offering of jelly babies, orange slices, and towards the end one of the pubs was giving out small glasses of beer and rapturous applause for those willing to partake. I hadn’t trained with sweets and orange so I didn’t accept any, but those who did seemed grateful.

At mile 23 was the official cheer point for The Banana Army (the fundraising group for Leukaemia and Lymphoma Research) and with my name emblazoned on my t-shirt, the support was uplifting. This surged my pace and it dropped from an average of 10:30 down to 9:45 for about 10 minutes.

From Westminster I knew there was very little left to go and tried to keep the head up, feet moving and resist the urge to slow. At the 800m sign I had very little left in the tank, and from my training runs remembered that 800m is not as short as it sounds. As I turned off Birdcage Walk I saw the banner saying 365yds to go and I started to push. Every single person you overtake in those last few yards is one more person you’ve beaten and with the clock ticking in front of you the adrenalin pumps hard.

Arms held high I crossed the line 4 hours 46 minutes and 37 seconds after leaving Blackheath. A quick smile for the camera, the adrenalin vanished and the pain started. To remove the timing chip from your laces you have to walk up a small ramp (to bring your feet up to a reasonable level with the volunteers). This ’small ramp’ at the end of 26.2 miles felt like Everest and I had to use the hand rails to drag myself up.

From there I was ushered in front of the photographers for an official photo. Whilst in the queue the women behind me eloquently described the pain we were all feeling, but I shan’t repeat the language here.

Realising a childhood ambition and being able to do it whilst raising money for a great cause made me quite emotional so I sat down to collect my thoughts for a few minutes before trying to stretch and realising just how immobile I was. After shuffling to the lorry to collect my bag and change my trainers for looser fitting ones I subdued the pain from my blisters and muscles with a pint of lager surrounded by family and friends.

A great day, and a fantastic 4 months training. I never gave up the alcohol during the training but I did pretty much reduce my maximum intake to the governmental binge drinking definition [shockingly low by the way, binge indeed!], and I’ve never felt better. It’s quite a buzz to wake up at 7am sober on a Sunday morning and go for a run and i’d recommend it to anyone – but get professional advice on trainers!

Will I do it again? Never say never.

Like a lot of people i’ve pondered the ‘killer app’ over a pint, but these days i’m coming to the conclusion that everything has been invented, duplicated, and cloned to death. So rather than inventing new apps, i’ve been pondering the idea of making the apps become location – or familiarity – sensitive. Combine this with the molecular age of information and I think we could get some really great functionality.

Some examples.

• You connect to a WiFi and your apps fetch their latest data – Twitterific fetches the latest tweets, NetNewsWire fetches your latest RSS subscriptions, your email syncs – all automatically, no clicking into every app to hit refresh.
• You check Google Maps for your route and it automatically looks up who lives nearby from your contact list, checks their Facebook, Twitter or Buzz and tells you that “Hey, Sarah has just got back from Turkey and only lives 5 miles from your route, why not stop by”
• You identify your home WiFi SSID as trusted and your phone disables the keylock, logs you into Skype, and looks up a selection of recipes you might want to cook (if we got the Internet Fridge, this could be really cool!)
• If you are out of network coverage, but within a WiFi – e.g. underground – then your phone automatically logs you into Skype and redirects your mobile number.