Thursday, March 12, 2009

Feature Update

As many users will have noticed, I launched a pretty major new release of on Feb 28, 2009. I had been working on a large set of changes for about 6 weeks, so I felt it was finally time to release what I had.

I must admit that some of the changes have been a little rough around the edges, as I rushed to complete features by my self-imposed deadline. But I feel that overall, the new version is a big improvement, though much more "heavy weight" than the original vision for

Here's a highlight of the what you'll find new in this release:

  1. New Design - Yes, it's RED. But a bit more designed than the previous site. I tried to make it easier to understand that you can install the bookmarklet to get started.
  2. Link Page Sidebar - The comment area has moved to a side-bar (rather than the top of the page) where most people have extra real-estate. Anyone using a wide monitor should see more of the target web page without having to scroll. You can also click the expand/collapse triangle in the header of each sidebar panel to hide sections to make even more room for the Chat window.
  3. Real-time Chat - is now a fully AJAX application which allows you to chat in real-time with readers visiting your links. You can also see how many people are currently looking at your page (hidden feature - if you hover your mouse over the thumbnail of a user image in the chat window, it will tell you where in the world that person is located).
  4. Sponsored Link Pages - While not an end-user "feature", per se, is now accepting sponsorship to help pay for the costs of providing our service. If you get annoyed by the ads, you can shrink the Sponsored By panel so you don't have to look at it.

I had to punt a few features that were under development but not yet ready. These should be coming soon:

  • Formatting Fixes - There are some page formatting problems I know about - especially in IE. Most of my users are on Firefox or Safari, but I will be addressing the cross-browser problems shortly
  • Private Chat - You'll be able to create a link that only you and your friends know about to have a private chat about a web page.
  • RSS feeds - Subscribe to personal or tag-based pages of the mosts popular links.

Tuesday, March 10, 2009

What's a "Real User"? Twitter Bots are Crawling links.

One of the features of is that it can tell you how many users are visiting your link. One problem I've been noticing lately, is that I am counting users that aren't real people.

I experimented with twittering a link, and then looking at the viewers of that link shortly after. I'm seeing a number of Bots crawling twittered links right away. From my server logs I see the following request that got counted as real users: - - [10/Mar/2009:11:57:13 -0700] "GET /xL HTTP/1.1" 200 8416 - "Mediapartners-Google,gzip(gfe),gzip(gfe)" - - [10/Mar/2009:12:00:25 -0700] "HEAD /xL HTTP/1.1" 200 9468 - "AideRSS 2.0 (,gzip(gfe),gzip(gfe)"

xx.xx.xx.xx - - [10/Mar/2009:12:00:56 -0700] "GET /xL HTTP/1.1" 200 9468 - "Mozilla/5.0 (X11; U; Linux i686; en-US; rv: Gecko/20071204 Ubuntu/7.10 (gutsy) Firefox/ FirePHP/,gzip(gfe),gzip(gfe)" - - [10/Mar/2009:12:01:22 -0700] "GET /xL HTTP/1.1" 200 9469 - "WWW::Shorten::TinyURL/1.90,gzip(gfe),gzip(gfe)" - - [10/Mar/2009:12:01:35 -0700] "GET /xL HTTP/1.1" 200 9469 - "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-GB; rv: Gecko/2009011913 Firefox/3.0.6 TweetmemeBot,gzip(gfe),gzip(gfe)" - - [10/Mar/2009:12:08:03 -0700] "HEAD /xL HTTP/1.1" 200 9468 - "Chat Catcher - (,gzip(gfe),gzip(gfe)" - - [10/Mar/2009:12:08:33 -0700] "HEAD /xL HTTP/1.1" 200 9470 - "PycURL/7.19.3,gzip(gfe),gzip(gfe)"

Note that the link was posted at 11:57 - so these requests came in from 6 seconds to 10 minutes after Tweeting the original link. Only one of these listed was a "real human". So, is counting 6 unique users that are actually Bots.

To fix this, I'm going to be changing my counting algorithm to require that you've successfully received a cookie (as a normal web browser would do), AND that your browser has successfully made an AJAX request to the server to receive a Chat update. This should eliminate all Bots and Crawlers as falsely being reported as "Vistors" to links.