URL Shortener Battle Heats Up

There have been some really interesting battles going on in the URL-Shortener space since I've started compiling data on all the competitors.

The two giants are TinyURL.com and Bit.ly. Interestingly, Twitter changed their default shortener from the classic tinyurl, to the up-and-coming bit.ly (following their announcement of venture-back financing). As a result, bit.ly has surpassed tinyurl in daily usage for June.

The rest of the pack is orders of magnitude behind these two leaders in usage. Still, there are some interesting battles in the back of the field. Digg.com is ramping up following their innovative (but much maligned) toolbar. Ow.ly also seems to be getting some good traction on the strength of their Twitter application suite, HootSuite.

Amazon has an Integrated URL Shortener - AMZN.COM

Kyle Mulka just tweeted an AMZN.COM url to a product page. I was amazed that this didn't get converted to a longer URL, but that Amazon allows for this alternate page representation. Here's an example:

http://amzn.com/076790818X

I started playing with a number of other shortcuts besides product codes and found some other URL's that do redirect to other Amazon.com URL's:

• amzn.com - You can go to amazon's home page with this shorter version.
• amzn.com/books - Each of the major sections of the Amazon store seem to have their own shortcut (other examples: amzn.com/bestsellers, amzn.com/kindle, amzn.com/electronics, etc.
• amzn.com/bill - Information about your Amazon bill.
• amzn.com/help - Amazon's Help page.

Interestingly, I note that almost no one seems to be twittering these short URL's directly (less than one per day on Twitter right now - and only 15 people have sent updates including them in the last two weeks).

I also note that Google does not have any pages in it's search index using the amzn.com domain.

As Twitter becomes more popular, I think more sites will be incorporating URL shorteners into their applications, enabling their users to share their pages without resorting to 3rd party services. Amazon's tool, like Digg's "Digg Bar", is also a handy address-bar shortcut; I can envision training myself to type amzn.com/kindle instead of visiting amazon.com and then navigating to the kindle section.

Update: Analysis of AMZN.COM shortcuts

My curiosity piqued, I decided to see what mysteries lay behind the AMZN.COM domain. I wrote a program to search through an English dictionary looking for valid shortcuts. I found that they break down into a few distinct categories.

Feature Pages

The biggest category (496 shortcuts) I found are "feature" pages. These seem to be highly customized pages about a collection of product types. I'm assuming these are hand crafted by Amazon's editors.

accessories, adobe, ads, advertise, advertising, airborne, aldo, allied, angel, apparel, apple, archery, artisan, artists, arts, ati, atlanta, atlantis, att, austin, auto, automotive, autos, av, avon, baby, badminton, baking, ballet, bargain, bargains, baseball, basket, basketball, beauty, bedding, benefit, bernardo, beverages, bibles, blackberry, blankets, bleachers, blizzard, block, boating, bob, bodybuilding, bolt, bombay, books, borders, bosch, boss, bostitch, bostonian, brass, braun, broadband, broadway, brother, business, calendars, camera, cameras, camping, campus, canning, cannon, canon, car, careers, cars, catcher, cellular, celtic, champs, charities, charity, chat, cheap, chicago, chocolate, choice, christian, christianity, christy, citizen, classical, clearance, clifford, climbing, clothes, clothing, coffee, college, columbia, comics, command, computer, computers, concert, concord, condiments, confessions, construction, contractor, cooking, corporate, costume, costumes, country, crafts, crane, creative, crew, cricket, criterion, dallas, dance, decca, decks, decor, delta, denver, detroit, diamonds, discount, discounts, diving, dolls, dove, drapers, drugstore, drums, dummies, easter, eastman, educational, ei, electronics, emerging, engagement, engineering, entertaining, entertainment, essentials, et, exercise, eyewitness, fans, fantasy, faulkner, feed, feeding, fencing, festivals, financial, financing, fishing, fitness, flash, flashlight, flip, flowers, focus, food, football, footprint, footwear, fossil, fox, fragrance, fragrances, franklin, fresco, freud, fuji, furniture, galaxy, games, gaming, garden, gardening, gateway, gear, gilbert, ginkgo, giving, global, golf, gourmet, gps, graduation, green, griffin, grizzly, groceries, grocery, guess, guitars, gymnastics, hallmark, halloween, halo, hannah, hannibal, hardware, harry, health, hearts, heaters, hefty, hi, hiking, hips, history, hitchcock, hobbies, hobby, hockey, holmes, home, hoover, horizon, horror, household, hue, hulk, hunting, ice, indies, industrial, ink, insomnia, instruments, ironing, irwin, isi, jazz, jello, jet, jewelry, jobs, journey, kaiser, kids, kitchen, kraft, landscaping, latin, leadership, leapfrog, learning, leatherman, lee, levels, levis, librarian, libraries, licensing, lighting, literature, lodge, lucky, lush, macintosh, magazines, majestic, marvel, maverick, may, meats, medical, memory, metal, method, mi, microwaves, milwaukee, mini, minneapolis, mixers, monster, monsters, motorcycle, movies, mozart, murray, music, nabisco, nara, nautilus, naxos, nec, nfl, nick, nikko, nonfiction, nordstrom, notebooks, nuance, nursery, nursing, observer, office, olympics, olympus, oneida, oracle, organic, orvis, oshkosh, outdoors, outlet, paint, palm, paramount, parsons, parts, patio, patrician, patton, pc, pearls, penny, performance, pets, philadelphia, phillips, philosophy, phoenix, photo, pillows, planters, plumbing, po, polar, polaroid, polo, pools, practices, prepare, preschool, presto, princess, processing, professional, progressive, projects, puma, puzzles, pyrex, read, reese, reference, relic, religion, remington, replacements, resolutions, revere, ricci, rival, rockport, rocks, rodeo, romance, roots, rousseau, royal, rpg, ruby, running, russell, safety, sage, sale, salter, sandpaper, sands, sante, sauces, save, scholastic, science, scientific, seafood, seattle, security, sees, services, sesame, sewing, sex, shakespeare, sharp, shoes, shorts, shun, silhouettes, silver, simmons, simplicity, singer, skating, sledding, snacks, soccer, software, solutions, spanish, spices, sports, spreads, spring, squash, stanley, stones, storage, strings, summer, sunbeam, sunshine, survivor, swimming, switch, tang, tape, target, taxes, tech, teens, television, tennis, textbook, textbooks, thanks, theater, thomas, timex, toner, tools, topics, torino, toys, training, transformers, travel, trinity, triumph, truck, trucks, trump, tuna, tuttle, tv, twilight, umbra, universal, university, unlocked, upgrade, urban, used, verve, video, vikings, vinyl, vivo, viz, volleyball, vs, wallace, warner, watches, waterskiing, weber, werner, west, westinghouse, wheeled, wheels, wiley, winds, wire, wireless, wood, woodworking, wrangler, wrestling, write, xerox, yoga, zenith, microsoft

Product Pages

For some reason, Amazon has chosen to create shortcuts to individual products in their catalog. I found 100 shortcuts in this category.

america, apartment, arthur, badlands, balls, barrels, barry, beans, beats, body, bottom, brazil, breakaway, breakthrough, buckets, century, chaplin, chef, commitment, confidence, conversations, cookie, corner, cracker, dash, dating, deadline, deceit, details, dow, drawing, dungeon, eades, earl, espionage, evolve, excel, fish, fit, garcia, gunslinger, hillman, hive, icon, inclusion, indispensable, inspire, intrusion, israel, joss, kimball, kindle, leo, loopholes, millionaire, mira, mummy, noel, nostradamus, nourish, oasis, organizing, oyster, palmer, paralysis, passion, passionate, peacock, persuasion, poe, portfolio, presence, profitable, prosper, revved, santa, sarah, screenplay, seal, self, selling, service, ship, sloan, spies, startup, stocks, stranger, swimsuit, talent, toxic, traveler, waste, we, whistle, wind, winners, wired, wizard, wolves

Sports Team Pages

I found 178 custom pages dedicated to products related to sports or sports teams.

aggies, akron, alabama, angels, arizona, arkansas, army, arsenal, athletics, auburn, avalanche, baylor, bears, bills, blues, bradley, braves, brewers, broncos, brown, bucks, buffalo, buffaloes, bulls, butler, cal, california, campbell, capitals, cardinals, centenary, charleston, chiefs, cincinnati, citadel, colorado, colts, connecticut, cornell, cowboys, coyotes, creighton, cubs, dartmouth, davidson, delaware, devils, dodgers, dolphins, drake, drexel, ducks, duke, eagles, evansville, flames, florida, flyers, georgetown, georgia, giants, hampton, hartford, harvard, hawaii, hawks, heat, hoops, houston, howard, idaho, illinois, indiana, indians, iowa, irish, islanders, jacksonville, jets, kansas, kentucky, kings, lafayette, lamar, liberty, lightning, lions, longhorns, louisville, magic, maine, manhattan, marquette, marshall, maryland, massachusetts, mavericks, memphis, mercer, mets, miami, michigan, minnesota, mississippi, missouri, monmouth, montana, nationals, navy, nebraska, nets, nevada, niagara, northeastern, northwestern, ohio, oilers, oklahoma, orange, oregon, orioles, pacers, pacific, packers, panthers, patriots, pennsylvania, phillies, pirates, pistons, pittsburgh, portland, princeton, providence, raiders, rangers, rays, reds, rice, richmond, rider, rockets, rockies, saints, senators, sharks, southern, spurs, stanford, stars, steelers, stetson, suns, syracuse, temple, tennessee, texans, texas, tigers, titans, tulane, tulsa, twins, ucla, utah, vermont, virginia, volunteers, wagner, warriors, washington, wild, winthrop, wisconsin, wyoming, xavier, yale, yankees

Search Pages

14 pages are shortcuts for product searches.

bowman, browning, buckley, choose, converse, curling, lauder, olympic, pickup, plot, poker, razor, rink, slugger, steiner, thermostat, thermostats, wilson

Website

20 additional shortcuts send users to pages not on the primary www.amazon.com domain.

affiliates, associates, auction, cell, de, developer, fresh, investor, meters, news, phrase, phrased, phrasemaking, phraseology, phrases, pr, press, queue, soap, wine

Special

114 shortcuts defied categorization. They are generally special features of the Amazon web site.

access, advantage, agreement, anywhere, atonement, audible, bach, bank, barney, bill, birthday, blacks, cancel, capote, card, cards, cart, chronicle, claim, claims, classics, communications, connect, contact, coop, current, dad, daily, deals, delivers, direct, directory, discover, education, environment, episodes, feedback, garage, gifts, goldberg, greenwood, habitat, hd, heart, help, holiday, honor, horton, howell, images, inaugural, inauguration, international, jaws, journeys, kaplan, labels, library, lippincott, locker, marketing, marketplace, martian, mcgraw, mobile, mom, montreux, morningstar, nolo, nonprofit, occasions, orders, packaging, passes, password, phone, pin, pioneer, prime, privacy, productivity, publishers, push, quiver, refunds, reporting, return, returns, rewards, search, sell, shades, sos, sox, spa, studios, summons, tags, ted, today, tracking, trading, usability, valentine, vault, visa, visual, vote, wastewater, waters, wedding, wishes, zen, zing

Other

Perhaps most interestingly, I found 68 shortcuts that no longer seem to function specially (they default to the home page), or they reference a secured page not intended for Amazon customers

alerts, alien, ash, awards, beta, blocks, board, brooks, buick, community, credit, dell, documents, egghead, errors, es, evening, exec, favorites, fiesta, friends, game, ginger, go, ibm, input, java, join, lifetime, local, manage, mutter, new, orchestras, phantom, preferences, profile, recommendations, reminders, review, reviews, schwab, sellers, star, status, store, tag, text, traffic, wolf, yellow,
anniversary, communities, continue, damages, dexter, discussion, discussions, dylan, forum, forums, japan, japanese, links, madonna, mae, patents, supernatural

Time-Score - A New Algorithm for Calculating Time-Weighted Scores

One of the reasons I created Go2.me was as a test platform for a new social scoring algorithm I've developed called "Time-Score". You'll note that Go2.me calculates the most popular links on the site and updates it real-time.

Each time a user interacts with content on Go2.me, the site accumulates points as follows:

• Sharing a Link - 2 points.
• Favoriting an Link - 2 points.
• Commenting on a Link - 7 points.
• Viewing a Link - 1 point.

By time-weighting the scores, actions taken today are twice as valuable as actions taken yesterday (we say the "half-life" of a link-score is 1 day). So, if 100 people visit a link today, it will rise to the top of the popular list (with over 100 points). By the same time tomorrow, the score will have decayed to just over 50 points if no further people visit the link. Each day, it will be cut in half, so that after a week, it will effectively have less than 1 point, and more recently active links will rise to the top of the popularity list.

This ensures that the most popular links have to continually get the attention of the Go2.me audience to stay at the top. This scoring algorithm also applies to every tag applied to Go2.me links. For example, the most popular videos are shown using the same scoring algorithm.

I'm applying for a patent for this algorithm now, as I think it has several advantages over other scoring algorithms that I've seen or implemented in the past:

• Real-time updates - Scores are instantaneously updated and can be compared against all other scoring values in the database.
• No Batch Processing - The scoring decay function (half-life) does not require any batch processing of older scores to normalize them for subsequent days.
• Combines scoring events at distinct times - There is a well defined relation to combine scores at different times, based on the characteristic half-life of the score.

Since most sites don't publish how they do popularity ranking, I can't be sure if my algorithm is unique. Blog posts about Reddit's algorithm indicate some similarity - but they lack a mechanism to combine scoring events at different times (they calculate all events as if they occurred at the time the post was first submitted).

The code for the Time-Score algorithm is quite simple. I have granted a royalty-free license for all non-commercial users. If you're interested in licensing this technology for your commercial application, please contact me at info@go2.me.

Exhaustive List of URL Shorteners

Go2.me started as a small hobby project, and has grown in scope as I've found many possible features to add.

There are a wide variety of Link Shorteners in the market today. The first products appeared in 2001, and people have used them to shorten 100's of millions of URL's to date. With the advent of micro-blogging sites, like Twitter, the demand for URL shorteners has grown even greater.

TinyURL is the first URL Shortener to gain wide popularity. Started in 2002 by Kevin "Gilby" Gilbertson, TinyURL remains the single most popular URL Shortening service, with a claimed 145 million links shortened to date, and 1.5 billion clicks per month (nearly 600 per second) on their shortened links (as of January, 2009).

There are a number of reasons why people are using URL Shorteners:

• Fixing Links - Original links may be too long to fit on a single line in an email message, causing some email clients to wrap them, making the link unusable.
• Beautifying Links -Original links are long and ugly - a shorter one looks better in email or IM messages.
• Space requirements - Twitter messages only allow 140 characters - long URL's won't fit, or don't allow enough explanatory text to accompany a shared link.
• Spam and Phishing - Spammers sometimes use URL Shorteners to disguise the true destination of a link. They can claim it will take a reader to one destination, when if fact it goes to another. This can make it more difficult for spam filters to recognize spam messages, and can also fool users into falling for a phishing scheme.
• Link Tracking - Individuals and marketers who are sharing links, often like to know how many people are clicking their link, or know what site people are clicking from.

TinyURL defined the baseline functionality by which we can judge follow on competitors. As an early entrant, the feature set is basic, with a couple advanced features.

• Redirection - Redirects from a short URL back to the original.
• Web Form - Type a link into a web form to shorten it.
• Custom Alias - Select your own text (if available) for a tiny link (e.g. http://tinyurl.com/blog-go2me.
• Bookmarklet - Install a bookmarklet to create a short URL from the current page.
• Relative Redirection - TinyURL links can be appended by additional URL path text to redirect to a "sub page" of the original shortned URL.
• Link Preview - Turn on preview mode (remembered by cookie) to display the target link BEFORE redirecting to it (used to confirm link is not spam, adult, or phishing).
• API - TinyURL supports a simple web request that will return the tiny URL from a long link - very simple but it works for many applications, for example Twhirl.

In a future post, I'll review the most popular (stand-alone) URL Shorteners (and throw in a comparison to the Go2.me service as well); sites that do URL shortening as an added feature (e.g., FriendFeed's "ff.im") will not be considered.

"Complete" List of URL Shorteners

The following list is EVERY URL shortener I have found to date (and I'll update it as I find more - if you know of additional shorteners, please send mail to shortener@go2.me). As a measure of their relative popularity, they are listed in order of their published Alexa Ranking (most as or Jan 2009, but some updated in Apr 2009). The sites that are highlighted will be the subject of my next post - a competitive review of the features of the most popular URL Shorteners.

Update 1: I've estimated the number of Tweets per Day that each service is being used to send. The front runners on the list are sorted by that statistic, rather than Alexa rank as it seems a purer measure of popularity.

Update 2: Twitter Rate numbers from May 2009 - also showing relative change in rank of each shortener from last statistics in April.

Update 3: Updated June 11, 2009. Note Stumble Upon's new shortener su.pr is already getting lots of mentions on twitter, even though it's still in limited beta.

The LAST Update: I'm going to post updated stats in a fresh blog posts from now on.

Thanks to Chris Koss for doing the competitive research for this post.

Feature Update

As many Go2.me users will have noticed, I launched a pretty major new release of Go2.me on Feb 28, 2009. I had been working on a large set of changes for about 6 weeks, so I felt it was finally time to release what I had.

I must admit that some of the changes have been a little rough around the edges, as I rushed to complete features by my self-imposed deadline. But I feel that overall, the new version is a big improvement, though much more "heavy weight" than the original vision for Go2.me.

Here's a highlight of the what you'll find new in this release:

1. New Design - Yes, it's RED. But a bit more designed than the previous site. I tried to make it easier to understand that you can install the bookmarklet to get started.
2. Link Page Sidebar - The Go2.me comment area has moved to a side-bar (rather than the top of the page) where most people have extra real-estate. Anyone using a wide monitor should see more of the target web page without having to scroll. You can also click the expand/collapse triangle in the header of each sidebar panel to hide sections to make even more room for the Chat window.
3. Real-time Chat - Go2.me is now a fully AJAX application which allows you to chat in real-time with readers visiting your links. You can also see how many people are currently looking at your page (hidden feature - if you hover your mouse over the thumbnail of a user image in the chat window, it will tell you where in the world that person is located).
4. Sponsored Link Pages - While not an end-user "feature", per se, Go2.me is now accepting sponsorship to help pay for the costs of providing our service. If you get annoyed by the ads, you can shrink the Sponsored By panel so you don't have to look at it.

I had to punt a few features that were under development but not yet ready. These should be coming soon:

• Formatting Fixes - There are some page formatting problems I know about - especially in IE. Most of my users are on Firefox or Safari, but I will be addressing the cross-browser problems shortly
• Private Chat - You'll be able to create a link that only you and your friends know about to have a private chat about a web page.
• RSS feeds - Subscribe to personal or tag-based pages of the mosts popular links.

What's a "Real User"? Twitter Bots are Crawling Go2.me links.

One of the features of Go2.me is that it can tell you how many users are visiting your link. One problem I've been noticing lately, is that I am counting users that aren't real people.

I experimented with twittering a link, and then looking at the viewers of that link shortly after. I'm seeing a number of Bots crawling twittered links right away. From my server logs I see the following request that got counted as real users:

66.249.65.99 - - [10/Mar/2009:11:57:13 -0700] "GET /xL HTTP/1.1" 200 8416 - "Mediapartners-Google,gzip(gfe),gzip(gfe)"

67.202.28.213 - - [10/Mar/2009:12:00:25 -0700] "HEAD /xL HTTP/1.1" 200 9468 - "AideRSS 2.0 (postrank.com),gzip(gfe),gzip(gfe)"

xx.xx.xx.xx - - [10/Mar/2009:12:00:56 -0700] "GET /xL HTTP/1.1" 200 9468 - "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.1.11) Gecko/20071204 Ubuntu/7.10 (gutsy) Firefox/2.0.0.11 FirePHP/0.0.5.26,gzip(gfe),gzip(gfe)"

66.29.162.122 - - [10/Mar/2009:12:01:22 -0700] "GET /xL HTTP/1.1" 200 9469 - "WWW::Shorten::TinyURL/1.90,gzip(gfe),gzip(gfe)"

89.151.116.52 - - [10/Mar/2009:12:01:35 -0700] "GET /xL HTTP/1.1" 200 9469 - "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-GB; rv:1.9.0.6) Gecko/2009011913 Firefox/3.0.6 TweetmemeBot,gzip(gfe),gzip(gfe)"

216.121.78.20 - - [10/Mar/2009:12:08:03 -0700] "HEAD /xL HTTP/1.1" 200 9468 - "Chat Catcher - (http://chatcatcher.com/bot.htm),gzip(gfe),gzip(gfe)"

91.121.89.155 - - [10/Mar/2009:12:08:33 -0700] "HEAD /xL HTTP/1.1" 200 9470 - "PycURL/7.19.3,gzip(gfe),gzip(gfe)"

Note that the link was posted at 11:57 - so these requests came in from 6 seconds to 10 minutes after Tweeting the original link. Only one of these listed was a "real human". So, Go2.me is counting 6 unique users that are actually Bots.

To fix this, I'm going to be changing my counting algorithm to require that you've successfully received a Go2.me cookie (as a normal web browser would do), AND that your browser has successfully made an AJAX request to the server to receive a Chat update. This should eliminate all Bots and Crawlers as falsely being reported as "Vistors" to Go2.me links.

Go2.me Goes International

This morning, Go2.me was "discovered" by Web 2.0 directory Go2Web20.net. As a result, there was a burst of activity, especially in Europe, of people visiting Go2.me and writing blog posts about it (in French and Italian).

It's been a pleasant surprise to see a bunch of new faces trying the service out, and twittering about Go2.me.