Skip navigation

Monthly Archives: November 2008

Today I went to Mashup Camp and gave a session on Web Hooks. It was mostly my standard talk with minor updates, but framed in the context of mashups. It seemed to go well, and I think I was even invited to talk about them at Google. John Musser from Programmable Web was there, but missed the talk. However, I did get to talk to him and he seems genuinely interested in it. He mentioned perhaps even including event hooks as part of the PW directory.

Anyway, here are the slides I unshifted upon the deck I usually show. The notes are sort of poor quality since I added them afterward in a rush to get these online.

Today Gnip announced that they’re winding down XMPP support. Why? They say it’s not worth the work to support it right now. The XMPP world still lacks a leading server standard, leaving them to deal with a scattered array of various implementations. On top of this, many of their consumers are unwilling to deal with running their own XMPP server and end up using Google Talk or, which throttle throughput since they’re popular servers.

Web hooks don’t have these problems. This is probably why Gnip continues to use them as their primary mechanism for real-time push notifications. As ideal as XMPP sounds for real-time notifications, it lacks the simplicity and ubiquity of HTTP. Not only is it a new server implementation to worry about, it’s another type of server in the mix!

Anyway, this means I need to go back and edit my slides about Gnip. Despite this seeming like a victory for web hooks (when I contacted Jud Valeski about this, the Gnip CTO responded “Web hooks reigns!”), I was looking forward to Gnip providing a gateway between the world of web hooks and the world of XMPP. They’re likely to return to XMPP eventually, but this may be an opportunity for somebody to provide a focused XMPP to web hooks bridge.

As a sidenote, I’ve thought it interesting that Gnip is aware of “web hooks” as they use it in their diagrams, but their public use of the phrase has otherwise been lackluster. In their write-up of the announcement, ReadWriteWeb seems to have taken it upon themselves to call Gnip’s use of web hooks “Restful push.” Not bad. I’d probably call it that myself if it wasn’t about more than push. But that’s what the social media guys are into, and I don’t have a problem with that since social media is a trendy vehicle for web hooks.

I always try to point out that web hooks are nothing new. In fact, most “new” technology is not new. The mouse, for example: in 1984, when Apple introduced the Macintosh 128K, the mouse and GUI exploded in popularity and quickly became the standard model for interfacing with computers. Little did people know, the mouse had already been around for 21 years! It was invented at SRI by our pal Douglas Engelbart way back in 1963.

It’s interesting to note that the mouse is sort of useless without the GUI. The two are pretty strongly coupled. Perhaps the latent success of the mouse was in part due to the GUI taking about 20 years to be perfected and made ready for mainstream acceptance. In any case, most new technology is not new. Remember that, kids!

The name “web hooks” might be new, but the idea isn’t. The technology certainly isn’t; it’s just HTTP. In fact, you can say the same thing about REST, which wasn’t coined until 7 years after HTTP.

So what was the first major application of web hooks?

The classic example of a web hook is PayPal’s Instant Payment Notification feature, also known as IPN. This feature allows online retailers to handle real-time purchase confirmations from PayPal. After giving PayPal a URL for IPN, they perform an HTTP POST to it whenever the seller receives a payment. This allows the retailer to better integrate the rest of their system with PayPal.

It was a pragmatic solution to achieve simple server-to-server communication. What blows my mind is why it hasn’t been used for much more since then, hence this blog and my slight obsession with the idea. There have been others that have made use of this more recently, and I’m going to cover those guys on this blog, but still… IPN has been around for over 5 years.

I’m not sure how long PayPal has had the feature, but I know it dates back to at least 2003. And PayPal wasn’t the only one. There used to be (and still are) a lot of alternatives to PayPal which provide the same feature. Even Google Checkout has an IPN equivalent.

But the popularity and success of PayPal has given them loads of experience with IPN. They’re basically web hook veterans. I’d love to talk to some of their engineers about their experience providing the IPN feature. How it evolved, maybe how it scales, how the community likes using it, what kinds of tools have been made around it…

Actually, one thing I didn’t realize until I researched it for this post, is that there are 3rd party services that consume the IPN web hook to provide extra features for the retailer. For example, Scrobbld extends the functionality of PayPal with a nice web UI to manage post-sale affairs without even entering your PayPal username. It’s based entirely on consuming the PayPal IPN.

Now an interesting thing to note about PayPal’s implementation of IPN is their confirmation step. Instead of just sending the IPN in a fire-and-forget fashion, they want you to perform an HTTP POST back to them with all the data from the IPN. They use this to prevent spoofing and let you know the IPN did, in fact, come from them.

The HTTP status code you respond to the IPN with is used to determine if they should resend the notification. If you return anything other than 200 OK, they try to resend the IPN in full for up to four days.

And although some people like the idea of a structured payload document in their web hook posts, PayPal IPN data is straight up simple key/value request parameters. This eliminates the need for the consuming script to parse anything beyond what their CGI library probably already takes care of. In the case of PHP, all the data is right there in $_POST.

So in general, I like PayPal’s approach to IPN. They’ve had plenty of time to get it right and do a good job at keeping it simple. Although they might not be the very first example, they’re definitely the most used example of web hooks in action.

Now, I was just thinking about the mouse and GUI analogy I made at the beginning here. Web hooks are a complement to web API’s in much the same way the mouse is a complement to the GUI. In fact, like the mouse, without their API counterpart, I think web hooks are much less useful. Without a GUI, how useful is a mouse?

Perhaps the reason web hooks are taking longer to be adopted is similar to the mouse waiting for the GUI to develop. Even though IPN was around in 2003, it wasn’t until 2005 that web API’s started to become popular. Now that API’s are pretty commonplace, I guess it makes sense web hooks are starting to grow in popularity…

It’s always nice to start a blog with some really juicy content. Luckily I recently put together a 40 minute talk on web hooks and put all the slides on SlideShare, which I can share here. The talk itself is mostly in the notes, but the slides play a big part. At some point I should have a video recording to put up here. Until then, enjoy my slides since I put a lot of work into them! I also wrote an introduction to the talk on my blog.