Skip navigation

I haven’t been posting here lately, but lots of amazing things have been happening in the world of webhooks. At SXSW I gave my latest version of a talk to explain webhooks, and I went on to describe the greater “evented web” that they unleash. It also explains some of the motivation I have behind all this.

Push/callbacks are better than polling.

This should be an easy argument. It’s not just more efficient, it’s more real-time. I once heard something like 40% of delicious.com requests returned 304 Not Modified. I imagine similar numbers for other popular sites. The more real-time you try to be with polling, the worse it gets. Don’t call us, we’ll call you. It’s more efficient.

RPC solutions provide more value than messaging solutions.

Although you can argue that RPC can be powered by messaging, and messaging can be used for RPC, the point is that they are different mindsets. RPC is about triggering code that does something. Messaging is about putting a piece of data in another bucket. When you try to call somebody, you usually don’t want to get their voicemail, right? This analogy goes even further in that, like voicemail, it must be checked on the other end. Messaging just pushes polling somewhere else.

More importantly, the focus on triggering code is central to this value proposition. Generative systems are more valuable than sterile systems. When did the web get interesting? When it became about more than just static content, and code was put in the loop to generate dynamic content. The point is that if you prioritize code to receive a message before humans, you open up many more possibilities.

RPC is about making things happen. Messaging stops short short of that by just moving data around.

HTTP is the defacto RPC protocol.

HTTP is everywhere. There are powerful free servers, clients in every major programming environment, and people know it well. It’s proven and it just works. HTTTP’s simple design also allows it to be extremely versatile. It’s basically the TCP of the application layer.

However, the best thing is that HTTP is RPC. This subtle fact has been true ever since CGI was introduced. We’ve gone through building RPC on top of it with XML-RPC and SOAP, but wisely settled on a form of RPC that’s just HTTP and is even aligned with HTTP semantics: REST.

If HTTP is RPC and HTTP is everywhere, it is our defacto RPC protocol. Especially for web applications that breath HTTP, it almost doesn’t make sense to think of any kind of inter-application communication that isn’t HTTP. Turtles all the way down!

HTTP RPC + Indirection = Webhooks

David Wheeler said, “All problems in computer science can be solved by another level of indirection.” Webhooks, and all callbacks, are about taking a procedure call and performing it on a variable function. This is indirection and this is very powerful. This is why Unix pipes work. STDIN and STDOUT are not hardcoded values, they’re variables that you can control.

Now imagine if all the web applications you used had extension points that you could effectively hook together with any other application. Well, that’s what webhooks are about.

Although it’s not the most compelling story, this blog post is a terribly effective analogy. So effective, non-techies can read it and “get webhooks” … in some cases leading them to rally for webhooks as much as I do! The analogy focuses on a non-computer, real-world analogy based on telephone calls. Then it follows up with a more concrete example that helps explain the possibilities:

A concrete example of a story made possible from webhooks that might be a useful scenario for many of you involves Twitter. Let’s say Twitter supported webhook callbacks for when somebody follows you. Right now you get an email, and from there you can decide what to do manually: follow them back, block them, or do nothing. I used to go out of my way to block users that I knew were spam bots, but now there’s so many it’s not worth the time. And of course I also generally follow back people that I actually know. If Twitter would simply call a script of mine whenever somebody followed me passing along the user ID, I could very easily run this logic in a PHP script or a simple App Engine app. Or perhaps I’d use Scriptlets (ahem, which was made exactly for these kinds of web scripts). It would work like this:

First, use the Twitter API to look up the user from the ID, and grab their name. Then use the Facebook API to check if that name shows up in my list of friends on Facebook. If so, use the Twitter API to follow them back. Otherwise, if they’re following over 1000 users and that number is more than twice the number that’s following them (which is roughly the heuristic I use manually), use the Twitter API to block them. All automatic.

Definitely worth the read, if I do say so myself. It’s also worth pointing people that want a quick understanding of webhooks. What kind of analogies have you come up with?

A while back I was interviewed on the Nearsoft blog about webhooks. It goes into more details on the whole push, pipes and plugin use cases. Real-time web is a hot topic these days, so I had to mention how the webhooks movement relates to that trend. Here’s an excerpt about using webhooks for real-time notifications:

Notifications seem to be [a big draw for webhooks] … as in, tell me when something new has been posted, or when something has changed. But with your code on the receiving end of the notification, you can decide exactly how you get notified. For example, tell me about changes over Twitter, not email. In fact, no, use this other hook script that uses cloud telephony to call my cell phone and use text-to-speech to tell me.

Real-time notifications, exactly how you want them.

So much has been going on in the world of webhooks that it’s hard to keep up. From PubSubHubbub developments, to new services supporting webhooks … to this plugin for WordPress called HookPress. HookPress opens the WordPress plugin API over webhooks! It was developed by my new buddy Mitcho. He made an excellent screencast to demonstrate the power of HookPress, and the power of the emerging webhooks ecosystem. Check it out:

Last month I talked about webhooks at Glue, a new conference on the “glue” of the web. One of the other speakers was Josh Elman of the Facebook Platform team. I ended up on the flight back with him, so we talked about webhooks. He seemed excited about them as a fan of the callback/hooking pattern in classical programming. In fact, he mentioned how they were experimenting with them already for notifications within the Facebook Platform, but he brought up the issue of batching. Something I hadn’t thought of before, but is important for large-scale implementations that are likely to be posting a lot of events to a target endpoint.

Later, when I read about Google Wave, and how they use webhooks in their API for creating bots, they mentioned that they may batch events. I’ve yet to crack open my Google Wave developer account and play with their implementation, but I’ve since realized a very simple convention for batching: JSON lists as the outer structure.

This works because an event object is ALWAYS going to be a key/pair object. So receiving code just has to check if the payload JSON is an array or an object. If it’s an array, handle each object inside as separate events. If it’s an object, handle as you would.

This doesn’t work for POST variables, since it’s very difficult to do arrays in general, but particularly as the outer data structure. It’s just not made to be able to do that. So this would require you to use at least JSON. It works in XML as well, but because there’s more variety in XML payloads, it’s probably harder to settle on such a simple convention as this.

Let me know what you think. Or if any of you already know how Google Wave batches events in their Robot API.

The webhooks ecosystem is growing pretty fast and it’s really hard to keep up! There’s also a growing number of webhooks evangelists giving talks and writing blog posts about webhooks. It would be great if we could work together to help push this movement even further.

I really would like this blog (and eventually website) to be a place to learn about how to implement webhooks, best practices, and standards, as well as a place to see the movement grow. This means seeing services adopt webhooks, how they’re using them, how their users are using them … new services that provide infrastructure in this kind of event-driven programmable world… and software projects that help. Not just my projects like Hookah and Protocol Droid (yet to be announced), but all the plugins people are writing for various frameworks and open source projects.

So we need more writers on this blog. A lot of people send me links to services that adopted webhooks and I wish I could write about them all. I usually mention them on my Twitter, but that’s not enough to really say everything worth saying about some of them.

If you’d like to join us and write on this blog about webhooks, recurring or as a one-off piece, about anything from how you implemented webhooks, to how you did something clever with them, or why people should adopt them … get in touch with me. :)

Our beloved webhook debugging tool known as PostBin just got a little bit more useful. You can now include a URL in the query string when registering your PostBin callback that will get invoked when PostBin is posted to. This way you can use PostBin and still have your hook script run. Here’s an example of what it would look like:

picture-19

Notice you can just prepend your existing callback URL with a PostBin URL and a question mark. In this way, PostBin becomes a decorator to your hook scripts.

A lot of work went into this and a lot of plumbing and debugging. PostBin uses Hookah to invoke your passthrough callback, making PostBin the first real user of a Hookah instance. Hookah was rewritten in Twisted and got a lot of tweaks to make this work. Please consider using Hookah to handle asynchronous callback dispatching in your apps!

PostBin has a number of features planned, including private bins, request replaying (so you can capture events, then replay them to your passthrough endpoint for debugging), filtering, and more.

I forgot to mention it on here, but yesterday I gave a talk at Pivotal Labs. It’s a whole new talk that tries to get a little bit more into technical implementation details. The slides are also arguably more stylish. Pivotal recorded the talk, so video will be up soon, too. Until then, here are the slides.

And feel free to use these slides or bits from the deck for your own webhooks talk! You can download the original Keynote presentation here.

Follow

Get every new post delivered to your Inbox.