Skip navigation

Tag Archives: callbacks

Push/callbacks are better than polling.

This should be an easy argument. It’s not just more efficient, it’s more real-time. I once heard something like 40% of requests returned 304 Not Modified. I imagine similar numbers for other popular sites. The more real-time you try to be with polling, the worse it gets. Don’t call us, we’ll call you. It’s more efficient.

RPC solutions provide more value than messaging solutions.

Although you can argue that RPC can be powered by messaging, and messaging can be used for RPC, the point is that they are different mindsets. RPC is about triggering code that does something. Messaging is about putting a piece of data in another bucket. When you try to call somebody, you usually don’t want to get their voicemail, right? This analogy goes even further in that, like voicemail, it must be checked on the other end. Messaging just pushes polling somewhere else.

More importantly, the focus on triggering code is central to this value proposition. Generative systems are more valuable than sterile systems. When did the web get interesting? When it became about more than just static content, and code was put in the loop to generate dynamic content. The point is that if you prioritize code to receive a message before humans, you open up many more possibilities.

RPC is about making things happen. Messaging stops short short of that by just moving data around.

HTTP is the defacto RPC protocol.

HTTP is everywhere. There are powerful free servers, clients in every major programming environment, and people know it well. It’s proven and it just works. HTTTP’s simple design also allows it to be extremely versatile. It’s basically the TCP of the application layer.

However, the best thing is that HTTP is RPC. This subtle fact has been true ever since CGI was introduced. We’ve gone through building RPC on top of it with XML-RPC and SOAP, but wisely settled on a form of RPC that’s just HTTP and is even aligned with HTTP semantics: REST.

If HTTP is RPC and HTTP is everywhere, it is our defacto RPC protocol. Especially for web applications that breath HTTP, it almost doesn’t make sense to think of any kind of inter-application communication that isn’t HTTP. Turtles all the way down!

HTTP RPC + Indirection = Webhooks

David Wheeler said, “All problems in computer science can be solved by another level of indirection.” Webhooks, and all callbacks, are about taking a procedure call and performing it on a variable function. This is indirection and this is very powerful. This is why Unix pipes work. STDIN and STDOUT are not hardcoded values, they’re variables that you can control.

Now imagine if all the web applications you used had extension points that you could effectively hook together with any other application. Well, that’s what webhooks are about.

Our beloved webhook debugging tool known as PostBin just got a little bit more useful. You can now include a URL in the query string when registering your PostBin callback that will get invoked when PostBin is posted to. This way you can use PostBin and still have your hook script run. Here’s an example of what it would look like:


Notice you can just prepend your existing callback URL with a PostBin URL and a question mark. In this way, PostBin becomes a decorator to your hook scripts.

A lot of work went into this and a lot of plumbing and debugging. PostBin uses Hookah to invoke your passthrough callback, making PostBin the first real user of a Hookah instance. Hookah was rewritten in Twisted and got a lot of tweaks to make this work. Please consider using Hookah to handle asynchronous callback dispatching in your apps!

PostBin has a number of features planned, including private bins, request replaying (so you can capture events, then replay them to your passthrough endpoint for debugging), filtering, and more.

Today I did a 7 question interview for Jason Salas on web hooks. Pretty standard stuff, but it allowed me to bring up some things I haven’t written about yet. His final question was about barriers going forward with the movement, which I also answered with a general “what’s coming” sort of response:

It seemed like the biggest hurdle originally was getting people to wrap their heads around this idea. I would always talk about it in the abstract and go on about all the implications of what was essentially one line of code. I think there are enough fairly well-known examples now that it’s easier for people to join the party. Even then, the general perception of what’s possible is going to be limited by the examples.

Like AJAX, you can’t just build a popular example of AJAX without it being a useful tool itself. I can build all the web hook prototypes I want, but it’s not until the Googles and Facebooks implement them in a useful way that people will really see the value. Until then, we get incremental boosts by the smaller companies like Gnip, GitHub, and others. I’ve started working or talking with these guys to get them involved in a collective conversation around web hooks, so we can work out issues standing in front of adoption.

The issues people come up with are usually security and scalability related. As it turns out, some of these issues have been solved by these guys already doing it. So I’m trying to get more of them to share best practices and publicize their use of web hooks. This way people can start seeing the different ways they can be used. For example, the Facebook Platform, although pretty complicated and full of their own technology, is still at the core based on web hooks. They call out to a user-defined external web application and integrate that with their application. That’s quite a radically different use of web hooks compared to the way people think of them in relation to XMPP.

Moving forward, I think we’re going to see more libraries and tools that have solutions to scalability and security built-in. I’ve started one project called Hookah that I’m hoping to get released soon. It provides basic callback request queuing and management of callback URLs so you really can implement web hooks with a single line of code for each event. We’re also starting to see similar helper libraries for frameworks like Django and Rails.

Eventually we’ll be seeing specs for doing specific things on top of web hooks. One of the first things on my list of standards to look into is the way in which you register and manage callbacks in a programmatic way. Many web hook providers use a web interface to manage your callback URLs. We’ll see some neat things happen when you can manage them via APIs so that tools can set callbacks with services on your behalf.

Anyway, one of the reasons I’m so attached to the idea of web hooks is that I see a lot of long-term potential. Especially when you integrate them into other visions of the future, like the Web of Things. When you combine the Programmable Web with the Web of Things, you get a world of Programmable Things.

That’s where I’d like to see this end up.

I once had to build a shopping cart framework for a friend’s company in about three days. It was a challenge considering all the complexities involved in pretty much any instance of a shopping cart. Most shopping cart solutions would hardcode predefined rules or configurable options for determining tax, shipping, and promotional discounts. This was kind of ridiculous since those are usually the points where it varies quite a bit from cart to cart. My solution was to provide hooks because I knew these guys were programmers and were smart enough to implement their own callback functions to define whatever rules they wanted. It turned out to be very powerful, one of their favorite features of the framework, and it was trivial to implement.

So how does an architecture like that translate to a hosted e-commerce checkout solution? Obviously, web hooks! Amazon recently figured this out as they just announced a Callback API for their Checkout solution under their Amazon Payments arm. It lets people use their hosted “checkout solution” while letting them customize the rules for calculating taxes, shipping, and promotional discounts in the most powerful way: code.

This is a fairly significant example of web hooks because it breaks away from the notification use-case that people seem to be caught up with. By actually processing the results of a hook callback request, you’re opening up your application logic to the end user. This is user contributed logic, an under appreciated and mostly unrecognized trend for the emerging service platform aspect of the web.

I don’t want to tangent too much into a rant on the web I want (and I think we all want, but don’t realize it). Instead, I’ll summarize some details on this new Amazon announcement of web hooks. The full documentation for which can be downloaded in PDF format.

You define the callback endpoints in a configuration XML document used for setting up Checkout. They use HMAC authentication for hook requests with added timestamp nonsense, very similar to the way they do Amazon Web Services. The payload is somewhat surprisingly not XML, but instead a URL encoded key/value pair format. The strange thing is that they try and do nested data structures this way, which makes me think JSON would be the better choice. And just to confuse you slightly, in the documentation they use XML to describe the data structures they pass to you and expect back, even though XML is not actually involved. Nevertheless, the documentation is quite good.

Anyway, I’m happy to see a use-case emerge in a way that starts to hint at the aspect of this model I’m more interested in: customization via code. Notifications, or “push,” are really just something along the way. Although this is a trivial example, you should be able to start imagining how it could be used to build a plugin architecture for your web app. The ability to allow your users to extend and build new functionality they can share with others does not seem like an empty value proposition!