Skip navigation

Category Archives: Ecosystem

I haven’t been posting here lately, but lots of amazing things have been happening in the world of webhooks. At SXSW I gave my latest version of a talk to explain webhooks, and I went on to describe the greater “evented web” that they unleash. It also explains some of the motivation I have behind all this.

So much has been going on in the world of webhooks that it’s hard to keep up. From PubSubHubbub developments, to new services supporting webhooks … to this plugin for WordPress called HookPress. HookPress opens the WordPress plugin API over webhooks! It was developed by my new buddy Mitcho. He made an excellent screencast to demonstrate the power of HookPress, and the power of the emerging webhooks ecosystem. Check it out:

The webhooks ecosystem is growing pretty fast and it’s really hard to keep up! There’s also a growing number of webhooks evangelists giving talks and writing blog posts about webhooks. It would be great if we could work together to help push this movement even further.

I really would like this blog (and eventually website) to be a place to learn about how to implement webhooks, best practices, and standards, as well as a place to see the movement grow. This means seeing services adopt webhooks, how they’re using them, how their users are using them … new services that provide infrastructure in this kind of event-driven programmable world… and software projects that help. Not just my projects like Hookah and Protocol Droid (yet to be announced), but all the plugins people are writing for various frameworks and open source projects.

So we need more writers on this blog. A lot of people send me links to services that adopted webhooks and I wish I could write about them all. I usually mention them on my Twitter, but that’s not enough to really say everything worth saying about some of them.

If you’d like to join us and write on this blog about webhooks, recurring or as a one-off piece, about anything from how you implemented webhooks, to how you did something clever with them, or why people should adopt them … get in touch with me. :)

Our beloved webhook debugging tool known as PostBin just got a little bit more useful. You can now include a URL in the query string when registering your PostBin callback that will get invoked when PostBin is posted to. This way you can use PostBin and still have your hook script run. Here’s an example of what it would look like:

picture-19

Notice you can just prepend your existing callback URL with a PostBin URL and a question mark. In this way, PostBin becomes a decorator to your hook scripts.

A lot of work went into this and a lot of plumbing and debugging. PostBin uses Hookah to invoke your passthrough callback, making PostBin the first real user of a Hookah instance. Hookah was rewritten in Twisted and got a lot of tweaks to make this work. Please consider using Hookah to handle asynchronous callback dispatching in your apps!

PostBin has a number of features planned, including private bins, request replaying (so you can capture events, then replay them to your passthrough endpoint for debugging), filtering, and more.

Now that I’ve told you how cool AppJet is for hook scripts, allow me to obsolete it. This morning (as I try to avoid falling asleep as I type) I’m announcing Scriptlets, a tool for throwing online quick web scripts in any of several languages. It’s perfect for writing hook scripts. In fact, it’s what I’ve been dreaming of for web hooks for several years.

When I started thinking about web hooks I figured, as the early adopters, most advanced users would have access to some kind of cheap PHP web hosting for hook scripts. Only later did these cloud code runners appear like App Engine and AppJet. AppJet in particular is the closest to my ideal, as I mentioned before. But it was limited to a particular language and still felt slightly too heavyweight for simple hook scripts (by the name alone it was obviously intended for apps, not scripts).

The ideal I envisioned was a site like a paste bin (I’m keen on Pastie). You know, where you paste some code, pick a language, and you’d get a nicely syntax highlighted, numbered row display of the code with a simple, disposable URL you could use to show the kids in IRC your code that wasn’t working. I wanted that, but that also gave you a URL to run the code. This way I could use it for one-off web scripts that accepted request parameters and everything. No more random one-off PHP scripts on my server. And of course, it would be perfect for hook scripts.

Actually, when I started talking with friends about this, none of this cool cloud computing stuff was big yet. App Engine and AppJet weren’t even around. The closest thing was Ning, which used to host PHP code for you for free. One of my friends (actually, the one that coined “scriptlets”) was interning at Google and knew, but couldn’t tell me that App Engine was coming. Once it came out, he and I started a project like Scriptlets for just Python… but we ran into a wall when I misread the Python documentation and thought App Engine wasn’t letting us eval anything more than one line long. It turns out this was completely wrong, but we’re both pretty busy and we moved on to other things.

But when Google announced Java support on App Engine, even though I hate Java, I knew that the JVM meant they now supported a bunch of languages. I revisited our code, solved the stupid problem, and got several other languages working. And tonight I launched it… with support for 3 languages: JavaScript (uh oh, AppJet!), Python (uh oh, App Engine!), and PHP (uh oh.. uhh… commodity PHP hosting?). Ruby is on its way, and potentially others.

All Scriptlets languages provide an environment with access to the web request and response, and most of the features of that language. All the constraints of the App Engine environment are inherited, meaning no long running processes, no abitrary file system access and no socket connections. (I’m so glad I didn’t have to solve the sandboxing problem by myself!) But you do get to make web requests, and this is exposed with a custom function in JavaScript and PHP called fetch(). It’s a pretty limited environment and not even intended for state maintaining scripts (yet), but it should still prove useful.

Anyway, think of Scriptlets as another community service for the web hooks ecosystem. I have a bunch of these in mind that I’ll be building out. Like PostBin, Scriptlets is open source on GitHub and waiting for you to submit patches. Enjoy!

One feature of web hooks is that they’re somewhat self documenting. With web APIs you need to know what you’re requesting and how to request it, but with web hooks, you’re just handed the data. Some of it might need explanation, but you can get started pretty quickly by just capturing actual data and seeing what it gives you. When I first started playing with web hooks, I wrote a handler script that would email me the data. This worked okay, but it could easily be better.

So today I’ve released PostBin, a tool for capturing and inspecting data from web hooks. It’s sort like of a cross between pastebin and Mailinator, but for HTTP POST requests. It’s very straightforward and simple. No signups, no configuration. Just click a button, get a bin with a unique URL, capture POST requests with that URL, then view those requests and the data in them at that URL.

The source is available and patches are welcome. If you have useful source changes to push back to me, they’ll likely be deployed to the production app running on Google App Engine. If you just have suggestions, you can leave them on our UserVoice page.

Hope you guys find this useful!

I like to claim you can implement web hooks with one line of code. The reality is you need to do more than make inline HTTP requests. You need to manage user callback URLs, you need to make the requests asynchronous, you probably want a retry scheme, you might want authentication or verification, and in some cases a response handling mechanism. To do these things requires quite a bit more work, and those that have done it so far haven’t yet disclosed the details of how they’ve implemented this. I hope to change that, and to speed things up, I’ve started a simple community project I’m announcing today called Hookah.

The goal of Hookah is to take most of the work out of implementing web hooks. Ideally, Hookah will allow you to have all the features you need for successfully deploying web hooks and only need to add a few lines of code to your app. In fact, one line for each event! On top of that, the interface for Hookah is HTTP, so it’s not that different than directly invoking web hooks and works easily from any language.

The project is on GitHub and, as you can see, it’s very simple. Currently what we have in the master branch does asynchronous event dispatching with retries. This should be enough to get anybody started with web hooks. It’s written in Python for simplicity and readability, but Python is pretty ubiquitous and because it’s an HTTP server it doesn’t require you to know Python to use it. Again, it’s very simple and very early, but I hope it can be the start of a collective effort to make web hooks easier to implement.

For now we can bootstrap off the Web Hooks discussion group to talk about it, so if you have anything to say or ask, join up. There is a rough roadmap in the README and if you’re interested in contributing, let us know.

Okay, it’s not really a dispute. That was sarcasm in the title. Just to be clear, since there’s always been a tiny bit of useful friction between the idea of XMPP and web hooks, it’s important to remember they are not mutually exclusive. Anybody hyping a battle between the two is trying to create or is imagining a controversy that isn’t real. Many proponents of web hooks are XMPP proponents as well. In fact, the video some are pushing around to promote this supposed throw down is of two XMPP supporters (including djabberd author Brad Fitzpatrick) demoing a pubsub system based on web hook callbacks. And it takes place at an XMPP meetup, so of course there was going to be some proselytizing.

Nevertheless, Jabber/XMPP is a messaging protocol. Web hooks provide a model for functional extensibility, so they are a platform for many different things. A push-based pubsub messaging system is just one use. Even though I originally wrote about web hooks as a notification mechanism, my mind was nowhere near pubsub. I was more focused on the idea of web service integration and orchestration. With the commoditization of CGI-enabled web hosting, I was thinking about how the popularity of web programming combined with the easy invocation of HTTP requests could be used to make a more useful and functionally extensible web. A more programmable web.

With that said, there is nothing wrong with discussing what you can and can’t do with web hooks and XMPP, but there is not some angry fight between camps. Done. Over. Moving on.

As it turns out, the project demoed in that video, which is called pubsubhubbub, is something I had previously stumbled upon while browsing projects a friend of mine was involved in. There wasn’t much of a description, and I didn’t dive too deep into the code, so I bookmarked it to come back to it. After that demo happened, it seemed I misheard (from several people) that it was some XMPP pubsub system, which totally confused me because… it’s not. It’s a neat, distributed pubsub implementation built on web hook callbacks created by Brad Fitzpatrick and Brett Slatkin (a Google App Engine developer). Hopefully they’ll put some more documentation up as it develops, but it’s just really neat to see some XMPP folks build an open pubsub system with web hooks. Cheers to them!

My involvement at NASA inadvertently got web hooks written about on O’Reilly Broadcast yesterday. Kurt Cagle did a nice write-up on his take on web hooks, and it’s possible there will be more coming from Cagle on the topic. Although the post at first seemed pretty framed around syndication and push, the fact that he says things like “server-side mashups” and sees web hooks as a means to “create orchestration of web services” shows he gets the greater significance of this simple mechanism.

I just wanted to cover a few things that were brought up by Cagle and a few others that have written about web hooks recently.

Replacing Syndication

It seems a lot of people see web hooks as an alternative to poll-based feeds and syndication. Although I’ve claimed before that “feeds are not the answer,” it was in context of the vision of pipes for the web. Feeds were not invented for pipelining. They were invented for simple content syndication, and I think they do a pretty good job at that use-case: answering the question, “Hey, what’s new from you?” That said, Cagle seems to be spot on about web hooks and syndication working together.

One of my original arguments for web hooks was that polling sucks. “Hey, what’s new from you?” becomes “Are we there yet? Are we there yet? Are we there yet?” The thing is, web hooks alone don’t let you ask, “What’s new from you?” Nor do they provide a persistent reference to data. Web hook payloads should be ephemeral.

What seems like an obvious solution in this use-case is to provide a feed and a web hook for notifications of updates to the feed. This way you have the feed, which is nice for people that like polling, but also to have a persistent resource on the web for that content stream. And then you have the optional web hook for getting notified of updates, potentially with the updated data so if you don’t want to retrieve the feed, you don’t have to.

There’s a slightly verbose standard spec (as many are) called GetPingd that shows a way you can do this, but I imagine there are simpler approaches. One thing GetPingd points out is that this is all very similar to the ping services for blogs to notify search engines of new content. The missing element of that system is the ability for anybody to subscribe to the notifications. That is part of the essence of web hooks, as Timothy Fitz recently tweeted:

Remember, HTTP callbacks are nothing new. It’s exposing them to the user that makes it a web hook, and that’s where the emergence is.

Anyway, I can understand asking the question, “Will web hooks replace feeds/syndication?” as a thought experiment in trying to understand this new paradigm. But I have to say, after thinking about this for a long time, they won’t. They might replace certain use-cases for feeds, but if feeds were broken enough that web hooks would replace them, it would have happened already.

HTTP vs XMPP

Now this is where we get into some interesting waters. A lot of people bring up good points on both sides. My stance is simply that web hooks are simpler and just as effective for the majority of use-cases, and therefore the obvious winner. There are less pieces, simpler APIs, existing infrastructure, and it’s debatable whether XMPP is inherently any more performant.

You have to consider the use-cases, though. Part of the vision of web hooks is to have a standard HTTP event mechanism for the web. I just don’t see every web service throwing up an XMPP stack along with their HTTP stack. The two can and will work together when necessary, but as Cagle notes, “web hooks in general may be superior for orchestration.” Remember, web hooks are about more than message passing.

Standards

Cagle briefly touched on the standards issue. I’m sure that having a nice standards document would make for great adoption propaganda, and I know quite well the significance of agreed upon conventions in technology. However, I’m not in a hurry to over engineer anything, and I’m not going to assume we’ll know everything about the implications of this mechanism that we can encode them in a document that will either be ignored or adopted by everybody, making it harder to adapt to change. The longer we can put off standardizing, the better.

In the article, Cagle compares it to AJAX, in that the community isn’t very standards oriented. I’m not exactly sure what the AJAX community would benefit from standards. I’m quite happy that AJAX wasn’t limited by a standard to only use XML. There is nothing wrong with options. That’s kind of the whole point of technology: to provide new avenues, options, and choices for empowerment. Tools will always be used however the tool user finds useful, which is not always how the toolmaker intended.

I would much rather provide examples, and rigorously defined patterns of usage and implementation than try to define a standard. When a globally accepted convention is necessary, then we can work one out (with a useful, ideally proven, implementation), but it will probably be about some aspect of web hooks, not the model as whole. I think the biggest aspect ripe for standardization is for machine-friendly announcement of hooks and mechanism for registering callbacks (aka subscribing). But this is not preventing web hooks from being useful, otherwise nobody would be using them already.

Email is one of our oldest and most used Internet systems. Frankly, though, I hate its dated implementation. Yet I understand how tested and universal it is, and as such, think it would be a good idea to allow it as a means of interaction with web services. I just don’t want to have to touch its crappy internals. This led me to develop the now defunct Mailhook (succeeded by smtp2web and Astrotrain) as a way to leverage web hooks to help ease the pain of accepting email in our web applications. It was a simple SMTP server that would pass parsed email to a URL you specify as if it were a form post.

What Mailhook did not do was provide programmatic access to existing mailboxes. That’s okay, though. We’ve got IMAP with libraries in most languages, so that’s pretty taken care of, right?

Meh.

In my ideal world, you could access mailboxes via web services. The advantage of this would be that you didn’t have to depend on local IMAP library support, nor would you have to deal with its slightly arcane semantics.

So I pondered this for a while and would occasionally talk to people about it. One day, while working at CommerceNet, I mentioned it to Lisa Dusseault, an IETF Application Area Director and standards architect. She expressed interest in the idea and we ended up scheduling a few hack dates to simultaneously work on a standard spec and implementation for accessing IMAP via HTTP.

The result was this IETF Internet-Draft and a rough implementation of an IMAP to HTTP gateway using this proposed interface. Lisa summarized this project on her blog. It was a very simple, RESTful approach using Atom for mailbox listings.

If this sort of interface were adopted by major email providers, not only would email finally have a nice API, but it would become better integrated with our web ecosystem. For example, email messages would have URLs, and mailboxes would have Atom feeds. Sure Gmail might have that sort of thing, but as a standard it has the potential to be an implementation/provider-agnostic interface you can generally expect to have available.

Here’s a result of the prototype I built with her. My inbox is pretty much directly represented just fine in a feed reader as it would be in an email client.

httpmail2

One of the nice things about Mailhook that none of the successors ended up doing was parsing the email message for you into a user-friendly data structure of standard HTTP parameters. This way you wouldn’t have to parse a MIME document. Because smtp2web was designed for App Engine and Python, they expected you could parse it yourself quite easily. My approach let you skip this and not become dependent on a local library. I also translated the semantics of file attachments to file uploads, which let you take advantage of the existing support there.

The last bit of thinking I did on this HTTPMail project was along those lines of convenience parsing. Perhaps as an optional interface, you could get URLs not only to different message representations, but to the insides of messages, including MIME parts and attachments. For example:

http://mail.provider.com/progrium/inbox/message-id
would get the full rfc822 representation of the message
http://mail.provider.com/progrium/inbox/message-id.txt
would get the text/plain MIME part
http://mail.provider.com/progrium/inbox/message-id.html
would get the text/html MIME part
http://mail.provider.com/progrium/inbox/message-id/myfile.zip
would get the myfile.zip attachment
http://mail.provider.com/progrium/inbox/message-id/headers.json
might get a JSON representation of all the headers

Anyway, I just want to be clear the point of this is that if you have this kind of interface… or even a gateway service like the one I implemented, it makes reading mailboxes from web scripts that much easier and accessible. And like Timothy Fitz recently suggested, this would make email a team player in our web ecosystem.