Skip navigation

Now that I’ve told you how cool AppJet is for hook scripts, allow me to obsolete it. This morning (as I try to avoid falling asleep as I type) I’m announcing Scriptlets, a tool for throwing online quick web scripts in any of several languages. It’s perfect for writing hook scripts. In fact, it’s what I’ve been dreaming of for web hooks for several years.

When I started thinking about web hooks I figured, as the early adopters, most advanced users would have access to some kind of cheap PHP web hosting for hook scripts. Only later did these cloud code runners appear like App Engine and AppJet. AppJet in particular is the closest to my ideal, as I mentioned before. But it was limited to a particular language and still felt slightly too heavyweight for simple hook scripts (by the name alone it was obviously intended for apps, not scripts).

The ideal I envisioned was a site like a paste bin (I’m keen on Pastie). You know, where you paste some code, pick a language, and you’d get a nicely syntax highlighted, numbered row display of the code with a simple, disposable URL you could use to show the kids in IRC your code that wasn’t working. I wanted that, but that also gave you a URL to run the code. This way I could use it for one-off web scripts that accepted request parameters and everything. No more random one-off PHP scripts on my server. And of course, it would be perfect for hook scripts.

Actually, when I started talking with friends about this, none of this cool cloud computing stuff was big yet. App Engine and AppJet weren’t even around. The closest thing was Ning, which used to host PHP code for you for free. One of my friends (actually, the one that coined “scriptlets”) was interning at Google and knew, but couldn’t tell me that App Engine was coming. Once it came out, he and I started a project like Scriptlets for just Python… but we ran into a wall when I misread the Python documentation and thought App Engine wasn’t letting us eval anything more than one line long. It turns out this was completely wrong, but we’re both pretty busy and we moved on to other things.

But when Google announced Java support on App Engine, even though I hate Java, I knew that the JVM meant they now supported a bunch of languages. I revisited our code, solved the stupid problem, and got several other languages working. And tonight I launched it… with support for 3 languages: JavaScript (uh oh, AppJet!), Python (uh oh, App Engine!), and PHP (uh oh.. uhh… commodity PHP hosting?). Ruby is on its way, and potentially others.

All Scriptlets languages provide an environment with access to the web request and response, and most of the features of that language. All the constraints of the App Engine environment are inherited, meaning no long running processes, no abitrary file system access and no socket connections. (I’m so glad I didn’t have to solve the sandboxing problem by myself!) But you do get to make web requests, and this is exposed with a custom function in JavaScript and PHP called fetch(). It’s a pretty limited environment and not even intended for state maintaining scripts (yet), but it should still prove useful.

Anyway, think of Scriptlets as another community service for the web hooks ecosystem. I have a bunch of these in mind that I’ll be building out. Like PostBin, Scriptlets is open source on GitHub and waiting for you to submit patches. Enjoy!

I have a quick story to share that shows how cool it is to have web hooks. I was writing another post and ended up going on a tangent with this, so I figured I’d give it its own post.

At the last SuperHappyDevHouse, I presented Get Achievements, an XBox Live-style achievement platform I built one night. The only thing was, it had no pretty aesthetic to it, so I rushed what you see there now during DevHouse so I could present it at our lightning talks. I was hacking with a designer friend up until somebody came upstairs to tell me I was up soon, but just then I was struck with a great idea: what if I could add achievements to the DevHouse wiki? Get Achievements requires you to make backend changes to your code, and PBwiki is a hosted service. However, PBwiki recently added web hooks, so theoretically I could use them to trigger the achievements. Could I rig it up in the time I had left?

What I needed to do was have an event in PBwiki POST to a URL on Get Achievements that unlocks an achievement. I needed something simple and quick, so I figured I’d make a login achievement. With this, if I login to the wiki, I’d be rewarded with fanfare and a box saying “Login Achievement Unlocked” as you can do with Get Achievements. But how can I make PBwiki POST to Get Achievements when you login? Well, I can’t, it’s a hosted service. But like I said, they had web hooks, and they included logging in as a hookable event.

At this point I was downstairs less than 10 minutes from my 5 minute lightning talk. I let some people go ahead as I rushed through this sitting on the floor with my laptop. I needed a place to write a hook script for PBwiki that would POST to Get Achievements. Very simple. AppJet was the first place that came to mind, and because I didn’t have to set up an application config file or upload anything with an SDK, I was able to add achievements to the SuperHappyDevHouse wiki in less than 10 minutes… thanks to hooks and AppJet.

I got up there and showed them how cool Get Achievements was, and then I dazzled them with a demo of the DevHouse wiki with a login achievement. Sure, it was more a demo of web hooks than Get Achievements at that point, and because not everybody knows what the hell I’m talking about most of the time, I didn’t get the standing ovation I was looking for, but it went well. I had to go around after, asking my friends if they really got how awesome what I just did was.

From idea to a (mostly) working demo for something that would normally involve making backend code changes, I did as a user, in a panic, terribly fast. Just an example of what’s possible in a world with web hooks…

Just because I think web hooks will change the world doesn’t mean I have to take it seriously. I sort of default to a serious tone in my writing. Hopefully this graphic will lighten things up a bit. It was thrown together by a friend after I came up with the tag line he used in it.

I’ve heard several reports of people getting excited about web hooks from my talks or posts here, and then when they find out what’s actually going on they somehow feel cheated. “As if something that simple could do all that!” I don’t know. Perhaps that’s what happened with this guy. Anyway, I’m considering using it as the unofficial web hooks tag line.

Webhooks: So simple youll think its stupid

I’m putting together a new version of my presentation on Web Hooks and I’d like to get feedback on my previous versions. I’m referring to the talks I have slides for on Slideshare. If you’ve seen any of them, at what point did it all click for you? What made most sense and what made the least sense? What was funniest? What did you think it could do without? I’m trying to see what I should elaborate on or just drop for my new deck.

Also, what do you think I skimped on? What do you want to hear more about? If you have any other kind of feedback, or even suggestions based on content on this blog, let me know in the comments. Thanks!

MailChimp, an email marketing tool used by the likes of Firefox and Media Temple, recently announced the addition of web hooks to their API. Their primary use-case is to keep your email list in sync with theirs. They push to your callback URL with several types of events regarding list changes, including subscribes, unsubscribes, profile updates, cleaned addresses, and email changes.

Their API documention explains web hooks and how you use them with MailChimp. They also mention PostBin as an excellent utility to help see the event data as they’re triggered in their system.

One feature of web hooks is that they’re somewhat self documenting. With web APIs you need to know what you’re requesting and how to request it, but with web hooks, you’re just handed the data. Some of it might need explanation, but you can get started pretty quickly by just capturing actual data and seeing what it gives you. When I first started playing with web hooks, I wrote a handler script that would email me the data. This worked okay, but it could easily be better.

So today I’ve released PostBin, a tool for capturing and inspecting data from web hooks. It’s sort like of a cross between pastebin and Mailinator, but for HTTP POST requests. It’s very straightforward and simple. No signups, no configuration. Just click a button, get a bin with a unique URL, capture POST requests with that URL, then view those requests and the data in them at that URL.

The source is available and patches are welcome. If you have useful source changes to push back to me, they’ll likely be deployed to the production app running on Google App Engine. If you just have suggestions, you can leave them on our UserVoice page.

Hope you guys find this useful!

I like to claim you can implement web hooks with one line of code. The reality is you need to do more than make inline HTTP requests. You need to manage user callback URLs, you need to make the requests asynchronous, you probably want a retry scheme, you might want authentication or verification, and in some cases a response handling mechanism. To do these things requires quite a bit more work, and those that have done it so far haven’t yet disclosed the details of how they’ve implemented this. I hope to change that, and to speed things up, I’ve started a simple community project I’m announcing today called Hookah.

The goal of Hookah is to take most of the work out of implementing web hooks. Ideally, Hookah will allow you to have all the features you need for successfully deploying web hooks and only need to add a few lines of code to your app. In fact, one line for each event! On top of that, the interface for Hookah is HTTP, so it’s not that different than directly invoking web hooks and works easily from any language.

The project is on GitHub and, as you can see, it’s very simple. Currently what we have in the master branch does asynchronous event dispatching with retries. This should be enough to get anybody started with web hooks. It’s written in Python for simplicity and readability, but Python is pretty ubiquitous and because it’s an HTTP server it doesn’t require you to know Python to use it. Again, it’s very simple and very early, but I hope it can be the start of a collective effort to make web hooks easier to implement.

For now we can bootstrap off the Web Hooks discussion group to talk about it, so if you have anything to say or ask, join up. There is a rough roadmap in the README and if you’re interested in contributing, let us know.

Today I did a 7 question interview for Jason Salas on web hooks. Pretty standard stuff, but it allowed me to bring up some things I haven’t written about yet. His final question was about barriers going forward with the movement, which I also answered with a general “what’s coming” sort of response:

It seemed like the biggest hurdle originally was getting people to wrap their heads around this idea. I would always talk about it in the abstract and go on about all the implications of what was essentially one line of code. I think there are enough fairly well-known examples now that it’s easier for people to join the party. Even then, the general perception of what’s possible is going to be limited by the examples.

Like AJAX, you can’t just build a popular example of AJAX without it being a useful tool itself. I can build all the web hook prototypes I want, but it’s not until the Googles and Facebooks implement them in a useful way that people will really see the value. Until then, we get incremental boosts by the smaller companies like Gnip, GitHub, and others. I’ve started working or talking with these guys to get them involved in a collective conversation around web hooks, so we can work out issues standing in front of adoption.

The issues people come up with are usually security and scalability related. As it turns out, some of these issues have been solved by these guys already doing it. So I’m trying to get more of them to share best practices and publicize their use of web hooks. This way people can start seeing the different ways they can be used. For example, the Facebook Platform, although pretty complicated and full of their own technology, is still at the core based on web hooks. They call out to a user-defined external web application and integrate that with their application. That’s quite a radically different use of web hooks compared to the way people think of them in relation to XMPP.

Moving forward, I think we’re going to see more libraries and tools that have solutions to scalability and security built-in. I’ve started one project called Hookah that I’m hoping to get released soon. It provides basic callback request queuing and management of callback URLs so you really can implement web hooks with a single line of code for each event. We’re also starting to see similar helper libraries for frameworks like Django and Rails.

Eventually we’ll be seeing specs for doing specific things on top of web hooks. One of the first things on my list of standards to look into is the way in which you register and manage callbacks in a programmatic way. Many web hook providers use a web interface to manage your callback URLs. We’ll see some neat things happen when you can manage them via APIs so that tools can set callbacks with services on your behalf.

Anyway, one of the reasons I’m so attached to the idea of web hooks is that I see a lot of long-term potential. Especially when you integrate them into other visions of the future, like the Web of Things. When you combine the Programmable Web with the Web of Things, you get a world of Programmable Things.

That’s where I’d like to see this end up.

A few days ago at BarCamp Miami, software architect Ryan Teixeira gave a talk about web hooks loosely based on my Programmable World of Tomorrow talk. It would have been nice to see, but I guess the next best thing is to see the slides on SlideShare. Take a look:

 

 

Nice job, Ryan!

Okay, it’s not really a dispute. That was sarcasm in the title. Just to be clear, since there’s always been a tiny bit of useful friction between the idea of XMPP and web hooks, it’s important to remember they are not mutually exclusive. Anybody hyping a battle between the two is trying to create or is imagining a controversy that isn’t real. Many proponents of web hooks are XMPP proponents as well. In fact, the video some are pushing around to promote this supposed throw down is of two XMPP supporters (including djabberd author Brad Fitzpatrick) demoing a pubsub system based on web hook callbacks. And it takes place at an XMPP meetup, so of course there was going to be some proselytizing.

Nevertheless, Jabber/XMPP is a messaging protocol. Web hooks provide a model for functional extensibility, so they are a platform for many different things. A push-based pubsub messaging system is just one use. Even though I originally wrote about web hooks as a notification mechanism, my mind was nowhere near pubsub. I was more focused on the idea of web service integration and orchestration. With the commoditization of CGI-enabled web hosting, I was thinking about how the popularity of web programming combined with the easy invocation of HTTP requests could be used to make a more useful and functionally extensible web. A more programmable web.

With that said, there is nothing wrong with discussing what you can and can’t do with web hooks and XMPP, but there is not some angry fight between camps. Done. Over. Moving on.

As it turns out, the project demoed in that video, which is called pubsubhubbub, is something I had previously stumbled upon while browsing projects a friend of mine was involved in. There wasn’t much of a description, and I didn’t dive too deep into the code, so I bookmarked it to come back to it. After that demo happened, it seemed I misheard (from several people) that it was some XMPP pubsub system, which totally confused me because… it’s not. It’s a neat, distributed pubsub implementation built on web hook callbacks created by Brad Fitzpatrick and Brett Slatkin (a Google App Engine developer). Hopefully they’ll put some more documentation up as it develops, but it’s just really neat to see some XMPP folks build an open pubsub system with web hooks. Cheers to them!