Skip navigation

Tag Archives: cgi

Push/callbacks are better than polling.

This should be an easy argument. It’s not just more efficient, it’s more real-time. I once heard something like 40% of requests returned 304 Not Modified. I imagine similar numbers for other popular sites. The more real-time you try to be with polling, the worse it gets. Don’t call us, we’ll call you. It’s more efficient.

RPC solutions provide more value than messaging solutions.

Although you can argue that RPC can be powered by messaging, and messaging can be used for RPC, the point is that they are different mindsets. RPC is about triggering code that does something. Messaging is about putting a piece of data in another bucket. When you try to call somebody, you usually don’t want to get their voicemail, right? This analogy goes even further in that, like voicemail, it must be checked on the other end. Messaging just pushes polling somewhere else.

More importantly, the focus on triggering code is central to this value proposition. Generative systems are more valuable than sterile systems. When did the web get interesting? When it became about more than just static content, and code was put in the loop to generate dynamic content. The point is that if you prioritize code to receive a message before humans, you open up many more possibilities.

RPC is about making things happen. Messaging stops short short of that by just moving data around.

HTTP is the defacto RPC protocol.

HTTP is everywhere. There are powerful free servers, clients in every major programming environment, and people know it well. It’s proven and it just works. HTTTP’s simple design also allows it to be extremely versatile. It’s basically the TCP of the application layer.

However, the best thing is that HTTP is RPC. This subtle fact has been true ever since CGI was introduced. We’ve gone through building RPC on top of it with XML-RPC and SOAP, but wisely settled on a form of RPC that’s just HTTP and is even aligned with HTTP semantics: REST.

If HTTP is RPC and HTTP is everywhere, it is our defacto RPC protocol. Especially for web applications that breath HTTP, it almost doesn’t make sense to think of any kind of inter-application communication that isn’t HTTP. Turtles all the way down!

HTTP RPC + Indirection = Webhooks

David Wheeler said, “All problems in computer science can be solved by another level of indirection.” Webhooks, and all callbacks, are about taking a procedure call and performing it on a variable function. This is indirection and this is very powerful. This is why Unix pipes work. STDIN and STDOUT are not hardcoded values, they’re variables that you can control.

Now imagine if all the web applications you used had extension points that you could effectively hook together with any other application. Well, that’s what webhooks are about.

When the web was started, it was about these hyperlinked HTML documents… just pages of information. The web as a whole was collectively just about content. Then, thanks to a bunch of hacks that led to CGI, you had this optional tool as the webmaster to help augment static content with dynamic content. Most immediately, this was used for search, a feature that the web did not come with built-in.

Skip ahead about ten years and this concept of running arbitrary code on web requests with CGI was used and understood enough to finally fix this mostly centralized, one-way flow of information. Finally, anybody could easily publish content on the web through blogs, wikis and comments. This caused such a change in the use of the web that we decided to call it Web 2.0 (and haven’t been able to avoid going meta since–sorry). This two-way flow of information turned it into a collective conversation that the marketing folks today call “social media.”

As a byproduct of using CGI more and more, the web also became generally more dynamic. For example, the commercialization of the web happened once we figured out how to securely use this CGI business to do payment processing. The first killer “application” on the web was the shopping cart. The web slowly started to provide functionality along with content. Today in the industry we talk less about web pages (providing information) and more about web applications (providing functionality).

Our web today is not just a social media platform, but an application platform. And applications do things. Applications represent the augmentation and automation power of computing. I know we’re social beings and communication is our primary means of interaction, but forgive me if I think the empowerment of computational utility is cooler than social media. I’d rather use computers to solve problems holding humanity back from self-actualization than to merely add more channels to the echo chamber. Ahem.

I think social media is important, but it’s stealing the spotlight from the functional potential of the web. Beyond sharing-information-with-people-we-know. We’re still caught up in content, yet it was functionality (e-commerce or “the shopping cart”) that allowed the web to be commercialized. It seems like we under appreciate this aspect of the web.

So what’s the point? This whole time I’ve been trying to set up for an assertion about the future of the web. Here it is:

The web was originally about content (web pages). Then it got functionality (CGI, early web apps). It used this functionality to fully democratize its content (blogs, wikis, etc). Next it will democratize functionality. We’ll have user-contributed functionality just as we had user-contributed content.

What does it mean to have user-contributed functionality? Kind of what it sounds like. Just like you can “contribute” a photo to Flickr, you’d be able to “contribute” a feature (new functionality) to Gmail.

It’s kind of like open source, although a bit more consumer friendly. Like open source, if you want a program to do something different or work with another program, you can make it do that yourself. You can even share a patch so others can get that same functionality. The difference with web applications is that most will never give you their source. And if they do, it would be a nightmare to try and integrate everybody’s patches with the latest deployment. Open source just doesn’t quite translate to the world of web applications.

So if you can’t have access to the source, how can you contribute functionality? Gee, that ad-blocker you have in your browser is pretty slick. Did they need the browser source to make it? No? What’s that? Yes, a plugin system! How do plugin systems work? Right, they provide hooks for external code to run.

I think you see what I’m getting at. Web hooks open web applications up to functional extension and personalization. The plugin metaphor also holds about the ease of use. Not everybody can write a plugin, but anybody can install a plugin. User-contributed functionality will be just as easy to install (if at all), and even easier to write than most plugins. Plus, not only can it be shared between users, but potentially across web applications because the web is a common protocol.

So is user-contributed functionality just plugins for web applications? Yes! I’ve been saying web hooks will enable push, pipes and plugins for a while… but who knows what I mean by that. It’s taken several years just to get people to understand what web hooks are, hopefully it won’t take as long to convey what role they can play. User-contributed functionality seemed like a pretty good way to convey their power to customize and extend.

Anyway, there you have it. It’s already starting anyway. What are Facebook Applications but plugins over HTTP, submitted by users? How long before we see Gmail Labs go from just features by internal teams to features by users?

So, I invite you to imagine a world where what you can do with applications on the web is not limited by those that made them.

I’ve got a story for you. And it ends with a fairly unique take on the future of the web. I’m not about to call it Web 3.0 because I detest this long running meme of “let’s define an era before it happens,” but it is the future. And it’s two parts because it’s late and I’m tired.

So one of the key characteristics of Web 2.0 was this idea of “two-way media” or “conversational media” where users could easily publish their own content and comment on the content of others. This gave rise to the phrase “user-contributed content,” where we let the end-user create the content. It took over ten years for this idea to catch on and become a reality. It’s so great, and we’d never go back.

Some of us are privy to the fact that this idea of a two-way, read-write web was the original vision for the web. Tim Berners-Lee’s first web browser included an editor, but publishing didn’t work because the write verbs in HTTP weren’t implemented in early web servers…

Wait, come to think of it, even to this day web servers don’t implement write methods.

Does Apache itself handle PUT or POST requests as intended? No, Apache delegates the proper handling of these to CGI scripts or a module that essentially runs CGI scripts more efficiently. Most web servers still do not implement the write methods of HTTP. They don’t because before we got around to it, something happened.

In fact, I’m not sure many of us realize this fundamental change (other than the original developers of the web) because we’ve all just accepted the way things are as the way things are. Something big happened in 1993 that changed the entire conceptual model of the web.


Web of linked HTML documents

In the beginning, the web was conceptually about serving up these HTML files that would link to each other. If you recall, the path of the URL was just a subset of the server’s filesystem that was mounted to be served by CERN HTTPd or whichever web server. Apparently the killer feature of the web was rendering HTML (so you had inline hyperlinks, among other things), which you had to get somehow, and so GET was all that mattered from HTTP. If you wanted to put something online, you didn’t need a browser, screw HTTP, you just had to FTP a file to the server. Easy enough, right? It was, back then.

Nobody cared to do anything interesting with PUT or POST. However, they were interested in using this fancy new web protocol to access other protocols. So they started hacking the web servers to run scripts that would query WAIS or some other obscure protocol, usually for search because that was a big problem then.

Eventually Rob McCool drafted a spec for CGP or Common Gateway Protocol that would make a standard way for these little scripts to be run by web servers. It was eventually renamed to CGI (since it’s more of an interface than a protocol), implemented in Apache, and that was that. Now you could run scripts on web requests! Most people used CGI for search, which makes a lot of sense, but the rest of the content was still directly served up HTML files because, well, that’s how the web works, yeah?

Slowly, more and more people started doing clever things with CGI. A few people decided to respect the HTTP spec and utilize the proper verbs for write actions, but this CGI thing was kind of a hack, I don’t think ever seriously intended by Tim Berners-Lee, and so people were abusing the semantics, doing destructive things with GETs, ignoring DELETE and PUT. Well, even today people still do this, just less so… anyway…

Eventually our websites got complicated enough and ambitious enough with CGI that almost all requests would go through CGI instead of serving up static HTML documents. And instead of silly filename paths of files that aren’t really on the server, we can make up useful, descriptive paths with the date and title in them.

Today our web is not about serving up files on the server, but generating files to serve. Our web of pages is now generated by “higher-order” web applications that are no longer simple scripts, but complex software.


Apps are "higher-order" nodes

If you ask me, those little hacks got a little out of hand, but I guess it’s for the best. Otherwise, we wouldn’t have Gmail or Amazon or Wikipedia… we wouldn’t have web applications that did useful things, we’d just have a bunch of static HTML documents linked to each other managed by webmasters and central authorities. None of this democratized media business.

Thanks to CGI we got the read-write web, but we also made the web way more useful than it was intended. Suddenly browsing to a URL would run some code. And code… well, code can do anything.

Next we’ll build on this idea and see how web hooks can change the game again!