Skip navigation

Email is one of our oldest and most used Internet systems. Frankly, though, I hate its dated implementation. Yet I understand how tested and universal it is, and as such, think it would be a good idea to allow it as a means of interaction with web services. I just don’t want to have to touch its crappy internals. This led me to develop the now defunct Mailhook (succeeded by smtp2web and Astrotrain) as a way to leverage web hooks to help ease the pain of accepting email in our web applications. It was a simple SMTP server that would pass parsed email to a URL you specify as if it were a form post.

What Mailhook did not do was provide programmatic access to existing mailboxes. That’s okay, though. We’ve got IMAP with libraries in most languages, so that’s pretty taken care of, right?

Meh.

In my ideal world, you could access mailboxes via web services. The advantage of this would be that you didn’t have to depend on local IMAP library support, nor would you have to deal with its slightly arcane semantics.

So I pondered this for a while and would occasionally talk to people about it. One day, while working at CommerceNet, I mentioned it to Lisa Dusseault, an IETF Application Area Director and standards architect. She expressed interest in the idea and we ended up scheduling a few hack dates to simultaneously work on a standard spec and implementation for accessing IMAP via HTTP.

The result was this IETF Internet-Draft and a rough implementation of an IMAP to HTTP gateway using this proposed interface. Lisa summarized this project on her blog. It was a very simple, RESTful approach using Atom for mailbox listings.

If this sort of interface were adopted by major email providers, not only would email finally have a nice API, but it would become better integrated with our web ecosystem. For example, email messages would have URLs, and mailboxes would have Atom feeds. Sure Gmail might have that sort of thing, but as a standard it has the potential to be an implementation/provider-agnostic interface you can generally expect to have available.

Here’s a result of the prototype I built with her. My inbox is pretty much directly represented just fine in a feed reader as it would be in an email client.

httpmail2

One of the nice things about Mailhook that none of the successors ended up doing was parsing the email message for you into a user-friendly data structure of standard HTTP parameters. This way you wouldn’t have to parse a MIME document. Because smtp2web was designed for App Engine and Python, they expected you could parse it yourself quite easily. My approach let you skip this and not become dependent on a local library. I also translated the semantics of file attachments to file uploads, which let you take advantage of the existing support there.

The last bit of thinking I did on this HTTPMail project was along those lines of convenience parsing. Perhaps as an optional interface, you could get URLs not only to different message representations, but to the insides of messages, including MIME parts and attachments. For example:

http://mail.provider.com/progrium/inbox/message-id
would get the full rfc822 representation of the message
http://mail.provider.com/progrium/inbox/message-id.txt
would get the text/plain MIME part
http://mail.provider.com/progrium/inbox/message-id.html
would get the text/html MIME part
http://mail.provider.com/progrium/inbox/message-id/myfile.zip
would get the myfile.zip attachment
http://mail.provider.com/progrium/inbox/message-id/headers.json
might get a JSON representation of all the headers

Anyway, I just want to be clear the point of this is that if you have this kind of interface… or even a gateway service like the one I implemented, it makes reading mailboxes from web scripts that much easier and accessible. And like Timothy Fitz recently suggested, this would make email a team player in our web ecosystem.

14 Comments

  1. Hey David, over the last year we’ve been building something like this for both email and files. The solution is basically a file crawler plus a local web server. The goal is to provide an easy way to share the content on a workstation or laptop with friends and coworkers without server infrastructure.

    It’s good to see others are thinking along the similar lines.

    • Dean: I might have missed where David came from. Hi, I’m Jeff. :)

      I should point out this was not specifically to solve a problem like “sharing content” or even anything terribly end-user focused, but to build developer infrastructure. The goal here is to ease the friction of working with different protocols relative to the web. I’m not so crazy as to say everything should be HTTP, but I wouldn’t mind if everything was, or at least had an HTTP interface or gateway.

      Julien: Yes, I like the idea of those, too. I just love adapters. But in the context of the programmable web, HTTP is the preferred protocol. Since Gnip dropped XMPP, there’s still room for XMPP to HTTP or HTTP to XMPP adapters. There is the issue of operational sustainability, though…

  2. This reminds me greatly of Email to XMPP gateways…

  3. Hi Jeff. I’m trying to build a new email service provider that would bring somethong like your httpmail. I don’t want to talk about implementation or languages but discuss about concept should be great. Concerning Julien comment on XMPP->HTTP gateway i think it’s not the same goal. XMPP is a protocol just like HTTP and XMPP client would only parse XMPP well-formated message. So these kind of gateways are great only for interoperating operations (like share contents) or others. Personaly i think httpmail like a layer on top of smtp or/and imap with ability to RESTify the underlying protocol and give webservice a api-free mail protocol (yeah it’s my name of this protocol :)). if we share the same ideas i would be happy to open discussion with you.

  4. I’ve been developing something similar for a intranet portal.

    To make development easier, I created RESTful web services for IM (XMPP), email (Exchange, IMAP, POP, SMTP), files (MS/Samba/CIFS), and databases (MSSQL).

    Then, all I had to do was authenticate a user and impersonate them for all those services. All these services used their same Active Directory domain credentials.

    People used it mainly if they couldn’t VPN/RDP to their machine. E.g., mobile access, other person’s computer, et al.

    Some people even started using feed readers to browse stuff directly.

    I chose to use SSL and do HTTP Basic authentication on the portal application as well as the web services. It made everything easier (no session management).

    • Is this open source? Could it be? ;)

        • Alex
        • Posted July 2, 2009 at 4:26 am
        • Permalink

        Open source? It’s public domain. There’s nothing secret here.

        The api’s weren’t that hard. Once everything is on paper (Visio), it all falls into place.

        The hardest part was going to be managing state/session, so I skipped it and went straight HTTP Basic. This let’s the browser handle session/state.

        I built it on Java/ColdFusion. (Barf! An organization policy.)

        And like I said, it’s a pretty standard Active Directory shop. So authN, authZ was already taken care of.

        Like someone said, XML/JSON is direct relation, RSS is custom translation. I made XML first, then added JSON and RSS.

    • I know there’s no secret, but there’s still work in designing a proper REST API for these other services. If you’re saying your implementation is public domain, let’s see a link!

  5. Okay.
    I’ll put something together.
    Names changed (to protect the innocent), of course.

  6. > I also translated the semantics of file attachments to file uploads, which let you take advantage of the existing support there.

    BTW. My implementation creates email in just one request. Even with attachments.

    To accomplish this from JavaScript on the client side was roundabout. JavaScript cannot access file uploads directly, only submit their parent form to the server. The attachments are uploaded via an IFrame, like Gmail, and their binary data returned to the client/JavaScript. This is a separate passthrough web service, leaving no trace on the server. That was the real sleight of hand.

    The message is sent as a custom web service request containing the metadata (from, to, subject) and the message (text, html, attachments).

    Originally I built it to send inline base64 encode binary (like real email), but then later changed it to allow either inline or multipart.

    This mimics the email read request. Can either opt for inline binary or link uri to binary.

  7. Okay.
    Here’s a cleaned diagram of the email svc.
    I’ll post more later.

    email
    – svr
    – mbx
    – fol
    – msg

    svr
    – get entries: GET /api/email
    – get mbxs: GET /api/email & query
    – search: GET /api/email & query

    mbx
    – get entries: GET /api/email/mbx0
    – get subfols: GET /api/email/mbx0 & query
    – search: GET /api/email/mbx0 & query

    fol
    – create a fol: POST /api/email/mbx0 & XML
    – get a fol: GET /api/email/mbx0/fol0
    – get a fol’s entries: GET /api/email/mbx0/fol0 & query
    – get a fol’s subfols: GET /api/email/mbx0/fol0 & query
    – get a fol’s msgs: GET /api/email/mbx0/fol0 & query
    – search in a fol: GET /api/email/mbx0/fol0 & query
    – rename a fol: PUT /api/email/mbx0/fol0 & XML
    – delete a fol: DELETE /api/email/mbx0/fol0
    – delete a fol’s entries: DELETE /api/email/mbx0/fol0 & query
    – delete a fol’s subfols: DELETE /api/email/mbx0/fol0 & query
    – delete a fol’s msgs: DELETE /api/email/mbx0/fol0 & query

    msg
    – send a msg: POST /api/email & XML
    – get a msg: GET /api/email/mbx0/fol0/msg0
    – get a msg’s atts: GET /api/email/mbx0/fol0/msg0 & query
    – get a msg’s att: GET /api/email/mbx0/fol0/msg0 & query
    – edit a msg: PUT /api/email/mbx0/fol0/msg0 & XML
    – move a msg: (2 requests)
    1. POST /api/email/mbx0/fol0/msg1 & XML
    2. DELETE /api/email/mbx0/fol0/msg0
    – copy a msg: POST /api/email/mbx0/fol0/msg1 & XML
    – delete a msg: DELETE /api/email/mbx0/fol0/msg0

    • Cool stuff. You should put it up on GitHub. :)

        • Alex
        • Posted July 9, 2009 at 8:50 am
        • Permalink

        Email could be extended to include multiple servers.

        Obviously, a files api looks just like email, with a few changes.

        files
        – main
        – svr
        – fol
        – fil

        files
        – create a fil: POST /api/files/svr0/fol0 & XML
        – get a fil’s binary: GET /api/files/svr0/fol0/fil0 & query

        If you wanna see the full thing.

        main
        – get entries: GET /api/files
        – get svrs: GET /api/files & query
        – search: GET /api/files & query

        svr
        – get a svr’s entries: GET /api/files/svr0
        – get a svr’s subfols: GET /api/files/svr0 & query
        – search: GET /api/files/svr0 & query

        fol
        – create a fol: POST /api/files/svr0 & XML
        – get a fol: GET /api/files/svr0/fol0
        – get a fol’s entries: GET /api/files/svr0/fol0 & query
        – get a fol’s subfols: GET /api/files/svr0/fol0 & query
        – get a fol’s fils: GET /api/files/svr0/fol0 & query
        – search in a fol: GET /api/files/svr0/fol0 & query
        – rename a fol: PUT /api/files/svr0/fol0 & XML
        – delete a fol: DELETE /api/files/svr0/fol0
        – delete a fol’s entries: DELETE /api/files/svr0/fol0 & query
        – delete a fol’s subfols: DELETE /api/files/svr0/fol0 & query
        – delete a fol’s fils: DELETE /api/files/svr0/fol0 & query

        fil
        – create a fil: POST /api/files/svr0/fol0 & XML
        – get a fil: GET /api/files/svr0/fol0/fil0
        – get a fil’s binary: GET /api/files/svr0/fol0/fil0 & query
        – edit a fil: PUT /api/files/svr0/fol0/fil0 & XML
        – move a fil: (2 requests)
        1. POST /api/files/svr0/fol0/fil1 & XML
        2. DELETE /api/files/svr0/fol0/fil0
        – copy a fil: POST /api/files/svr0/fol0/fil1 & XML
        – delete a fil: DELETE /api/files/svr0/fol0/fil0

  8. You could also spec out more advanced actions, requiring more than one request.

    Like moving folders, et al.

    But it would be just combining existing functionality.


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: