Don’t be so PuSHy

You know three things that would be cool?

  • the ability to subscribe to the output of a Flickr API call in a feed aggregator
  • the ability to get the results of Flickr API calls as…

Oh wait. That was a while ago. Wouldn’t it be great if you didn’t have to poll our API over and over just to see if photos were there, only to find out you waited too long since last time and now because of the results size limits you can’t get them all and you have to figure out how many you missed and then you have to make another call with the right offset to get those results but of course in between then and now the result set changed a bit so you aren’t sure if you really got them all and…

Wouldn’t it be great if Flickr had something that could just PuSH photos to you as they appeared, kind of like this?

Introducing the new (and experimental) flickr.push API methods. These allow you to subscribe to new uploads and updates from your contacts and favorites from your contacts. Let’s dive right in and see exactly how it all works:

"there is a virtuous circle in this ecosystem"

The 20,000 ft overview is basically this:

  1. You make an API call to Flickr asking to subscribe to one of several different photo feeds, providing a callback URL in the arguments.
  2. A little verification dance ensues during which we make a request to your callback URL. If you respond appropriately we’re all good and from then on…
  3. Live(-ish) updates are POSTed from Flickr to your callback URL in Atom 1.0 format.

Subscribing

The subscription system is based as closely as possible on Google’s Pubsubhubbub protocol, with a few wrinkles. One is that Flickr acts as the hub and the publisher all rolled in to one. We’re obviously not really “publishing” separate feeds of every single user’s contacts’ photos and faves to a central hub somewhere, we only create the feeds on demand when someone subscribes to them. So we couldn’t, for example, publish them all to a 3rd-party hub like Superfeedr. But the whole pubsubhubbub metaphor still works pretty well.

Another difference is that the subscription happens via an authenticated API call and not an HTTP POST; hopefully the reasons for this are obvious. We’ll get into them in detail a little bit later. But even though the mechanism for the subscription request is different we’ve tried to follow the protocol as closely as possible and keep the parameters the same. The Google PubSubHubbub Core 0.3 section on how the subscription flow works is a good place to start, and the rest of this post assumes you’ve read that and more or less understand what the interactions should be between hub and subscriber. Done? OK, here are the methods:

flickr.push.getTopics

This method just tells you what you can subscribe to. It returns something like this:

<rsp stat="ok">
  <topics>
    <topic name="contacts_photos" />
    <topic name="contacts_faves" />
  </topics>
</rsp>

yeah, yeah, you already get that part. You can currently subscribe to contacts’ photos (new uploads and updates) or contacts’ faves. So subscribe already!

flickr.push.subscribe

This method (which requires an authentication token with read permissions) takes almost exactly the same arguments as a “proper” PubSubHubbub subscribe HTTP request would. Wee differences:

topic – unlike the topic argument in the HTTP version (which is a URL), this is just one of the topic types returned by flickr.push.getTopics.

secret – currently not supported, so this parameter is omitted.

callback – this must be unique, i.e. you can’t use the same URL for more than one subscription.

Everything else works as you would expect – verification (either synchronous or asynchronous), the hub challenge string, subscription expiration/refreshing, unsubscribing (with flickr.push.unsubscribe) etc. Which brings us to:

flickr.push.getSubscriptions

This method also requires an authentication token with read permissions, and returns a list of subscriptions for the authenticated user, like so:

<rsp stat="ok">
  <subscriptions>
    <subscription topic="contacts_photos" callback="http://example.com/contacts_photos_endpoint?user=12345" pending="0" date_create="1309293755" lease_seconds="0" expiry="1309380155" verify_attempts="0" />
    <subscription topic="contacts_faves" callback="http://example.com/contacts_faves_endpoint?user=12345" pending="0" date_create="1309293785" lease_seconds="0" expiry="1309380185" verify_attempts="0" />
  </subscriptions>
</rsp>

Oh yeah, the docs:

flickr.push.subscribe
flickr.push.unsubscribe
flickr.push.getTopics
flickr.push.getSubscriptions

Feeds

The format of the feed that gets posted to your endpoint is currently limited to Atom 1.0, i.e. exactly what you’d get from something like

http://api.flickr.com/services/feeds/photos_public.gne?id=_YOUR_NSID_HERE_&format=atom

For the contacts_faves topic type it’s the same thing but with the addition of the atom:contributor element to indicate the user who faved the photo.

And that’s about it. Questions?

Privacy and Restrictions

The astute observer may notice that not all photos are being sent in the PuSH feeds. Since this is a new (and experimental) feature for Flickr, we’ve basically turned all of the privacy/safety restrictions on it up to 11, at least to start with. PuSH feeds currently only contain images that have public visibility and safe adultness level. In addition, users with their “Who can access your original image files” option set to anything other than “anyone” and users who are opted out of the API will not have their photos included in PuSH feeds.

While this may be a bit restrictive (for example since the API call is authenticated and the photos are coming from your contacts technically you should be allowed to see contacts only or friends/family photos for contacts that allow it),  we feel that since this is a new thing it’s better to start conservative and see how the feature is being used. It’s possible that we may relax some of these restrictions in the future, but for now a PuSH feed is essentially what a signed-out user could get just by grabbing the RSS feeds from various people’s photostreams.

You will also notice that for now we’ve limited the feature to pro account holders only.

So… What?

screens beget BACON!!!

So what can you do with it? There’s the obvious: any web application which currently does some kind of polling of the Flickr API to get photos for its users can potentially be altered to receive the push feeds instead. More timely updates, cheaper/simpler for the application and as it turns out cheaper for Flickr, too – it’s often easier on our servers to push out events shortly after they happen and we’ve got them (often fresh in our cache) than it is to go and dig them up when they’re asked for some time later.

Surprise!

Some of the more interesting things that we hope these API methods will enable revolve around the more real-time nature of the events they expose. As an example of what’s possible in this space, Aaron Cope has created a little application he calls “Pua”. It’s a wonderfully simple way to surf Flickr without having to do much of anything; Pua takes you on a ride through your contacts’ photos and favorites, as they happen. Have a read about exactly what it is, why it’s called Pua and why he made it. If you ask nicely maybe Pua will give you an invite code.

Later

Hopefully there will be much more to come. Finer-grained controls on the subscriptions (safety levels, visibility levels, restricting to just new uploads or only certain types of updates, lightweight JSON feeds, etc.), new types of subscriptions (photos of your friends/family, photos from a particular location, photos having a particular tag, something to do with galleries…), and maybe some other stuff we haven’t thought of yet. Hey, wouldn’t it be cool if you didn’t need to run a web server on the other end to be the endpoint of the feeds?

Let us know what you’d like to see! What works, what doesn’t, what we got wrong and how to make it more useful to the people who want to Build Stuff (that’s you).

Fine Print

"there is a virtuous circle in this ecosystem"

The Flickr PuSH feeds are part of the Flickr API, and thus fall under the API Terms of Service Agreement. This means all the usual things about respecting photo owners’ copyrights and all also the other good bits about API abuse. In other words, don’t try to subscribe to all of Flickr. Trust me, we’ll notice.

Flickr now Supports OAuth 1.0a

We’re happy to announce that Flickr now supports OAuth! This is an open standard for authentication, which is now fully supported by the Flickr API. You can get started by going to our OAuth documentation. As part of this announcement, we would also like to note that the old Flickr authentication is now deprecated, and is expected to be disabled early 2012.

I'm Guarding the Door

I’m Guarding the Door by Frenck’s Photography

OAuth is very similar to the old Flickr auth in a lot of ways. You start by getting a request token (frob in the old flow), redirecting the user to the authentication page, and then getting a token which can be used to make authenticated requests. With proper OAuth support, though, you will be able to use one of the many libraries available in a variety of languages to get started.

In addition to this, we have streamlined the authentication process across desktop, mobile and web, and have simplified the user experience by removing the anti-phishing step for the Desktop flow, which is no longer necessary.

Currently, we only support OAuth 1.0a, but we have plans to eventually support OAuth 2.0. The decision was based on the fact that OAuth 2.0 is still an evolving definition that is rapidly changing.

We wanted to make the transition to OAuth seamless to the user, so we created a method to exchange an old token, with an OAuth token. The application has to simply make an authenticated request to flickr.auth.oauth.getAccessToken, which returns an OAuth auth token and signature for that user which are tied to your application. The exchange is meant to be final, so the old authentication token is scheduled to expire 24 hours after this API method is called.

Now, it’s your turn! Go read our OAuth documentation if you already have an application, or visit our developer guide for more information on how to get started. If you experience any problems, or have any questions or suggestions regarding our OAuth implementation, please post to our developer mailing list.

Refreshing The API Explorer

Most people know that Flickr has an API. As it wouldn’t be much use without documentation, we have that, too. (There’s even a list of methods and information about each available via the API itself.) What if I told you there was also a way to experiment with it from the comfort of your browser, no coding required?

Sink Explorer

Sink Explorer by Zabowski

Well, that’s what Flickr’s API Explorer offers. It’s an easy way to customise requests by filling in simple form fields, whether the method requires authentication or not, and to see the responses that are returned. It’s great for one-off prototype scripts where you quickly want to find some data, for seeing whether a method does what you think it does, or to sanity-check some code that’s not doing the right thing.

It’s been around for years, but nobody ever seems to have made much of a fuss about it. (The only mention I can find on this blog is an interview with a certain API developer singing its praises.) However, it’s needed a little attention to bring it up to date, and so I made some time to teach it a few new tricks.

Firstly, it now offers a choice of output response. While it doesn’t offer every format that the API does, the three (and a bit) available – the default XML, JSONP (or raw JSON), and PHP serialized data – should cover a lot of ground. Secondly, the Explorer pages now have proper URLs, so it’s possible to link to the API method for fetching the list of pandas, for example. Finally, for the most popular of those response types – XML and JSON(P) – responses are now pretty-printed and syntax highlighted, as are the examples in the API documentation pages for each method. That is to say, the returned values are indented and have line breaks, while the name, attributes and quoted values of the elements are coloured appropriately.

Now that you know that it exists, and that it’s all freshened up with spiffy features, why not go and play around? Have fun!