Don’t be so PuSHy

You know three things that would be cool?

  • the ability to subscribe to the output of a Flickr API call in a feed aggregator
  • the ability to get the results of Flickr API calls as…

Oh wait. That was a while ago. Wouldn’t it be great if you didn’t have to poll our API over and over just to see if photos were there, only to find out you waited too long since last time and now because of the results size limits you can’t get them all and you have to figure out how many you missed and then you have to make another call with the right offset to get those results but of course in between then and now the result set changed a bit so you aren’t sure if you really got them all and…

Wouldn’t it be great if Flickr had something that could just PuSH photos to you as they appeared, kind of like this?

Introducing the new (and experimental) flickr.push API methods. These allow you to subscribe to new uploads and updates from your contacts and favorites from your contacts. Let’s dive right in and see exactly how it all works:

"there is a virtuous circle in this ecosystem"

The 20,000 ft overview is basically this:

  1. You make an API call to Flickr asking to subscribe to one of several different photo feeds, providing a callback URL in the arguments.
  2. A little verification dance ensues during which we make a request to your callback URL. If you respond appropriately we’re all good and from then on…
  3. Live(-ish) updates are POSTed from Flickr to your callback URL in Atom 1.0 format.

Subscribing

The subscription system is based as closely as possible on Google’s Pubsubhubbub protocol, with a few wrinkles. One is that Flickr acts as the hub and the publisher all rolled in to one. We’re obviously not really “publishing” separate feeds of every single user’s contacts’ photos and faves to a central hub somewhere, we only create the feeds on demand when someone subscribes to them. So we couldn’t, for example, publish them all to a 3rd-party hub like Superfeedr. But the whole pubsubhubbub metaphor still works pretty well.

Another difference is that the subscription happens via an authenticated API call and not an HTTP POST; hopefully the reasons for this are obvious. We’ll get into them in detail a little bit later. But even though the mechanism for the subscription request is different we’ve tried to follow the protocol as closely as possible and keep the parameters the same. The Google PubSubHubbub Core 0.3 section on how the subscription flow works is a good place to start, and the rest of this post assumes you’ve read that and more or less understand what the interactions should be between hub and subscriber. Done? OK, here are the methods:

flickr.push.getTopics

This method just tells you what you can subscribe to. It returns something like this:

<rsp stat="ok">
  <topics>
    <topic name="contacts_photos" />
    <topic name="contacts_faves" />
  </topics>
</rsp>

yeah, yeah, you already get that part. You can currently subscribe to contacts’ photos (new uploads and updates) or contacts’ faves. So subscribe already!

flickr.push.subscribe

This method (which requires an authentication token with read permissions) takes almost exactly the same arguments as a “proper” PubSubHubbub subscribe HTTP request would. Wee differences:

topic – unlike the topic argument in the HTTP version (which is a URL), this is just one of the topic types returned by flickr.push.getTopics.

secret – currently not supported, so this parameter is omitted.

callback – this must be unique, i.e. you can’t use the same URL for more than one subscription.

Everything else works as you would expect – verification (either synchronous or asynchronous), the hub challenge string, subscription expiration/refreshing, unsubscribing (with flickr.push.unsubscribe) etc. Which brings us to:

flickr.push.getSubscriptions

This method also requires an authentication token with read permissions, and returns a list of subscriptions for the authenticated user, like so:

<rsp stat="ok">
  <subscriptions>
    <subscription topic="contacts_photos" callback="http://example.com/contacts_photos_endpoint?user=12345" pending="0" date_create="1309293755" lease_seconds="0" expiry="1309380155" verify_attempts="0" />
    <subscription topic="contacts_faves" callback="http://example.com/contacts_faves_endpoint?user=12345" pending="0" date_create="1309293785" lease_seconds="0" expiry="1309380185" verify_attempts="0" />
  </subscriptions>
</rsp>

Oh yeah, the docs:

flickr.push.subscribe
flickr.push.unsubscribe
flickr.push.getTopics
flickr.push.getSubscriptions

Feeds

The format of the feed that gets posted to your endpoint is currently limited to Atom 1.0, i.e. exactly what you’d get from something like

http://api.flickr.com/services/feeds/photos_public.gne?id=_YOUR_NSID_HERE_&format=atom

For the contacts_faves topic type it’s the same thing but with the addition of the atom:contributor element to indicate the user who faved the photo.

And that’s about it. Questions?

Privacy and Restrictions

The astute observer may notice that not all photos are being sent in the PuSH feeds. Since this is a new (and experimental) feature for Flickr, we’ve basically turned all of the privacy/safety restrictions on it up to 11, at least to start with. PuSH feeds currently only contain images that have public visibility and safe adultness level. In addition, users with their “Who can access your original image files” option set to anything other than “anyone” and users who are opted out of the API will not have their photos included in PuSH feeds.

While this may be a bit restrictive (for example since the API call is authenticated and the photos are coming from your contacts technically you should be allowed to see contacts only or friends/family photos for contacts that allow it),  we feel that since this is a new thing it’s better to start conservative and see how the feature is being used. It’s possible that we may relax some of these restrictions in the future, but for now a PuSH feed is essentially what a signed-out user could get just by grabbing the RSS feeds from various people’s photostreams.

You will also notice that for now we’ve limited the feature to pro account holders only.

So… What?

screens beget BACON!!!

So what can you do with it? There’s the obvious: any web application which currently does some kind of polling of the Flickr API to get photos for its users can potentially be altered to receive the push feeds instead. More timely updates, cheaper/simpler for the application and as it turns out cheaper for Flickr, too – it’s often easier on our servers to push out events shortly after they happen and we’ve got them (often fresh in our cache) than it is to go and dig them up when they’re asked for some time later.

Surprise!

Some of the more interesting things that we hope these API methods will enable revolve around the more real-time nature of the events they expose. As an example of what’s possible in this space, Aaron Cope has created a little application he calls “Pua”. It’s a wonderfully simple way to surf Flickr without having to do much of anything; Pua takes you on a ride through your contacts’ photos and favorites, as they happen. Have a read about exactly what it is, why it’s called Pua and why he made it. If you ask nicely maybe Pua will give you an invite code.

Later

Hopefully there will be much more to come. Finer-grained controls on the subscriptions (safety levels, visibility levels, restricting to just new uploads or only certain types of updates, lightweight JSON feeds, etc.), new types of subscriptions (photos of your friends/family, photos from a particular location, photos having a particular tag, something to do with galleries…), and maybe some other stuff we haven’t thought of yet. Hey, wouldn’t it be cool if you didn’t need to run a web server on the other end to be the endpoint of the feeds?

Let us know what you’d like to see! What works, what doesn’t, what we got wrong and how to make it more useful to the people who want to Build Stuff (that’s you).

Fine Print

"there is a virtuous circle in this ecosystem"

The Flickr PuSH feeds are part of the Flickr API, and thus fall under the API Terms of Service Agreement. This means all the usual things about respecting photo owners’ copyrights and all also the other good bits about API abuse. In other words, don’t try to subscribe to all of Flickr. Trust me, we’ll notice.

Flickr now Supports OAuth 1.0a

We’re happy to announce that Flickr now supports OAuth! This is an open standard for authentication, which is now fully supported by the Flickr API. You can get started by going to our OAuth documentation. As part of this announcement, we would also like to note that the old Flickr authentication is now deprecated, and is expected to be disabled early 2012.

I'm Guarding the Door

I’m Guarding the Door by Frenck’s Photography

OAuth is very similar to the old Flickr auth in a lot of ways. You start by getting a request token (frob in the old flow), redirecting the user to the authentication page, and then getting a token which can be used to make authenticated requests. With proper OAuth support, though, you will be able to use one of the many libraries available in a variety of languages to get started.

In addition to this, we have streamlined the authentication process across desktop, mobile and web, and have simplified the user experience by removing the anti-phishing step for the Desktop flow, which is no longer necessary.

Currently, we only support OAuth 1.0a, but we have plans to eventually support OAuth 2.0. The decision was based on the fact that OAuth 2.0 is still an evolving definition that is rapidly changing.

We wanted to make the transition to OAuth seamless to the user, so we created a method to exchange an old token, with an OAuth token. The application has to simply make an authenticated request to flickr.auth.oauth.getAccessToken, which returns an OAuth auth token and signature for that user which are tied to your application. The exchange is meant to be final, so the old authentication token is scheduled to expire 24 hours after this API method is called.

Now, it’s your turn! Go read our OAuth documentation if you already have an application, or visit our developer guide for more information on how to get started. If you experience any problems, or have any questions or suggestions regarding our OAuth implementation, please post to our developer mailing list.

Refreshing The API Explorer

Most people know that Flickr has an API. As it wouldn’t be much use without documentation, we have that, too. (There’s even a list of methods and information about each available via the API itself.) What if I told you there was also a way to experiment with it from the comfort of your browser, no coding required?

Sink Explorer

Sink Explorer by Zabowski

Well, that’s what Flickr’s API Explorer offers. It’s an easy way to customise requests by filling in simple form fields, whether the method requires authentication or not, and to see the responses that are returned. It’s great for one-off prototype scripts where you quickly want to find some data, for seeing whether a method does what you think it does, or to sanity-check some code that’s not doing the right thing.

It’s been around for years, but nobody ever seems to have made much of a fuss about it. (The only mention I can find on this blog is an interview with a certain API developer singing its praises.) However, it’s needed a little attention to bring it up to date, and so I made some time to teach it a few new tricks.

Firstly, it now offers a choice of output response. While it doesn’t offer every format that the API does, the three (and a bit) available – the default XML, JSONP (or raw JSON), and PHP serialized data – should cover a lot of ground. Secondly, the Explorer pages now have proper URLs, so it’s possible to link to the API method for fetching the list of pandas, for example. Finally, for the most popular of those response types – XML and JSON(P) – responses are now pretty-printed and syntax highlighted, as are the examples in the API documentation pages for each method. That is to say, the returned values are indented and have line breaks, while the name, attributes and quoted values of the elements are coloured appropriately.

Now that you know that it exists, and that it’s all freshened up with spiffy features, why not go and play around? Have fun!

Inspiration Tuesday

When I poke my head above my monitor at FlickrHQ, I can see a lot of headphones. Ever wonder what inspires us to write code (besides fried chicken sandwiches and cocktails)? Well I did a little snooping in the playlists of a few Flickreenos:

Zack

Zack in headphones

Top Tracks:

  • Fitz & The Tantrums – MoneyGrabber
  • Ear Infections mix – Goldenchyld
  • Live In San Diego – Goldenchyld
  • Little Birdy – Brother
  • Parliament – Red Hot Mama


Ross

Ross in headphones

Top Tracks:

  • David Gray – Flame Turns Blue
  • Kings of Convenience – Singing Softly to Me
  • Grizzly Bear – Southern Point
  • The Roots – The Next Movement
  • Iron & Wine – Jezebel

Trevor

Trevor in headphones

Top Tracks:

  • Crawler – Cosmic Tide
  • Quest for Fire – Next to the Fire
  • Bison BC – Die of Devotion
  • Radiohead – Little by Little
  • DMX Krew – Do It All Nite


Scott

Scott in headphones

Top Tracks:

  • Wax Tailor – No Pity
  • Ray Charles – Eleanor Rigby
  • Steve Martin – How to Meet a Girl
  • KRS-One – MC’s Act Like They Don’t Know
  • Quantic – Snakes in the Grass


Dubious friendship

Oh, and I suppose I should share mine too:

•  Blueprint Car Crash – Gun Moll
•  Little Dragon – Come Home
•  Angus & Julia Stone – Black Crow
•  Plants and Animals – Celebration
•  of Montreal – Coquet Coquette


 

Find even more of what inspires us on our FlickrHQ Last.Fm Group

Photos by waferbaby

A YUI3 Module for the Flickr API

Flickr API YUI3 ModuleDuring the redesign of our photo page, we created a YUI3 API  module, used for all Flickr API transactions. This module has been designed to load component modules as needed, which makes for a speedy initial load. For example, the YUI3 IO module, which handles XMLHttpRequests in the YUI3 library, is loaded only if writing to the Flickr API. Read requests are all handled by the Get Module, which is included in the core of YUI3. A byproduct of this approach is that read requests can be made cross domain, without a proxy.

Reading from Flickr

Simply include gallery-flickr-api in your requirement definition. Then use the Y.flickrAPI.callMethod function, passing the Flickr API method name and callback as arguments. Have a look at the YUI3 Gallery page for more information about using gallery modules. Read requests can be made cross-domain without a proxy, since the request and response will be loaded as a script resource instead of using XMLHttpRequest.

YUI().use('gallery-flickr-api', function(Y) {

Y.flickrAPI.callMethod('flickr.panda.getPhotos',
{
api_key: 'your_api_key',
panda_name: 'ling ling',
extras: 'license, date_taken, owner_name',
per_page: 50,
page:1
},
{
success : function(response_object) {
//Success Code
},
failure : function(response_object) {
//Failure Code
}
});

});

Writing to Flickr

Okay, writing to Flickr API (i.e. post requests) is a little trickier. You will need to make sure that your API key is enabled to read and write. As soon as you make the first API request to write, the YUI3 IO module will be loaded (if it is not already). It is important to note that write requests can only be done on the same domain, which is the standard security restriction of the XMLHttpRequest for browsers. POST requests to the Flickr API, will require some sort of proxy which will accept the post request on your domain and handle the cross-domain communication with api.flickr.com

YUI().use('gallery-flickr-api', function(Y) {

Y.flickrAPI.callMethod('flickr.favorites.add',
{
api_key: 'your_api_key',
photo_id: a_photo_id
},
{
success : function(response_object) {
//Success Code
},
failure : function(response_object) {
//Failure Code
}
});

});

We hope that you make use of this new module. It is a great way to get quick prototypes and full scale applications made using the Flickr API in a short amount of time. See you in the App Garden!

The Joy of Popup Windows

The Joy of Popup Windows

As you’ve probably noticed, we’ve been working on the login process here at Flickr. In an effort to make this flow as painless as possible we’ve moved to a “contextual” login, which is to say, logging in doesn’t require you to leave the page you are looking at. We accomplished this by using a popup window.

We initially considered using an in-page modal dialog box instead of a popup, but quickly dismissed this approach for two security reasons. Firstly, to prevent phishing and cross site attacks it’s very important that login forms are not posted cross domain (Yahoo!’s Membership Platform authentication happens on yahoo.com, not flickr.com). Secondly the URL of the login page should never be hidden from the user.

An in-page modal fell foul of both of these concerns. A more suitable solution would be the popup browser window, once a favorite of advertisers.

Our initial approach

Revisiting our turn of the millennium Javascript for opening windows with the BOM, things looked simple enough: call window.open, make sure the window opened, then set a timeout to see if the window was closed:

function waitForWindowClose() {
	if (web1999Window && web1999Window.closed) {
		// the pop-up was closed, done with auth etc.
		window.location.reload();
	} else {
		// check again in a moment
		setTimeout(waitForWindowClose, 1000);
	}
}//cute function name by Scott Schiller
function webCirca1999(url) {
	try {
		web1999Window = window.open(url,'newWindow','width=640,height=650');
	} catch(e) {
		//handle blocked popup...
	}
	try {
		//this is to check if it really opened
		if (web1999Window.focus) {
			web1999Window.focus();
		}
	} catch(ee) {
		// also handle failure
	}
	waitForWindowClose();
	return false;
}

Unfortunately Yahoo!’s new user signup process closed our popup and opened a new one of their own. This triggered our waitForWindowClose{} function, web1999Window && web1999Window.closed evaluated to true, and the Flickr page refreshed, even though the user hadn’t logged in yet.

The next thing we tried was watching window focus. When the window blurred after clicking on the login popup, we’d know the popup was open, and when focus came back we’d check the user’s login credentials.

But focus tracking also had its problems. Some browsers didn’t always report the window blur event when a popup was open, resulting in inconsistent behavior. It was also possible for the user to accidentally focus back on the window and kill the flow. We tried several ways to track focus, and although behavior got better, it was never 100% reliable. For a feature like login that was obviously a problem.

The Cookie Solution

Ultimately we settled on a solution that forgot about popup tracking and window focus and instead concentrated on checking for the user’s login cookies. The code below polls to see if the login cookie is set, then makes an AJAX request to verify it, then refreshes the Flickr page:

function pollCookie(){

	if(document.cookie.match(sessionRegex)){
		authInProgress = false;
		checkAuth();
	}else{
		Y.later(20, this);
	}

}

This has the benefit of being very responsive, avoids the nightmare of focus tracking, and is low impact (as there’s no need for us to poll the server).

So today we have a reliable contextual login. Our popup system also made it possible to add google and facebook login with less pain than usual.

Flickr Shapefiles Public Dataset 2.0

An embarrassingly long time ago we released the first public version of the Flickr Shapefiles. What does this have to do with Captain America and a cat? Nothing, really.

Anyway, we haven’t completely forgotten about shapefiles and have finally gotten around to generating a new batch (read about Alpha Shapes to find out how it’s done). When Aaron did the first run we had somewhere around ninety million (90M) geotagged photos. Today we have over one hundred and ninety million (190M) and that number is growing rapidly. Of course lots of those will fall within the boundaries of the existing shapes and won’t give us any new information, but some of them will improve the boundaries of old shapes, and others will help create new shapes where there weren’t any before. Version 1 of the dataset had shapes for around one hundred and eighty thousand (180K) WOE IDs, and now we have shapes for roughly two hundred and seventy thousand (270K) WOE IDs. Woo.

The dataset is available for download today, available for use under the Creative Commons Zero Waiver:

http://www.flickr.com/services/shapefiles/2.0/

Little Johnny JSON

Today I'm a Cop.

Originally we provided the full dataset in our own home-grown XML format because, well, it seemed like a good idea. For version two we’re releasing the shapes in GeoJSON format. We think this is a Good Thing because unlike our  old XML format, at least one other person in the world already knows how to read and write GeoJSON. For example, Our friends over at Stamen Design and SimpleGeo have created a ridiculously easy-to-use JavaScript library called Polymaps which of course reads GeoJSON out of the box. With a few lines of JavaScript you can render the Flickr shapefiles and start using them without all that pesky XML parsing stuff:

Statez

Or if GeoJSON doesn’t suit you you can use a free tool like ogr2ogr to convert it to something that does.

Layers

1000 layers

(photo by doug88888)

The GeoJSON format allows grouping of features (and their related geometries) into FeatureCollection objects. A FeatureCollection seems to be roughly equivalent to a layer in a typical GIS, or a placetype in WhereOnEarth-speak. To make the dataset a little easier to manage we decided to break the shapes up into FeatureCollections based on placetype; one each for continents, countries, regions (states), counties, localities (cities), and neighbourhoods. Each of these is its own GeoJSON file in the dataset.

Here’s an example of what one of our GeoJSON objects looks like:

{
  "type": "FeatureCollection",
  "name": "Flickr Shapes Public Dataset 2.0 - Regions",
  "features": [
    {
      "type": "Feature",
      "id": 2344541,
      "properties": {
      "woe_id": 2344541,
      "place_id": "Cxf0SmObApi9R9T8",
      "place_type": "region",
      "place_type_id": 8,
      "label": "Barbuda, AG, Antigua and Barbuda",
    },
    "geometry":
    {
      "type": "MultiPolygon",
      "created": 1292444482,
      "alpha": 0.03,
      "points": 118,
      "edges": 17,
      "is_donuthole": 0,
      "link": {
        "href": "http://farm6.static.flickr.com/5209/shapefiles/2344541_20101215_724c4cae47.tar.gz",
      },
      "bbox": [-62.314453125,17.086019515991,-61.690979003906,17.93692779541],
      "coordinates": [
        [
          [[-61.739044,17.587740], [-61.735268,17.546171], [-61.690979,17.426649], [-61.765137,17.413546]
... etc

A file is a single FeatureCollection object which holds an array of Features, which each hold a Geometry which is a MultiPolygon, which holds an array of Polygons which in turn each consist of an array of LinearRings. Got it?

You’ve Been Superseded

We’ve also included a separate file/layer called flickr_shapes_superseded.geojson. This is a FeatureCollection that consists of all the WOE IDs that have been “superseded”. Occasionally (too often?) a WOE ID needs to be retired and replaced with a new one. We keep up to date with these and are always reverse-geocoding against the latest WOE IDs. However, there are plenty of old photos (and Flickr shapes) that have been assigned to one of these old IDs, and we have shapes for them (currently a little more than nine thousand). A simple solution might be to just re-assign these photos to the new WOE IDs (when a WOE ID is retired its replacement is specified), or even just re-run the reverse-geocoding process. This may not be what the owner of the photo wants; it might come out with a different result than they had at first (which they may have been perfectly happy with), if the size and location of the WOE rectangle changed (which it probably did). So it’s a problem without a clear solution. And since we have data for these old WOE IDs we’ve included their associated shapes in the dataset. In most cases there will be shapes corresponding to the new WOE IDs that will over time become more and more accurate as more photos get uploaded and end up being assigned to the new WOE IDs. But you can have the old ones too.

WTF

Sometimes, Clustr just gets it wrong. As mentioned in previous posts many of the Flickr shapes are just plain weird. It may be due to a lack of data (i.e. source photos), a weakness of the algorithm, an inappropriate choice of the alpha parameter or (shame!) a plain old bug. One of the things that surprised us was that Clustr was not supposed to output inner rings, or polygons with holes. It turns out that it does. In the GeoJSON output this can cause some weirdness (depending on what you use to render the shapes) since the GeoJSON is formatted with the assumption that each ring is a distinct polygon, instead of possibly one part of a single polygon with holes in it. This and other weirdness are known issues and something we shall strive to fix in the future, however we felt it best to release the existing dataset now rather than spend forever trying to get it perfect, and end up not releasing anything at all.

You may also notice that unlike version 1 of the dataset, there is only a single shape per WOE ID. All of the previous versions of the shapes are still available via the Flickr API, but in the interests of keeping the file size down we’ve limited this download to just the latest versions of each shape.

And as always, there’s lots more to do.