The Flickr photo page has gone through several revisions over the years. It was initially designed for 800×600 pixel displays, with a 500 pixel wide photo and a 250 pixel wide sidebar.

The 500×375 photo takes up 9.1% of the 1905×1079 pixels available in my viewport

By 2010, display resolutions had increased significantly, and 1024×768 became the new standard for our smallest supported resolution. We launched a re-designed photo page, designed for a width of 960. It featured a 640 pixel wide photo and a sidebar of 300 pixels.

The 640×480 photo takes up 14.9% of the 1905×1079 pixels available in my viewport

Since then the number of different display resolutions has increased and larger sizes have become more popular, but the number of users still on 1024×768 displays have made it hard to increase the width of the page beyond 960. We realized that we would always have to support smaller monitors, but that there was no reason not to give bigger photos to those with larger monitors. The recent launch of the 800, 1600, and 2048 photo sizes gave us a lot of different options for showing big, beautiful photos to members, and we wanted to take advantage of that. Starting today, we will display the biggest photo that we can on the photo page for your monitor.

The 1213×910 photo takes up 53.7% of the 1905×1079 pixels available in my viewport

Algorithmic

As you use the new liquid photo page, you may notice that the page content doesn’t always fill the entire viewport. This is because we created an algorithm for taking the width and height into account that will display content at a width that will best showcase the most common photo ratio, the 4:3. Here are the goals of that algorithm:

1. Show the biggest photo the window allows
2. Ensure the title and the sidebar are visible
3. Keep the width of the page consistent across all photo pages, regardless of the individual photo dimensions
4. Whenever possible, prefer native dimensions of a photo size (i.e., resist downsampling and never upsample)

Going Big

Big photos are really compelling. We knew from using the Flickr Light Box that our members’ photos look amazing at full screen, and we wanted to give the same experience on the photo page. This part of the algorithm was easy; as soon as the page starts loading, we read the innerWidth and innerHeight of the viewport (or the browser’s equivalent), and then go through the photo sizes that the photo owner allows us to display to find the best fit. If the photo is a little too big for the space we have to work with, we scale it down in the browser.

Providing Context

As great as a giant photo is, a photo is more than just its pixels. The context and story around a photo is just as important. Imagine a photo of a tiger; it’s impressive in its own right, but throw in a map showing that the tiger is in a public park, and a title stating, “A Tiger Escaped From the Zoo!” and then you really have something.</>

We decided that the title and the sidebar are important enough to make it worth showing a slightly smaller photo on the page. We adjusted the algorithm to take into account the width of the sidebar and its gutter (335 pixels) and the height of the first line of the title (45 pixels) when calculating how much available space there is for a photo.

Site Consistency

So far, so good. However, as we used the liquid photo page we noticed that it had one fatal flaw: Since the algorithm uses the dimensions of the photo that you are viewing to adjust the page width, it changes from photo to photo. This mean that if you’re browsing through some photos, the elements of the page are moving around from page to page. This is especially problematic with the header and the Next / Previous buttons; It’s incredibly difficult to navigate around if you always have to hunt around to find them first.

To fix this problem, we decided to make the algorithm ignore the dimensions of the currently displayed photo when calculating page width, and instead to always use the dimensions of an imaginary 4:3 photo. This means that the page width will always be the same for any given combination of viewport width and viewport height, and that the UI elements will be in the same places for each page. The downsides of this are that photos that aren’t 4:3 will have more whitespace around them and even potentially be cut off by the bottom of the page, forcing the viewer to scroll. Using a consistent width is definitely the lesser of the two evils, though. The current photo page has the same problem with photos that are taller than they are wide being below the fold, and we’ve been happily viewing them for years.

Going Native

These days, browsers do a pretty good job scaling a photo down. By default, most browsers err on the side of quality rather than speed, so the resulting photo should look good regardless of the size it is displayed. That being said, if we ever downsample a photo, then we are downloading more pixels than we need and throwing them away. This isn’t good for performance.

We adjusted the algorithm to favor native sizes, even if that means a slightly smaller photo is shown. We coded in detents, so that if a photo size is within 60 pixels of a native size, we will just use that size instead of downsampling a larger one. This means the page loads faster and that most common monitor resolutions will see photos at the native size, as this table illustrates (percentage use data from StatCounter):

Titles Are for Squares, Man

Square photos are an interesting loophole in the way we size photos. Because we’re targeting an imaginary 4:3 photo, square photos will be displayed with more actual pixels than any other size, taking up the full width and height allotted. While browsing the site we noticed this, as well as the fact that the title is never visible. In order to bring the overall pixel count more in line with landscape and portrait photos, we reduce the size of square photos a bit more than the others. This helps ensure that the titles are always visible as well.

Making it Fast

Now that the algorithm is complete, we need to work on the performance. We noticed that reading the viewport dimensions and resizing the page every single time you go to a photo is unnecessary and distracting (since the page loads with a width of 960 and must be adjusted after the JavaScript loads on the page). To fix this, we cache the viewport dimensions in a cookie that can be read by the PHP code that generates the page. The first time you go to a liquid photo page, we have no choice but to adjust the page width on the fly. But every other photo page you visit will have the dimensions stored from the last page, and the page will be rendered with the correct width from the start.

More to Come

We have a lot more changes in store for this year. Stay tuned!

The new Flickr Web Uploadr is the result of a good amount of prototyping, research and good old-fashioned testing across the team that built it. This article goes into some of the details behind the “grid” – the area where photo thumbnails are shown – and sheds a little light on some of the thinking and logic behind the scenes. It’s a little lengthy, but don’t worry, there are pictures!

In April 2012, Flickr started rolling out its new web-based upload UI to the masses. We’re stoked to see it out there, and user feedback has been overwhelmingly positive. The product is an ongoing work in progress and enhancements are still being added, but the core is quite well-established and the experience is a significant upgrade over the one provided by the previous web-based uploadr.

The new Flickr Web Uploadr. It’s powerful, it’s got a dark background, and it’s fast.

The new uploadr has also simply been fun to work on; there are numerous interesting challenges in terms of UI, interactions, performance and sheer scale on the front-end that we had to feel confident in tackling before we were able to commit to moving forward with the project.

Building The Grid: Prototypes

Initial discussions about the new Flickr uploadr weren’t too detailed, because I think everyone already had a pretty good idea of what we wanted to see in a browser: Something more desktop-like, feature-wise (like our older XUL-based Flickr Uploadr application) that would load and show photo thumbnails in a grid arrangement, with a desktop-like selection and batch editing model.

The next step was to start building a prototype in plain old HTML, CSS and JavaScript, and then figure out how many photos we could potentially get into the thing before it broke down. Could the grid handle selection and editing of 1,000 items? 10,000 items? I was cautiously optimistic. A continuous joke I had with the team was that I had built this before, in 2005: The project was an adventurous redesign of Yahoo! Photos, and joking aside, it actually did share a lot of design and interaction elements in common with what we were about to build. In 2005, we were targeting IE 6 and Firefox 1.5, so the landscape has changed a lot in terms of support and performance. Seven years later, it was fun to review some of the lessons and fun bits from the Y! Photos redesign as applicable to Flickr.

Prototype: Fluid Grid Layout

Some of the first prototypes involved building a grid layout, forming a two-column page that would be fluid to the browser width. We wanted to guarantee at least three photos per row would show in the grid, so the thumbnails could scale themselves relative to the browser size in order to fit in the space – easily done via CSS’ min-width and max-width attributes.

A very early version of the uploadr UI.

The earliest prototypes simply populated the DOM with a few hundred copies of a cloned photo item “template”, to give the idea of what a busy UI might look like. It was mostly just HTML and CSS at this point.

With the grid rendering in fluid form as a series of inline-block <li> elements, the next thing to start was the selection model.

Selection and Drag Events

Building a desktop-like selection and drag-and-drop model can be a technical challenge, given the underlying complexity. As anyone who’s built one of these will understand, there are a whole ton of interactions one must consider and account for between event monitoring, coordinate tracking, drag-to-select vs. rearrange intents, event cancellation, handling of invalid actions and so on.

Selection

In general, all user interactions start with watching mousedown() events inside the grid area. If mousedown() fires within “whitespace”, any existing selection is reset and mousemove() events are then used to draw a selection marquee which compares coordinates to the grid, highlighting items based on basic region intersection logic (for example, xyToRowCol(), points can be checked to see what grid row/column they fall within and thus “from/to” ranges can be established for a given marquee box.) Once a mouseup() event fires, selection can be completed and the mousemove() and mouseup() handlers released.

Testing the selection UI at various grid sizes.

The above marquee drawing and intersection logic is not terribly fancy, but things start to get interesting when you throw in additional positioning considerations like vertical offset from window scrolling (and drag-initiated window scrolling), browser window resizing affecting layout, positioning of the marquee UI vs. coordinates of the underlying grid items and so forth. Keyboard modifiers can also affect selection mode – whether selection is exclusive, additive or toggle-based – so an intersect does not also always mean “select this item”, too.

Marquee selection mode in action.

Dragging + Rearrange

When mousedown() fires on an unselected grid item, selection can immediately change to only that item (unless selection mode is additive or toggle-based via a modifier key.) If firing on an already-selected item, mousemove() is watched for a “threshold” of perhaps 4+ pixels of movement from the original coordinates, at which point “dragging” becomes active.

Once dragging has begun, the selected grid DOM elements are marked with a “disabled” CSS class, greying them out somewhat to indicate drag state, and mousemove() now moves around a cursor trailer that shows the count of items being rearranged.

Rearrange mode, once entered, is similar to the marquee selection mode except that now only a single mouse coordinate is checked in order to determine what row and column is the current “target” for rearrange – that is, what position the user intends to drag the selected photo(s) to. The logic here can get interesting in edge cases, because the user is able to insert both “before” and “after” a given target point based on whether the cursor is on the left side, or the right side of the target.

In terms of the UI, the current drag target simply has an “insert-before” or “insert-after” CSS class appended to it which results in the appropriate “insert point” marker (a CSS border) being applied to it.

Rearrange mode in action.

Once mouseup() fires on a valid rearrange target, the actual rearrange action is applied to both the UI and data model. The underlying JavaScript re-appends the dragged DOM nodes next to their new target sibling node and then splices the photo item array, matching the order of the array to the new layout shown in the UI.

Additional Selection Interactions

A few other use cases to consider: Clicking an item, then shift + clicking another should have the effect of setting an “anchor point”, and selecting a range of items from X-Y within the grid. The user should be able, once setting an anchor point, to “pivot” from that point by clicking while continuing to hold the shift key. (Put another way, holding shift should not set the anchor point when clicking.)

By holding CTRL (or the Command/Apple key on OS X), selection should be additive and toggle-based. My approach to this meant taking a “snapshot” of the selection when marquee drawing begins, and then applying the logic based on mouse coordinates and keypresses with each draw action. This way, you can draw a marquee over and out of an existing selection, causing it to “toggle” and reset accordingly without losing your original state. A new snapshot is only taken once the selection is finalized at mouseup() time.

Demo video: Uploadr Prototype UI

Here is a screencast of a very early version of the Uploadr grid UI, showing the basics of mouse-based selection interactions, scrolling and resizing. By this time, selection events were also firing and updating the “editr panel” area as well.

[flickr video=7177694856 secret=3fb0a325e4 w=500 h=398]

Enter The Keyboard

With mouse events working, additional consideration was given to keyboard shortcuts. We intended to have a UI that supported most if not all of the same selection, editing and rearrange actions that could be achieved via the mouse. An important part to making this work involved watching focus inside the grid, tracking the last-known selected item, and supporting the use of the arrow keys as a means of changing focus between grid items.

Focus-based navigation in the grid is interesting, more akin to mouse movement and hover behaviour. It is intentionally separate from keyboard-based selection (which is invoked with a toggle behaviour via the spacebar, or selection and editing of a single item via the return key.) Using this approach, it is relatively easy to navigate and build up a selection of items via the arrow keys and spacebar.

For rearrange, a cut-and-paste approach was used; CTRL or Command/Apple + X (“cut”) are used to begin rearrange, arrow keys set the target rearrange point, and CTRL + V or return will apply the rearrange at the given target. If active, pressing escape will exit rearrange mode.

Performance: Scaling The Front-End

An important step in the grid prototype, once it was rendering in a fluid fashion, was to see find all the ways in which we could get it to break down. Which browsers were first to choke under the DOM load as more nodes were written out? Was layout and rendering the bottleneck? Were too many events firing? Was the JS engine spending too much time updating the DOM?

After rendering several hundred photos in the UI, we started to see evidence of browsers getting laggy in terms of responsiveness, and CPU + RAM use trending upward. With plans to extend this UI to handle numbers of photos in the thousands, a number of optimizations were made up front including aggressive pruning of the DOM as the user scrolled the page.

In brief, the trick is to create a large page with no content and only generate the DOM to reflect the slice of the whole view being shown.

Given events like window scrolling and resize affecting browser coordinates and DOM layout, we are easily able to calculate and cache the changes as they happen, making quick lookups to determine precisely what range of grid items are in view for the user. A single “page” of grid items can then be generated on the fly, appended to the DOM and shown to the browser. Events like browser resize invalidates the coordinate cache, so the DOM reflows and the grid refresh / display process repeats itself in a throttled fashion when this happens.

Event Throttling: Responsiveness’ Dirty Little Secret

Native DOM events are useful, but they can fire quite aggressively and left unchecked, can really hurt the performance of your application. Scrolling and resize are good examples for the grid case, as we want the UI to respond with an updated display pretty quickly when scrolling – but we know that we only have to show new items when a new row comes into view, which is typically only every 200 vertical pixels. With resizing, we only need to reflow the grid when resizing has added or removed enough horizontal room that we’ve lost or gained a new column.

In short, if you know events will fire often, subscribe to all of them but only do expensive work if there are real changes to apply. Alternately, you could only let resize handlers (for example) fire once every 500 milliseconds and do the work every time, so your handler only fires twice a second in the worst-case scenario.

Cache The Hell Out Of The DOM

This was hinted at previously, but is worth repeating: Get references and read values once, particularly from the DOM, and cache them when initially retrieving and updating them in response to events. If you know what a value is going to be, don’t query for it.

In JavaScript, an internal lookup is far faster than reaching out to query the DOM for attributes like offsetWidth, for example. Simply reading certain attributes of DOM nodes can cause layout and reflow to happen in the browser, which means you’re making the browser do more work for information that is likely unchanged. Thrown into a loop mixed with DOM writes, this makes for pretty disastrous browser performance.

JavaScript frameworks like YUI et al should do their own caching of this data, but I see no downside in grabbing and storing this stuff locally yourself; as the implementer, you have the best idea of what data is most static and what is not.

Additionally, try to read at once and write at once to the DOM; don’t have loops that do a write and then a read, for example. Try to write DOM interactions that follow the browser’s rendering model, minimizing the back-and-forth of layout/reflow/display calculations. Use document fragments to build up collections of DOM nodes, and append them once to the DOM vs. using innerHTML, or – worse – multiple appendChild() calls. Don’t query className when you likely know what it’s going to be; track that state internally in JS, instead, and only write changes out to the DOM.

“Stateful” CSS Class Names

I’ve been a fan of the concept of “stateful” CSS – eg., .is_selected { border: red; } for years. Not only is state consistent, but using CSS in this way also encourages better separation of concerns (and less temptation to add or remove DOM nodes via JS when making changes.)

When you want to grey something out, for example, you may set a disabled property to true on a JS object. That easily translates to a CSS class name change including .disabled {} applied to the relevant DOM node. As a result, your DOM is logically reflecting your JS state. It’s also helpful when troubleshooting, because you can add the class name to nodes ad-hoc when testing UI features.

For the grid’s purposes, every grid item contains all relevant “states” and the markup for those states – selection, thumbnail, progress, overlay icons, messages, errors and so forth. This makes it very easy to change the item’s display with a single, or few additional CSS class names, and minimizes the amount of work JS has to do to update the DOM. It is also trivial to combine states this way, also – e.g., a photo upload that has a thumbnail, but is in a “failed” state because it’s over-size.

While uploading, for example, a grid item may have class="has-thumbnail working selected", then completes with class="has-thumbnail has-fullsize-thumbnail complete" when the upload has finished. All JS did here was update the class name (and while actively uploading, redraw a small progress meter on the item.) Thus, JS/DOM interaction is fairly minimal.

A single CSS change can also completely change the display of the grid, also. “Info view” is one example of this. When enabled, a single additional class on the grid container causes all photo items to show overlay icons with their privacy state, and additional icons if they have tags, are in a set and so on.

“Info” view, showing overlays with privacy, state and other information.

Broadcast Events FTW

Events are a great way for modular bits of code, written by the same or separate people, to work on separate problems independently. Among other things, the grid listens for events regarding file addition, removal, progress and success / failure states from the upload queue module. The grid generates and fires events itself reflecting changes around selection, editing and arrangement as the user is doing their work, which are picked up by the “editr panel” at left that updates to reflect the selection state. Provided that events are kept as simple notifications and relatively one-way, there is little risk of complex event-related tracing in the unlikely, er – event – that something that goes wrong.

Flickr uses YUI 3 extensively, and we write and plug our application code into the system as YUI 3 modules. In addition to the excellent modular framework approach, we take advantage of the DOM and Event functionality in particular.

In Summary

The grid is only one of several modules that make up the new Flickr Web Uploadr, and is primarily responsible for the display and updating of photo thumbnails, selection, arrangement and basic metadata. There is a lot more going on in terms of JavaScript and network state under the hood, including API calls and permissions; posts highlighting some of the other fun areas are forthcoming.

As it turns out, building a feature-rich browser-based application for millions of people that looks good, is fast and supports many use cases including constraints and unexpected error conditions, can be a challenge. It’s also part of the fun.

We recently had a Flickr Frontend Night at BayJax, the Bay Area JavaScript group. We’ll be posting the videos from those talks over the next couple of weeks.

First up! Frontend Engineer, DJ, and all-around nice guy Scott Schiller, with his great talk on Web Audio.

Scott used impress.js to make his awesome slides (using HTML and CSS Transitions). If you want to dig deeper into Scott’s talk, there is the HD version on YouTube, slides, the Wheels of Steel demo, and the HTML5 game he created, SURVIVOR.

Big thanks to Gonzalo Cordero for organizing the event, and to Ryan Grove and Allen Rabinovich for their great work filming it.

On a side note, if you’re in Austin for SxSW next week, be sure to check out talks by Flickr’s own Eric Gelinas (Geo Interfaces for Actual Humans) and Stephen Woods (Creating Responsive HTML5 Touch Interfaces).

Thanks to softdroid.net for creating a Ukrainian translation of this post.

Last year, we added support for OAuth 1.0 – a much better way to have your users authenticating against Flickr. More information on Flickr user authentication via this method is available here in our Developer Guide and specifically here.

If you are app already uses OAuth, then you can skip this post and look at some gorgeous photos instead. However – if your app still uses the old Authentication API, you will need to update it to OAuth by July 31st this year.

Updating to OAuth is easy and you don’t worry about any user impact. You can exchange an old auth token from the old Authentication API, to an OAuth access token. The process simply requires that you make an authenticated request to the flickr.auth.oauth.getAccessToken API, which will exchange the old token used to make the request, with a new OAuth access token. Again, everything is documented right here. Flickr member Jef Poskanzer has also written an overview and comparison between the two auth methods: http://acme.com/flickr/authmap.html.

After July 31st, we will no longer support the old Authentication API.

More information is available in the Flickr API FAQ’s, and in the Flickr Developer guide, but if you have specific questions about updating your app, you may file a help ticket here.

Thanks for making Flickr more fun by contributing to our growing collection of apps!

Your Engineering Team at Flickr
(@FlickrAPI)

I recently asked our friend Sean from Pleiades (which I will *never* be able to spell correctly) to write up a lil’ guest post on how we did something cool with Flickr machine tags and ancient sites of the world – and here it is!

  $ curl http://pleiades.stoa.org/places/440947682/json | python -mjson.tool

  {
    ...
    "id": "440947682",
    "title": "El Kab",
    "type": "FeatureCollection"
  }

  class RelatedFlickrJson(BrowserView):

      """Makes two Flickr API calls and writes the number of related
      photos and URLs for the most viewed related photo from the Pleiades
      Places group to JSON like

      {"portrait": {
         "url": "http://flickr.com/photos/27621672@N04/3734425631/in/pool-1876758@N22",
         "img": "http://farm3.staticflickr.com/2474/3734425631_b15979f2cd_m.jpg",
         "title": "Pont d'Ambroix by sgillies" },
       "related": {
         "url": ["http://www.flickr.com/photos/tags/pleiades:*=149492/"],
         "total": 2 }}

      for use in the Flickr Photos portlet on every Pleiades place page.
      """

      def __call__(self, **kw):
          data = {}

          pid = self.context.getId() # local id like "149492"

          # Count of related photos

          tag = "pleiades:*=" + pid

          h = httplib2.Http()
          q = dict(
              method="flickr.photos.search",
              api_key=FLICKR_API_KEY,
              machine_tags="pleiades:*=%s" % self.context.getId(),
              format="json",
              nojsoncallback=1 )

          resp, content = h.request(FLICKR_API_ENDPOINT + "?" + urlencode(q), "GET")

          if resp['status'] == "200":
              total = 0
              photos = simplejson.loads(content).get('photos')
              if photos:
                  total = int(photos['total'])

              data['related'] = dict(total=total, url=FLICKR_TAGS_BASE + tag)

          # Get portrait photo from group pool

          tag = "pleiades:depicts=" + pid

          h = httplib2.Http()
          q = dict(
              method="flickr.groups.pools.getPhotos",
              api_key=FLICKR_API_KEY,
              group_id=PLEIADES_PLACES_ID,
              tags=tag,
              extras="views",
              format="json",
              nojsoncallback=1 )

          resp, content = h.request(FLICKR_API_ENDPOINT + "?" + urlencode(q), "GET")

          if resp['status'] == '200':
              total = 0
              photos = simplejson.loads(content).get('photos')
              if photos:
                  total = int(photos['total'])
              if total < 1:
                  data['portrait'] = None
              else:
                  # Sort found photos by number of views, descending
                  most_viewed = sorted(
                      photos['photo'], key=lambda p: p['views'], reverse=True )
                  photo = most_viewed[0]

                  title = photo['title'] + " by " + photo['ownername']
                  data['portrait'] = dict(
                      title=title, img=IMG_TMPL % photo, url=PAGE_TMPL % photo )

          self.request.response.setStatus(200)
          self.request.response.setHeader('Content-Type', 'application/json')
          return simplejson.dumps(data)

Superheros Neil and Nolan (N&N) just gave their talk on our real-time PuSH system at Web 2.0 in New York. You can download the slides, or view the contents I’ve oh-so-roughly transcribed here:

Oh Hai My name is Nils.

My name is Neil Walker, I’m an Engineer at Flickr.

I mostly work on the back-end infrastructure portion of things. I’m going to be telling you about a system Nolan and I built to send real-time updates about things that happen on Flickr out to the things and people that want to know about them.

My portion of the talk is mostly going to be concerned with the back-end. How did we build the guts of the system and what we think are the key components.

What I’m not talking about.

This talk isn’t about building Twitter, or a full-scale pubsub system built from the ground-up to handle millions of updates per second. Instead it’s geared around what you can do with less. What if you’ve got an existing site with lots of functionality already built and you want to add some real-time notification capabilities to it? What can you do if you don’t have a lot of resources to throw at the problem?

It turns out that you can get pretty far with some bits and pieces that many sites will already have, and how to fit those bits together is what we’re going to cover.

This is Nolan

Hi, I’m Nolan Caudill, a backend engineer at Flickr. I work on most of our backend systems, but focus mainly on geo, the API, internationalization and localization, and general site performance.

The Fun Part

I’m going to focus mainly on why you would want to use the new push api, what problems it solves, and why it’s better in some cases than our traditional pull-based APIs. Also, I’m going to talk about how you get up and running with the new APIs. We know it’s a bit of mindflip from the traditional APIs and we want to help you with the transition on getting up and running with it.

So this was the Wired cover for March 1997. From the article: “Media that merrily slip across channels, guiding human attention as it skips from desktop screen to phonetop screen to a car windshield. These new interfaces work on the emerging universe of networked media that are spreading across the telecosm.” Whatever that means. So that was almost 15 years ago. “Kiss your browser goodbye”! didn’t exactly pan out.

Flickr PuSH

But jumping forward to now, we DO have all those screens. Most of us have 3, 4 or more devices with high-resolution displays on them, almost all of them with web browsers, and varying levels of human interfaces. Some of them even have cameras. Lots of us have 2nd or even 3rd monitors at work. For Flickr, having apps that can be used to explore photos on all those different screens regardless of user interface seems only natural, and that’s what we hoped to inspire with a real-time API. Before we get into the details I’d like to show a simple little application that Aaron Cope, who used to work at Flickr, built on top of our PUSH API, just to see what would happen.

Basically it’s a simple web page that can be run full-screen on almost any device that has a browser. It lets you subscribe to various streams on Flickr like photos from your friends, photos that your friends favorite, photos with particular tags, and photos from a location. It then receives these photos in more or less real-time and displays them full-screen with the title overlaid on top of the photo.

That’s it. No controls, no user interface after the initial point of telling it what you’re interested in.

So Flickr being a social website, you start to get photos of your friends, and the things that happen to them. You also get photos of things that your friends are interested in. As they fave them, they show up in your live stream. Then you might fave the same image, and more of your friends see it, fave it, and so on. It’s a very natural way for a popular object to percolate around your network, and often you don’t have to worry about missing something good because if it’s popular it’ll bubble back up as another one of your contacts faves it later on.

Interesting things start happening when you have devices with screens AND cameras – a photo makes it into someone’s stream, they take a photo of it and upload that, which makes it back into the original uploader’s stream, who faves it, etc. and you get a sort of live conversation taking place with photos. Knowing that when you upload a photo it’s going to appear on someone’s monitor or widescreen TV adds a new dimension to photosharing.

Eventually it can get a bit ridiculous.

A bit of history.

Anyway, we’ll go back to how all this got started at Flickr.

– A while ago we had the need to get new public uploads/updates to search partners in a timely manner. Why? – Being crawled is fine but is expensive for the crawler and crawlee and isn’t always accurate. – Making a specific API for that purpose could do it but is probably going to be clunky, require too much work by both parties, and means that whatever it is that’s hitting your API can potentially affect your site performance, or other API users. Pushing out updates where you control the flow is really the way to go. – What did we build?

‘headers’ => array( ‘Content-type’ => ‘application/atom+xml’, ‘User-Agent’ => ‘Flickr Kitten Hose’ )

Basically a firehose of public uploads/updates (incl. deletions/privacy changes) that we can aim somewhere and turn it on. Similar to the Twitter firehose, but for photos uploaded to Flickr. So essentially stuff happens on Flickr, we transform the events into some easily parseable format, bundle everything up into reasonable-sized blobs and POST it to a web server somewhere that consumes the data.

PubSubHubbub: Thanks, Google

Rather than invent a specific format/protocol we decided to pick something familiar. Something that already has a spec, is well-documented and is well-understood. So we picked a common Publish-Subscribe protocol. Google’s PubSubHubbub has definitions for how to publish a feed to a hub, how to subscribe and unsubscribe to topics and verify subscriptions and all the other good things that people who write specs like to specify. Of course we didn’t actually need a lot of the spec to accomplish our goals for just the firehose bit, but we did have in the back of our minds the thought of expanding on what we built.

So choosing at the start to meet an accepted spec was probably a good idea, and allowed us to get going without having to think too hard about what might come later – because someone else already figured out how that stuff should work.

What?

A hub with no subscription control and a single hard- coded endpoint. If you want to fit the firehose idea into the PubSub metaphor then essentially what we built at first was a hub (i.e all of Flickr) that has one hard-coded “topic” that can be subscribed to – namely all public uploads & updates. Except that the subscribing and unsubscribing part involves humans agreeing to things and then turning the firehose on, instead of machines POSTing to web servers and receiving callbacks, etc. But the pubsub metaphor still fits.

How?

So how did we build it? What I’m going to get into now are what I think are the key pieces of the back-end. I’ll be glossing over some parts of it that are pretty basic and not terribly interesting in the interests of focusing on the good stuff.

Async task system Gearman, etc

The first (and probably the most critical) part of what we built is an asynchronous job queue, or what we often call our Offline Task System. Essentially it’s a way of de-coupling expensive work from the web servers. Sooner or later you end up wanting to perform an operation that takes longer than you want to make a user at the other end wait (or places more load on your web servers than you care to suffer) and so you want to off-load that work somewhere else.

We use this concept EVERYWHERE. Examples: Modifying/deleting large batches of photos. Computing recommendations for who you might want to add as a contact. We have several hundred different tasks, and there are often thousands of them running in parallel. We built our own that fits our specific needs, but a great open-source example is Gearman. Mature, easy to set up and use and scales quite well.

Stuff happens on flickr.

So the very start of the flow of our system begins when stuff happens on Flickr. The obvious example is a photo upload, often involving cats. Other things we’re interested in are updates: Changing the title, the description or other meta-data of a photo. Also we want to provide updates when photos are deleted (or have their visibility switched to private, which in terms of updates should look the same as a deletion)

Insert a task. olt::insert(‘push_new_photo’, $photo_id);

When any of these things happen, we simply insert a task into our offline task system. It’s very low cost, doesn’t block and once that’s complete everything else that happens is completely de-coupled for the web servers.

Task runs.

Insert the event into a queue When the task runs, all it does is take the event it represents, transforms it into a little blob of JSON and sticks it in a queue.

Redis Lists

RPUSH kitten_hose blob_of_json

For the queues we chose to use Redis Lists. For those of you not familiar with Redis yet, it’s an open-source, memory-only, key-value store that supports structured data. Think memcached with commonly-used data structures like hashes, sets, and lists and efficient operations on them. At this point Redis is fairly stable and reliable, the performance is fantastic, and it’s dead-simple to use.

We accept a little bit of a HA compromise for the fact that it’s RAM-only and not clustered (yet), so if it’s down you’ll drop some updates on the floor. But our goal here is just building a firehose for updates – not a 100%-reliable archive of user activity. We can happily accept the trade-offs that using Redis implies.

Uhhh

Why didn’t we just insert into the queue in the first place? At this point it might occur to you that instead of inserting a task (into a queue) that when it runs just inserts something else into another queue, why not just insert the thing into the queue in the first place? We’ll get to that in a bit.

However, two useful properties of our task system are that 1) you can insert tasks with a delay before they run, and 2) task inserts with the same arguments as an existing task fail silently – i.e. you only get the one task.

The upshot is that you can set a delay on things like updates so that a number of updates that happen over a short period (example) only generate a single task – and when it runs it finds all of them.

Then what happens?

Summary:

So at this point you’ve got events happening all over Flickr, uploads and updates (around 100/s depending on the time of day), all of them inserting tasks, and the task system grabbing each event and sticking it in a queue, in the form of a Redis list.

The next step is obviously to consume the queue.

Cron

• Insert more tasks!
• Consume the queue
• LPOP kitten_hose

You could have a daemon that sits there and consumes the queue and makes posts to the endpoint and you’re done. Basically what we do is have a cron job that periodically looks at the queue and inserts one or more tasks (depending on the size of the queue) whose job is to drain a particular number of updates from the queue and send them to the endpoint.

At first this is going to seem needlessly complicated but you’ll see why when we get to generalize the system for an arbitrary number of subscribers. One thing it does however is provide a convenient way to throttle and buffer the output – depending on what your endpoint can handle you can twiddle the knobs on your task system to run more jobs in parallel, or turn it down to choke off the firehose (and maybe drop updates on the floor if the queue gets too big…). It gives you flexibility by decoupling the delivery mechanism.

Draw me a picture

It’s probably time for a simple diagram: Things happening on Flickr trigger the creation of tasks, that run asynchronously, and stuff things into a queue in Redis. Periodically more tasks get triggered to drain the queue and push whatever they find back out to the eventual destination.

It worked pretty well.

• Didn’t take the site down
• Backfill 3B photos

So it turns out that it all actually worked pretty well.

The decoupling of everything using the asynchronous task approach meant that it had pretty much zero impact on the rest of the site. We could develop it live, experiment with capacity etc without having any fear of impacting site performance for our users because of how loosely-coupled all the pieces were – if anything goes wrong the damage is limited to that little piece.

Eventually we decided to run a backfill to a 3rd-party search index of all of our public photos back to the beginning of Flickr (somewhere around 3 billion) and we were able to complete it in 2 weeks, using our existing task system and the only new hardware being one redis box for the queues. So, it worked really well!

Photos from your friends

So we started to ask what if we wanted to make it into something that might be useful on an individual level? It seemed to be pretty reliable and low-impact to the site and showed promise of scaling fairly well. The obvious thing that would be of interest on a user level would be photos from your contacts; Here’s my endpoint, POST stuff to me when my friends upload new photos (or change existing ones).

Other things that could be of interest are when your contacts favorite a photo, or when you or your contacts are tagged in a photo, or when someone anywhere on Flickr tags a photo with the tag “kitten”. But let’s start with photos from your contacts.

What?

• A list of users (we have those!)
• An endpoint (URL)

Looking at what we’d need to add to the system to support user-specific subscriptions there’s the obvious: a record of the subscription somewhere in a database, the callback mechanism as per the pubhub spec etc. I’m going to gloss over that because it’s pretty straightforward and not really interesting. What we need to look at to manage the updates and figure out who gets what though is basically a mapping of users (the contacts who upload photos) to endpoints (i.e. subscribers).

Moar Redis

SADD user_1234 endpoint_5678

So we turned to redis again. Redis offers a set datastructure that we can use to maintain this relationship pretty easily. When someone wants to subscribe to photos from their contacts, we create a redis set for each of their contacts and add to it a pointer to the endpoint.

The task again.

SMEMBERS user_1234

foreach $endpoint RPUSH $endpoint json_blob

So now we go back to the Task that gets inserted in response to something happening on Flickr. Where it used to just insert an event into “The” single queue, now it looks at the set of endpoints that are interested in uploads from that user and inserts the event into the queue for each of them. Hopefully now you can see why we have a task do this rather than do it on the web server while the user waits – it’s still going to be quick because the queue inserts in redis are constant-time, but if you have a lot of contacts they could add up. Best to just de-couple it all and not worry about running into that problem.

Cron again.

Insert tasks for each queue (endpoint)

Now we turn to draining the queues, and again you can see where the task system comes in. With a large number of endpoints all the cron job has to do is run through each of them, look at how many events need to be consumed and insert an appropriate number of tasks. So scaling the output part of the system (and actually most of the input too) then becomes a matter of scaling your task system – and that’s a problem that’s already been solved.

Cache is your friend.

One thing that plays a key role in the system that I haven’t talked about is a caching layer. Almost any site of a reasonable size these days has a memcached layer (or something similar) in between the front-ends and the databases. One of the great things this for a push system is that because you’re dealing with things as they happen, all the objects that you typically need to access to build your update stream are almost always in cache – because they were just accessed.

So it turns out that not only does a system like this have almost no impact on your normal page serving time (because of all the de-coupling), it also ends up having very little impact on your databases, due most things being in cache.

Redis Numbers

• 1000 subscriptions
• 50K keys (queues) / 300 MB
• 100 qps – 8-core / 8GB 5% cpu

And a few numbers showing how Redis is performing as our little DIY queueing and subscription system. 1000 subscriptions for various different things takes around 50K keys, consuming 300 MB of ram, and at about 100 qps on an 8-core Linux box it’s barely ticking over.

3 Things: Cache, Tasks, & Queues

So putting all the pieces together it’s actually pretty simple, and relies upon things that you probably already have: a caching layer, some kind of asynchronous task system and rudimentary queueing system. Even if you don’t have all of these they’re pretty well-understood pieces and there are lots of open-source options to choose from: Just grab memcached, gearman, and redis and off you go.

We think you can go a long way to building the back-end of a decent push update system with just these simple pieces. So that’s the end of my portion of the talk, and now I’ll turn it over to Nolan to talk more about the front-end, and how to make it easier for a client to consume updates.

Why Push? And what’s in it for me.

So now that Neil’s explained the original reasoning and how we built a system for real-time push, the question is why, as an API consumer, would you want to use it which leads into how to get started with it.

Flickr API = Great Success

Tens of thousands of keys making hundreds of millions of calls per day. By any measure, Flickr’s tradtional pull-based API has been incredibly successful. Developers that have used our API range from Fortune 10 businesses to PhD students to hundreds of thousands of other developers that just want to do something fun with photos and the data around them.

I checked the numbers this wekend and and on average, we’ve got around 10,000 different API keys making hundreds of millions of calls per day. It’s easy. Just use your browser.

One major reason for its popularity is due to how easy it is to use. When you combine data that people want with an API that’s easy to use, you thousands and thousands of apps built with it.

There’s effectively no barrier to entry to get started with using Flickr’s API, if you have a browser, you can access every single API method. And if you want to do this programatically server-side , say a nightly-cronjob that fetches your own uploads, you can can just do a simple ‘curl’. Either way, our pull-based API is a one-liner: whether that line is a URL in your browser, or a one-line shell script.

We’ll curl it for you.

It’s even simpler than that, if you need it to be. Even if you don’t have a server, or don’t want to read our documentation, just use the API Explorer on Flickr. Every API method we publicly support is represented here, not just in documentation, but on an actual real-life form for you to build queries with. Just fill out the form, press enter, and we’ll curl it for you and spit out the results in pretty JSON or XML format.

One side benefit of this pull-based API is that it also makes debugging easy. You can run tons of API queries as fast as you can debug. This is huge when building a new system. If you formed the call correctly, you get data. If you messed up, modify, rinse, and repeat. Flickr is a visual news feed.

So back to “why push”…

I’m the kind of programmer that needs to have a fairly-focused work environment, but I also like to know what’s going on in the world at the same time. Twitter is a really great news-before-its-News service, but for me it’s a bit too distracting to have running on a screen next to my code.

I’ve recently hooked up my second monitor to show me two Flickr real-time feeds: the first of which is the Commons, our group of accounts belonging to museums and various historical archives, which is motivational for me, reminding me that Flickr is really something special with real history being stored on it.

The second feed is what I consider my glimpse into what’s going on around me, the stuff that immediately affects me. So I’ve hooked up a push feed of photos geotagged a half kilometer around my house and a half-kilometer around the office.

Around my house, it’s usually people taking pictures of the Painted Ladies in Alamo Square but then I saw this picture come across. When your entire neighborhood is made up of densely-packed, 50+ year old homes made of questionable building materials, fires are scary things. I then jumped on Twitter and found out there was a multiple-alarm fire just a few blocks from my house. Also, around the office, just this past week, I saw pictures from San Francisco’s chapter of the OccupyWallStreet protests. The news wasn’t on the ground yet but here I was seeing live and extremely relevant things to me. Like they say, a picture can often say more than a thousand words.

Subscribing to a real-time stream of photos from a specific location is about as hyperlocal as you can be. Maybe I wouldn’t follow the right people on Twitter, or I wouldn’t check the local news site until later that evening, but people do take pictures of important things and events and post them to the site, and it doesn’t get much more timely than real-time.

Push is the opposite of Pull. And other obvious facts.

Ok, so back to the technology. Push is a completely different animal than our pull-based APIs. Consuming a real-time push feed flips most things about our API on its head. For starters, you’ll need a full-fledged, always-on web server that is exposed to the public Internet. And running on this web server will be a software that is based on a protocol described in a spec-with-a-capital-S. This demands a lot more from the developer than just a one-line curl.

Pull: Hit this URL and read Push: Read this spec and wait

With our regular API, you could basically treat it as a function call, that is “ask for something, get something”. With Push, now you’re implementing full protocol where the data will come *eventually*. Now you’ll need to wade through pages of dense and sometimes ambiguous instructions, so you can wait for us to tell you that your friend uploa a picture. Another big but obvious difference between the push and pull API is that with Push, your application only gets what we send to you. There’s no looking back in time. And due to this read-spec, write-code, and wait and wait some more debug cycle, it takes awhile to get the endpoint working correctly.

Personally, I have access to the Flickr codebase and I still had questions about how to make my endpoint do the right thing and it took me a few good hours one morning and more than one cup of coffee get working right.

So, seriously, why Push?

Because it’s fantastic. Well now that I’ve made Push seem a little scary, seriously, why Push?

The main thing, I think, is that you we give you things as they happen. If you are building a site that uses Flickr data, would you rather us send y data when we get it, or set up a bunch of cronjobs that fetch possibly non-changing data? Even if you like your cronjobs, what happens when yo have 100 users? 1000 users? 1000000? I do want to make one note about what we mean by “real-time.” As you saw from Neil’s presentation, there are levels of queues and consumers, each having a non-zero delay. We get you the photo data as soon as we can, but this might be a few seconds after we receive up to a minute or two. We understood we were building a real-time feed for photos uploaded to a website, and not sending direction data to a nuclear warhead. It was okay if things were off by a few seconds.

Flickr Globe

Also, receiving real-time photo updates are great for certain types of problems. So, we had the press in the office recently and we wanted to build something cool to run on our big screen behind the developers as we worked. Using the real-time push feed, one of our front-ends, Phil Dokas, was able to build an interactive globe, that charted where on Earth our photos were being uploaded from in real-time.

The great thing about this is, not only was it really pretty looking visualization, but it impressed upon me the magnitude of Flickr being used. The site felt really organic: you could see it being used and growing. For me, it was one of those ‘pale blue dot’ moments that Carl Sagan talked about, where you get a sense of the bigger picture of what you work on. When you work deep on the backend of Flickr, it’s easy to forget that these are real people from literally everywhere on Earth, uploading things they want to remember.

And using the pushed data made this really easy to build. And for reference, this about 3 minutes worth of publically geotagged photos.

I want to use it, BUT… …it sounds hard.

So, we think this stuff is awesome, and we want you to use it but we do understand it’s fiddly. The spec is dense and debugging is painful. So we’re going to give you a headstart on the whole thing. We’ve written a tiny web server that handles all the subscription handshake stuff and the parsing of photo data and we put it on GitHub, and open sourced it.

flickr-conduit

https://github.com/mncaudill/flickr-conduit

So we’re introducing, flickr-conduit which is a simple server written in node.js that handles the subscription stuff, keeps tabs on when to unsubsc users, and then finally receives the Flickr posts.

There’s a lot of moving parts to setting up a push endpoint: handling everything from getting users authenticating your API key, telling you what topics they’re interested in, setting up the subscription between your callback and Flickr, and then figuring out how to get those events to the use when Flickr sends them to you.

In the conduit repository, I’ve included a server that implements the push protocol and then just fires off events in JavaScript when Flickr sends something. I’ve also included a PHP application that handles authenticating the user, letting them pick their topics, and then finally showing them the photos they come in as just a demo of what it can do.

I didn’t have time to get a slide up for it, but Tom Carden from Bloom, took the PHP application I made and got it ready so that you can easily de it to heroku. It’s great and if you want to run your own conduit-server demo, I’d advise checking it out.

To start with, Conduit is the piece that represents your callback endpoint. You tell Flickr that this is your endpoint and it handles the rest of the subscription handshake stuff and when Flickr posts data to you server, it goes to your conduit server.

Pub/Sub all the way down

After I finished building this, I realized that unintentionally I took the idea of pub/sub, that is having a many subscribers expressing interesting in many published streams, and kept that spirit up all the way through to the end of my little node.js server and explaining this model sheds some light on how the conduit server works.

First, as Neil described, Flickr runs a pub/sub service. With conduit, you can run your own pub/sub server that talks to our published streams. And inside of conduit, we let your application code subscribe to certain events that conduit itself receives Flickr post events, and publishes them to your app code to handle how you wish. So there are technically 3 different pub/sub hubs before your app code, which sounds a little complicated, but if you grasp the mental model of how pub/sub works, you understand the full system. Also, keeping the pub/sub levels decoupled from each other provides a lot of simplicity and flexibility.

So, this is a small digression about architecture that I stumbled across while building, that when you’re building systems in the small, it sometimes makes a lot of sense to model the smaller system after the larger system. This may not be groundbreaking for some, but I just found it really neat.

I know ‘curl’. Now, tell me about Push.

With Conduit, we’ve removed a fair amount of the fiddly bits so you can get to doing the fun parts later. I’m going to step you through each of the steps of the whole pubsub flow to show you what each part does.

First, subscribe the user to a topic.

/callback?sub=user1234-contacts-faves

First, you’ll want to subscribe the user to a topic. Topics can be anything from their own uploads, to their contacts uploads, to their contacts faves and even photos being geotagged at a specific place.

During this subscription, you’ll specify a callback. Once you have conduit running, the callback URL you give Flickr in your subscription should p to the conduit server. One of the tricky parts of handling a multi-user pub/sub server is that you can have many users attached that are waiting for many different strea When Flickr posts a piece of data to your server, you only know the URL that it came in on, so the callback URL needs to be significant so you ro this piece of data to right consumer. One way to handle this is just create a globally unique identifier for a particular subscription, and tuck that in a database and then use that to ma URL to a subscribed user.

Personally, I’m a fan of reducing the number of moving parts as much as possible so I just use something simple, like the example, and make th URL itself identifiable. So for example if user 1234 wants to subscribe to her contact’s faves, the URL could simply be /user1234-contacts-faves. As long as this callback URL creation algorithm is repeatable, it’s easy to have a common dictionary throughout your app of how to handle a particular subscription.

The part of the play where Flickr asks a question.

Almost immediately, after making the API call to subscribe the user to a topic, Flickr will respond with a ‘subscribe’ request. Here Flickr is basically asking, “Hi. We just received a subscription request for this callback. Did you send this? If so, repeat the magic password back to me.” All this is described in detail in the spec but you don’t need to know any of that. Conduit handles all of this for you.

Debugging: the waiting game.

As I mentioned earlier, debugging the server was the most boring and thus most frustrating part of building an endpoint. I’d write some code, and to see if did the right thing, I had to wait for on my subscribed events happen and get sent to me. One shortcut I found is that I could subscribe to my own faves, and then I go through a test account and fave photos, therefore forcing events. This sped things up quite a bit.

Someone’s at the door!

/callback?sub=user1234-contacts-faves maps to user1234-contacts-faves

When an event does occur, Flickr will post it to your callback URL. Conduit then takes this callback URL and through whatever method you decide, creates an event name for it and then passes into a internal structure, In my running example, the callback you see up there gets the event name of “user1234-contacts-faves”. Like I said, you could do this mapping of URL to event name however you like, but this is simple and works for me.

The emitter emits.

user1234-contacts-faves

Conduit exposes something that node.js calls an “emitter’. It’s basically a pub/sub structure itself. So when a post comes in and you decode the callback URL into an event name, you tell Conduit about it. Effectively you’re saying, “Hey I just received some data and it has this event name. Give this data to anyone that is interested in user1234- contacts-faves.”

Finally, fun photo data.

Your application code registers with this emitter what events it’s interested in and when they come in, they do something fun with them. Now I’m going to explain a bit about the mini-app that I bundled with the conduit server to shows how this works.

Real-time stream

Flickr + conduit + node.js + socket.io. I wanted the final product to be a simple webpage that when my subscriptions were posted to, I’d see them simply shown up on the page. I glued a few fun tools together and got something really interesting.

PHP is my engine.

Handles the PuSH subscribing and printing out my JS. This entire sample app could easily have been written in JS, but PHP is what I what do all day at work and my JavaScript is a little rusty and I wanted to get something up and running quickly.

I was able to grab former-Flickr engineer, now Etsy CTO, Kellan Elliot-McCrae’s flickr.simple.php’s library to handle the authentication of the use well as posting of subscription requests.

So the PHP app I built, first authenticates the user with Flickr and then presents the list of real-time topics he or she can subscribe to. The user checks the topics they’re interested in and then gets redirected to a screen where the images will show up as they come in. I create the unique callback URLs and at the same time dump these event names into the Javascript on the page so that my socket.io code can my node server, what events it’d like to receive when they come in.

Meanwhile, back on the server…

As mentioned earlier, Flickr will post the photo data to the conduit server to my callback URLs. This callback URL directly becomes the event name, mapping exactly to the what the browser told socket.io that it was interested in. When this event comes in, conduit hands it to the emitter, which then broadcasts the event back out. The server-end of the socket then receives this and pumps down into the browser’s open arms.

And back to the browser.

The browser then receives the photo data and inserts it into the page. That was a whole lot of engineering just to get an image up on the screen and we know it, but with flickr-conduit, you can skip most of the details and just write fun code. This actually turns out to be a great way to explore Flickr.

On my extra monitor at work, I’ve subscribed to my contacts’ faves and all the updates from our Commons area (which includes museums and various archives) to get a weird blend of 100 year old pictures and things my friends find interesting.

So, in summary, this is the why and the how of Flickr added a real-time push feed on the cheap. Since we know the spec is dense, and there are several moving parts, we also hope flickr-conduit helps out new developers that will hopefully le to love our new real-time feeds as much as they love our existing pull feeds.

Thanks.

Related links:

conduit: https://github.com/mncaudill/flickr-conduit
tom’s conduit links: https://github.com/RandomEtc/flickr-conduit-front, https://github.com/RandomEtc/flickr-conduit-back
pubsubhubbub spec: http://pubsubhubbub.googlecode.com/svn/trunk/pubsubhubbub-core-0.3.html
pua: http://pua.spum.org (http://www.aaronland.info/weblog/2011/05/07/fancy/)
nolan’s conduit: http://nolancaudill.com/projects/conduit/
flickr globe: http://nolancaudill.com/~pdokas/flobe/
kellan’s post: http://laughingmeme.org/2011/07/24/getting-started-with-flickr-real-time-apis-in-php/

neil’s flickr post: http://code.flickr.com/blog/2011/06/30/dont-be-so-pushy/

This is a guest post by Teruhisa Haruguchi: Frontend Engineer in the Mobile & Presentation Service group. Main developer on the Flickr Photo Session project. Recent graduate from Cornell University with a concentration in Computer Graphics. Home country Japan.

Keeping everyone connected around photos

This post dives down into the inner working of Photo Session, which enables user to have a real time photo sharing experience on multiple platforms.

In order to synchronize the viewing experience on different browsers, we need to have a mechanism to pass the state of one browser to other browsers. There are emerging technologies that would allow messages to be passed to the browser by opening a connection (ex. WebSockets), but our goal was to enable this feature for multiple platforms. To support this requirement, we chose CometD as our server technology which enable browsers on multiple platform to handle push notification. CometD also allowed us to setup a server cluster to handle large number of requests.

Bayeux Protocol (a.k.a. Push Notification thingy)

The mechanism in which messages are passed around is as follows. As different browsers join a Photo Session, we connect them to one of our servers and leave that connection open. This is referred to as long polling. When Client X posts a message, it is first collected by the server it is connected to, Server A. Server B has a long polling connection open via Oort to listen for any message that comes through Server A. OortClient is simply a server-side client that enables message passing between servers. It employs the same mechanism to push message from server to client. Once Server B receives the message via Oort, it will then relay the message down to Client Y by using the same mechanism used by Oort.

This lightweight transport mechanism allows us to pass messages around with minimum latency. However, this comes with few limitations. One, it does not guarantee message ordering. Client Y would not know the message order which client X submitted the them. To hide this limitation, we simply pick the very last message that is received for any client. There would be cases where users will be temporarily be out of sync when multiple messages are passed around at once, but it will be correctly updated with subsequent messages. Another limitation is the lack of guarantee on message delivery. Since we do not check for ACK during message delivery, the client X does not know if the message has been received by client Y. Again, most of the message delivery limitation is masked by checking for the last message.