This is a guest post by Jason Gabriele. Jason is a developer from the Yahoo! Mobile Platform/CPG team, who has worked in the mobile space for years developing on devices ranging from a Motorola V3 flip phone to modern devices like the iPhone. Over the past few months, his focus has been on the Photo Session project working mostly on the frontend UI.

Photo Session started as a Hack Day project by Iain Huxley. The idea came from the desire to share and discuss photos live with friends, like being able to share photos from a recent vacation with his mother in Australia. He then created a demo which could provide realtime photo viewing synced across multiple users located anywhere in the world. At a Yahoo! Hack Day event, Photo Session was selected as a winner and Iain was awarded a Yahoo-branded beach towel. Later, the project evolved into a Flickr photo sharing project with chat provided by Yahoo! Messenger.

Photo Session

Photo Session allows you to share photos in realtime with your friends. You can slide through photos, draw on them, zoom in and out, and send messages with others present in the session using the built-in chat window. Users without Flickr accounts can still join as guests but can’t use certain features like the drawing tool or chat. Given the requirement to allow guest participants, we had to limit the photos to publicly-accessible photos only for the first release.

The Basics

• Create a Photo Session by visiting a set or photo stream and clicking on “Start a Photo Session” under the share menu
• Once in the photo session, advance through photos by clicking and dragging or using the arrow buttons (desktop only)
• Start drawing mode by clicking the pencil icon (must be logged into Flickr) and begin drawing on the photo. Your drawing will be shared with others in the session but will be cleared as soon as drawing mode is disabled
• Zoom in by using the +/- buttons or using the scroll wheel on the desktop, or by pinch-zooming on the iPhone and iPad
• You can view details about the photo using the Information icon in the lower-left on desktop browsers, or by clicking the “i” icon in the lower-right on the iPhone
• You can leave the group at any time and browse on your own using the “Browse on your own” button (not available on the iPhone). Return to the group session by simply clicking the button again
  
• You can hide the Photo Session toolbars by doing a single click on the current photo. Click again to bring them back

Browser Support

Since the primary use of Photo Session is sharing photos, we wanted to make sure the experience felt fast and responsive. This meant we would need to use features like CSS transforms and other CSS3 properties only available in the latest browsers. We also needed the rendering to be fast enough to support users rapidly advancing through photos. We decided to support Chrome, Firefox, IE9, Safari and iOS for our first launch. This would also simplify the QA process. We plan on adding support for Android in the future.

Future

Photo Session is in preview mode, so we may change features in the future depending on how people use them. We already have plans for some new features, but we would still love to hear from you so please provide feedback in the Flickr forums. We hope Photo Session makes sharing photos with your family and friends a more social and interactive experience!

When we began prototyping geofences, our goal was to encourage people to geotag more photos without having to be too concerned about privacy.

Introducing people to geofences

The term “geofence” may sound complicated, but our implementation is quite simple. A geofence is a user-defined boundary around a specific area on a map. We decided to keep the creation and editing process similar to geotagging on the photo page. We understand from developing the photo page map that it is important to provide a way to search for a location as well as simply drop something in the right place on the map. The geofence is represented by a selector-circle on a modal map panel with simple edit controls on the side.

In previous map interfaces we used canvas elements to generate scalable components like our new selector-circle. One example of this would be the location circle in photos nearby. Our resident browser performance task-master, Scott Schiller, decided this time to compare the performance of continuing to use canvas with that of the now widely supported css border-radius feature. Using border-radius made for a smoother performing interface in our tests against the early prototypes of geofences. We loved the result.

The only noticeable downside to our current implementation is that it attaches a click handler to the entire element which is displayed as a selector-circle with css. This means that the actual click-able area is not a circle at all, it is a square.

We are currently working to find the most elegant way to get around this issue. The code below shows one possible solution:

var root_node   = Y.one('#circle_container_node'),
    circle_node = root_node.one('.selector-circle');

function getCenterPoint() {
	var half_node_width = circle_node.get('offsetWidth')/2;
	
	return [(circle_node.getX()+half_node_width),(circle_node.getY()+half_node_width)];
}

function getDistanceFromCentroid(xy) {
	
	var centroid_point = getCenterPoint();

	return Math.sqrt(Math.round(Math.pow(centroid_point[0]-xy[0], 2.0)) + Math.round(Math.pow(centroid_point[1]-xy[1], 2.0)));
}

circle_node.on('click',function(e){
	
	if(getDistanceFromCentroid([e.pageX,e.pageY]) < half_node_width) {
		//handle
	}
	
});

Sustainably supporting a sill relevant, but older browser

Don’t you still have a fair amount of users on IE7, which does not support border radius? Yes, we do, and thanks for pointing that out so eloquently. We decided to use the CSS3PIE framework which uses clever tricks to make IE7 support modern css features like border-radius, box-shadow, border-image, multiple background images and linear-gradient as background image. What is awesome about using CSS3PIE, is that it allows us to give IE7 users a nearly identical experience as that enjoyed by modern browser users without a lot of branched code.

.selector-circle {
	border:solid 8px rgba(68, 68, 68, .4);
	background-color:rgba(0, 100, 220, .5);
	border-radius: 999px;
        /*Neeed for CSSPIE to support rgba*/
        -pie-background:rgba(0, 100, 220, .5);
}

Relative sizing

We determined a sensible size range of 50 to 10,000 meters for a geofence, then dreamt up a few ways to allow folks to select an appropriate size within this range on the map. A lot of people (myself included) are not very good at judging distance in kilometers or miles without a visual example. We establish a connection between changing to the radius dropdown menu, the size of the selector-circle, and the zoom level of the map by making sure that a change to one is reflected in all of them.

Something had to be done to keep the geofence visible on the map even if the selector-circle has become too small to be seen in a selected zoom level. The selector-circle was given a center-point which would not change with changes of radius or zoom. It is always visible even if the surrounding fence is too small

Geofences are privacy defaults, not filters

A geofence is only applied as a default to at the time of uploading or geotagging a photo. It will not, on it’s own, affect photos already in that area. We created a process which one could follow to apply geofences to existing photos as a convenience. Nolan Caudill goes into detail about the code behind this process in his post from last week. Now, imagine if you did not realize that a large number of your photos, which you intended to stay public were in an area covered by a geofence. After applying the geofence settings, they would be suddenly hidden from the map. Changing all of those photos individually or even as a batch in the organizr would be, what we in the industry call, a massive honking bummer. A final step was added to the end of the creation and edit of geofences to show which photos could be affected. We display a list of potentially affected photos with a color-coded indicators of what their privacy will be after the geofence settings are applied to help people to make as informed a choice as possible.

In most cases you will get an idea of which photos will be affected right away, but areas with lots of photos can be carefully assessed as well.

Showing all of the fences at once

In the prototype phase, geofences were displayed in a list and the only way to see them on a map was individually. We decided to create a map which displays all of the circles, in one place, over the list, to make it easier to see how fences might interact. One great example is that if two fences of similar privacy value (like family and friends for instance) overlap, the intersecting area is considered private. With this in mind, it is nice to at least be able to see on the map where your fences cross. In some later release, we would love to show an appropriate red color-code in these intersecting areas.

Most people will create geofences with names like Home, Office, and School. For most people, these places are pretty close to each other. Some people might have places which they want to protect and are further apart. It is important for the map flexible enough to include the entire globe if you were to have one or two geofences abroad. We use a center-point to represent geofences which are too small to be seen, and we calculate the best zoom and center of the map when we load of the geo preferences page.

World view

Neighborhood view

We launched geofences earlier this week. I want to give you a glimpse into some of the code that runs the feature.

Geofences are defined by three things: a coordinate pair, a radius, and a privacy level. We store the coordinates (ie, decimal latitude/longitude) in our database as integers, the radius as an integer representing meters, and the privacy level as an integer that maps to a constant in our code.

When an image is geotagged, we have to determine which geofences the point falls within. We do some complex privacy calculations that Trevor talks about in his introductory post.

First, we fetch all of your geofences from the database and then loop through each one, determining if the photo falls in any geofences. We limit the number of geofences you can create to ten, so running a point through 10 calculations is not that expensive.

We use the great-circle distance formula to figure out how far the image is from the center of a geofence. If this distance is less than the radius of the geofence, the geofence applies.

MySQL is a Great Hammer

When you create a geofence and want to apply it to existing photos, the backfill is a more involved process. In order to grab just the photos we care about for a geofence, we have to limit the number of photos that we select due to performance.

As mentioned, we store the latitude and longitude as integers in MySQL. Since radial queries are next to impossible with this setup, we do a first pass with a bounding box formula that encompasses the geofence and then use our great-circle distance formula to cull the photos that don’t fall in the geofence.

The bounding box formula we use takes into consideration geographic gotchas, like the poles and the 180 degree discontinuity (ie, the International Date Line). Since a geofence could overlap the International Date Line, we have to modify the DB query in doing the longitude part of our query since doing a simple BETWEEN won’t work. When the bounding box doesn’t cross the IDL, we can do “SELECT id FROM Table WHERE longitude BETWEEN $lon1 and $lon2” and when it does we do a “SELECT id FROM Table WHERE longitude < $west_lon AND longitude > $east_lon”.

In short, we use a query that MySQL is a good at it to limit our initial dataset and then use a little post-processesing of the data to get the points we actually care about for the geofence.

After we have this bundle of photos that the geofence backfill applies to, we then run our normal geoprivacy calculations, lowering the privacy if the combination of fences calls for it.

Concurrency, or the world keeps turning

The introduction of the geofences backfill pane also introduced some concurrency issues, which were fun to deal with.

The contract that we wanted to create for the user was that the calculated privacy that they saw in the preview pane for the backfill would exactly match the result of running the backfill. This seemed intuitive but the implementation was tricky.

There are two things that can affect a photo’s geoprivacy: the user’s default privacy and the geofences that exist. These things can change through time, that is between the time that a user hits ‘apply’ on the preview pane and when the backfill finishes running.

Whenever you deal with mutable state through the lifetime of a process, you have to change how you treat this state. Who can modify it? What can they modify? And what processes see this updated object?

The easiest way to deal with mutable state is to make it immutable. We do this by storing the state of the user’s geoprivacy world (that is, the default geoprivacy and the geofences) alongside the backfill task, so that when the task runs it uses this state instead of querying the DB, which is mutable, possibly having changed since the user hit the ‘apply’ button.

Storing this state ensures that regardless of how the user modifies their geoprivacy settings, the result of the backfill will match exactly what they saw in the preview pane, providing a consistent view of the world.

We also added the restriction of only allowing one backfill task per user to be running at a time, which simplified our bookeeping, our mental model of the problem, and the user’s expecation of what happens.

Lessons Learned

Geo, by itself, is not easy. The math is complex (especially for someone like me a few years out of school). Privacy is even harder. Since the project evolved quite a bit while we were building it, having a large amount of automated testing around the privacy rules gave us the confidence that we could go forward with new privacy demands without introducing bad stuff into code that was “done”. We learned to give ourselves plenty of time to get all this stuff right, since geo has so many edge cases and violating privacy is an absolute no-no.

During the project, we had to keep balance between our users’ privacy and their expectations, all while keeping a complex and new feature understandable and even fun to play with. The only way we could do this was being open and honest between engineering, design and the product team about what we were building. This was a total team effort from Flickr, and we’re very proud of the end result and the control that it gives our users over their presence on the Internet.

Also, if you’re interested in working on fun projects like this one, we’re hiring!

A little while ago we released some new API methods that enabled real-time updates for new photo uploads to be pushed out via a PubSub-like subscription system. Initially you could only subscribe to photos from your contacts and favorites from your contacts. Which was pretty neat, but that barely scratches the surface of stuff that happens on Flickr that people might be interested in. So we added some more stuff to subscribe to. Calling flickr.push.getTopics now gets you:

<rsp stat="ok">
  <topics>
    <topic name="contacts_photos" display_name="photos from your contacts" />
    <topic name="contacts_faves" display_name="favorites from your contacts" />
    <topic name="photos_of_contacts" display_name="photos of your contacts" />
    <topic name="photos_of_me" display_name="photos of you" />
    <topic name="my_photos" display_name="your photos" />
    <topic name="my_faves" display_name="your favorites" />
    <topic name="geo" display_name="photos from an area (geo)" />
    <topic name="commons" display_name="photos from the Flickr Commons" />
    <topic name="tags" display_name="photos with a tag (or tags)" />
  </topics>
</rsp>

The details for the extra arguments required by the new topic types are part of the flickr.push.subscribe API method documentation.

The my_photos and my_faves topic types are exactly like contacts_photos and contacts_faves, just scoped to your account. The photos_of_me and photos_of_contacts topic types create subscriptions that receive events when you or your contacts are tagged in a photo. The really interesting ones though are the next 3: commons, geo, and tags.

The Commons

One of the great new subscription types is for photos from the Flickr Commons. Set the topic type to commons, and set the nsids argument to a comma-separated list of NSIDs of Commons institutions you’re interested in (get them by calling flickr.push.getInstitutions) or just leave nsids empty to get all uploads and updates from the Commons.

Geo Subscriptions

With the geo topic type you can subscribe to photos from a particular area, specified as either a set of WOE IDs (also, here), a set of Flickr Place IDs, or by a point and a radius. The radial query is the obvious choice for creating subscriptions that aren’t well-known areas, or say, user-specified by dragging a circle on a map. WOE IDs on the other hand are incredibly useful since they represent many well-known geographic features. For example, London, England: WOE ID 44418.

Or, if you’re like me and completely mad about aviation and anything that flies, there are convenient WOE IDs for airports. Go somewhere like this, grab a bunch of airport codes for interesting places, use the Y! GeoPlanet APIs or the flickr.places.find method) to resolve the airport codes into WOE IDs and put them into a comma-separated list in the woe_ids argument of flickr.push.subscribe and watch the aviation photos roll in. For example:

• Hong Kong International, China (HKG): 24875607
• Innsbruck Airport, Austria (INN): 12510823
• Keflavik International Airport, Iceland (KEF): 12513445
• Le Bourget Airport, Paris (LBG): 22137770
• Kuala Lumpur International Airport, Malaysia (KUL): 28752278
• etc.

Tags

We’ve also added a very basic tag subscription type. With the topic type set to tags, you can provide a comma-separated list of tag names in the tags argument of flickr.push.subscribe and receive uploads and updates from photos containing any (i.e. OR mode) of those tags. Try something like kitten,cat (We’re not responsible for your bandwidth bill). The tags you specify should be all lower-case and not contain any spaces, so if you want to match something like “Justin Bieber” you’ll need to specify “justinbieber”. Coming soon: machine tags! robots:zomg=yes…

Warning: Code Alert!

In case you missed it, last month Kellan devoted a little Sunday-morning hack time to whipping up a little example of how to get started with the Flickr real-time APIs. It’s got a little bit about authentication, how to subscribe, processing a feed, and even some PHP. Thanks Kellan!

Are you a developer with a great idea for a photo hack?

Are you going to be in New York on August 20th and 21st?

If so, join us at the Photo Hack Day! Our very own Paul Mison will be on hand to give a presentation on the Flickr API and to answer any questions you have while hacking. The winner will have their hack featured on the NASDAQ billboard in Times Square (among other prizes), and all participants will get a year of free Flickr Pro. Sign up to reserve your spot at the event, which will be held at General Assembly in Manhattan.

If you’re participating in the Hack Day, Flickr engineers will be available to help on IRC: #flickrapi on chat.us.freenode.net. You can also go to developer.flickr.com for more information on the API and developing with Flickr photos. We will be announcing some new and improved APIs in advance of the Hack Day, so watch this space!

Bye bye Kodachrome by e_pics

Recently we released the Flickr Lightbox for iPad, iPhone and Android. We managed to create a pretty responsive interface. It took us a while to get there, and we learned a lot doing it. In this post we’d like to share a few useful key lessons we took away from the project.

Get Your Viewport Tag Right

The viewport tag is actually not that well understood, but you have to get it right, or everything is just going to be completely confusing. For the lightbox it looks like this:

<meta name="viewport" content="width=device-width; initial-scale=1.0; maximum-scale=1.0; user-scalable=0;">

This is the way we made sure that we could reliably position elements by pixel, and properly handle device rotation without fear. Other places on the web describe the details of how this tag works, but for us the most important was “maximum-scale=1”. This is because when you rotate the iphone it scales the content normally, unless you specify a max scale. Positioning becomes very difficult when the phone is scaling the interface you carefully crafted to fit on the device.

Simplify the DOM

On mobile WebKit you can use CSS for much, much more than you can on the desktop where you need to support bad browsers. This makes it very easy to clean the DOM down to the minimum. We did this on the lightbox by starting from scratch (the DOM for the desktop lightbox is quite complex to support a cross-browser layout.) We also actually nuke the existing DOM when the lightbox opens.

Keep It Responsive

I can not emphasize this enough. Of course, any UI must be responsive. But a touch UI doubly so. Apple clearly “got” this in the development of the first iPhone. Possibly because there is no “click” sound when you interact with the device it is very hard to tell if the device has registered your interaction without immediate feedback. Further, if there is even a tiny delay in how the device responds to touch interaction it feels clunky. Much clunkier than a slightly glitchy desktop UI. One of the things keep us from supporting the desktop lightbox on touch devices was that it felt very slow and clunky. Once we figured out that this was a matter of percieved responsiveness it was pretty clear where we needed to focus: percieved performance.

From a user experience standpoint this means that any interaction needs to give the user feedback. In the lightbox this means that when the user swipes the photo always moves with their finger. When the user hits the end of the photos, rather than not responding anymore, the photo continues to move but snaps back (with CSS transitions) to the last point.

Of course, for this interaction to work it needs to be very fast, any delay feels very awkward. So from an optimization standpoint we went after the performance of the swipe animation above everything else. The first thing to do is to use 3d CSS transforms. All the touch devices have 3d acceleration hardware which makes it possible to move the photo much faster than with just the CPU. The additional benefit of course is that when using transforms the animation does not block JS execution at all.

The code looks something like this:

    distance = e.touches[0].pageX - startX;
    absDistance = Math.abs(distance);
    direction = (absDistance === distance) ? 1 : -1;

    if (absDistance > 2) {
        thisPosition.setStyle('transform', 'translate3d('+distance+'px, 0px, 0px)');
    }

When we first tested this, however, we found performance quite disappointing. Another team pointed us to a trick: don’t use <img> tags. We got a huge performance boost when using <div> tags with the photo as a background image.

The next thing we noticed was a slight but perceptible delay between the touch event and the movement of the photo element. After some profiling we found that the YUI event abstraction was actually taking enough time to be perceptible, so we switched to native event handling. Which lead us to further optimization along the same lines: do as little as possible. Most things you do in JS (with some special exceptions) are blocking. So any work you do while touch events are being handled necessarily delays the feedback the user needs to know that their touches are being registered.

We went through the code path that happened during touch events. Anything that could wait until “touchend” was deferred there.

The last problem to solve was that the browser would crash with more than twenty or so slides loaded. It seems that the iPhone browser dies very quickly when it runs out of memory, especially when using 3d acceleration. So we implemented a simple garbage collector for the slide nodes:

    //remove all slides more than 10 positions away
    function pruneSlideNodes() {
        if (inTransition || moving) {
            if (pruneHandle && pruneHandle.cancel) {
                pruneHandle.cancel();
            }
            pruneHandle = Y.later(500, this, pruneSlideNodes); //wait
            return;
        }
        
        positionManager.each(function (value, key) {
            if ((Math.abs(key - parseInt(currentPosition,10)) > 10) && value.id) {
                
                Y.one('#' + value.id).remove();
                delete(value.id);
            }
            
        }, this);
    }

Final note:

Moving to YUI 3 has been huge for us, even on mobile tasks. The mobile lightbox takes advantage of several modules created for the desktop, most importantly a “model” module we created to manage what we call photo “contexts”. This meant that the logic of displaying slides is the same in all places, the view/controller code was all that we needed to create for this. Which also means that this logic exists in just one file.

When I poke my head above my monitor at FlickrHQ, I can see a lot of headphones. Ever wonder what inspires us to write code (besides fried chicken sandwiches and cocktails)? Well I did a little snooping in the playlists of a few Flickreenos:

Oh, and I suppose I should share mine too:

•  Blueprint Car Crash – Gun Moll

•  Little Dragon – Come Home

•  Angus & Julia Stone – Black Crow

•  Plants and Animals – Celebration

•  of Montreal – Coquet Coquette

Find even more of what inspires us on our FlickrHQ Last.Fm Group

Photos by waferbaby