Living In the Donut Hole

video by origami madness

A long long time ago (2005) in a galaxy far far way (Vancouver) when we joined Yahoo! and moved FlickrHQ to the Bay Area all but one or two members of the team lived within ten square blocks of each other in San Francisco’s Mission District.

John Allspaw, a long-time resident of the Mission used to regale us with stories of one of the neighbourhood’s notable quirks commonly referred to as the “donut hole”: The rest of the city could be covered in fog, or raining, but the moment you crossed over in the Mission the sky would open up and the entire neighbourhood would be bathed in sunshine.

When John and George Oates and I used to car pool between the city and the offices in Sunnyvale, we would drive up and down highway 280 and sure enough as you approached the city, at the end of the day, you would drive into an enormous blanket of fog the moment we passed the airport in Millbrae. And as soon as we’d pulled off the San Jose exit there would be an open stretch of clear sky all the way to Civic Center where it would stop again just as suddenly.

Some mornings, when I look out my kitchen window at the clouds hanging over Diamond Heights I like to pretend I can see the curvature of the inside of the donut hole itself. I was reminded of all this the other morning when I was generating some visualizations based on the shapefiles that are derived from the almost 100 million geotagged photos on Flickr.

Paris (Ile-de-France)

The larger, blue, contour is the “shape” of the city of Paris (or WOE ID 615702) according to Flickr. The smaller white contours are the child neighbourhoods of that WOE ID with public, geotagged photos. So, what’s going on then?

The first outline maps roughly to the extremities of the RER, the communter train that services Paris and the surrounding suburbs. This is a fairly accurate representation of the “greater metropolitain” area of Paris. Metropolitain areas, increasingly common in both popular folklore and government administrivia as more and more people shift from rural to urban living , are noticeably lacking from the Flickr hierarchy of place types and a subject probably best left for another blog post.

Untitled #1202166719

The rest, taken as a whole, follow closer to the shape of the old city gates that most people think of when asked to imagine Paris. Which one is right? Well, both obviously!

Cities long ago stopped being defined by the walls that surround(ed) them. There is probably no better place in the world to see this than Barcelona which first burst out of its Old City with the construction of the Eixample at the end of the 19th century and then again, after the wars, pushed further out towards the hills and rivers that surround it.

There are lots of reasons to criticize urban sprawl as a phenomenon but sprawl, too, is still made of people who over time inherit, share and shape the history and geography they live in. Whether it’s Paris, Los Angeles, William Gibson’s dystopic “Boston-Atlanta Metropolitain Axis” (BAMA) or the San Francisco “Bay Area” they all encompass wildly different communities who, in spite of the grievances harboured towards one another, often feel as much of a connection to the larger whole as they do to whatever neighbourhood, suburb or village they spend their days and nights in.

ORY

That’s one reason I think it’s so interesting to look at the shape of cities and see how they spill out beyond the boundaries of traditional maps and travel guides. In the example above the shape for Paris completely engulfs the commune of Orly, 20 kilometers to the South of central Paris, which makes a certain amount of sense.

It also contains Orly airport which isn’t that notable except that we treat airports as though they were cities in their own right because the realities of contemporary travel mean that airports have evolved from being simple gateways to captial-P places with their own culture, norms and gravity. So, now you have cities contained within cities which most people would tell you are just neighbourhoods.

We’re recently finished rendering the second batch of shapefiles and looking ahead I am wondering whether we should also be rendering shapes based on the relationship of one place to another. Rendering the shape of the child places for a city or a country (you can do this using the handy, if awkwardly named, flickr.places.getChildrenWithPhotosPublic API method) would allow you to see a city’s “center” but also provide a way to filter out parts of a shape with low Earthiness (aka water).

Americonia

The issue is not to prevent, or correct, shapes that provide a “false” view because I don’t think they do. As Schuyler observed, while we were getting all this stuff to work in the first place, and testing the neighbourhoods that meet San Francisco Bay they are really the shapes of people looking at the city. They are each different, but the same.

But maybe we should also map the neighbourhoods that aren’t considered the immediate children of a city but which overlap its boundaries. What if you could call an API method to return the list or the shape of a place’s “cousins”? What could that tell us about a place?

San Francisco

What does all of this have to do with donuts? Nothing really, but it’s a nice way to think about the problem and since we have a long and storied tradition of silly names for projects I imagine this one will stick too.

There are no fixed dates yet for when, or whether, any of this will make its way in to the API but quite a lot of it could be done with API methods already available today. One change we have made is to add a new flickr.places.getShapeHistory API method which include pointers to all the shapefiles that have been rendered for a place. I have dim and distant memories of possible reasons why not to do this, in the past, but the exercise in making donut shapes makes me think I was wrong. The more data and “nubby bits” that people have to work with the more interesting it will be for everyone.

Enjoy!

Front-end Performance: Doing More With Less

In a recent talk on ajax performance, Douglas Crockford explained the practice of “streamlining” code as including algorithm replacement, work avoidance and code removal.

Applying this to page load time and performance, browser “work” can be avoided by reducing the number of HTTP requests made. In the most common case this can mean eliminating images entirely from a design, or reducing multiple image references to a single sprite image where possible. Shaving off image requests is a wonderful way to incrementally improve both perceived and measured performance.

The use of sprites alone is not a new technique, but the approach taken in this case involves a low-cost, low-risk update to a common component based on legacy code, reduces the number of HTTP requests and simultaneously improves the UI – all good things!

Legacy code: The good and the bad

Flickr has UI components and widgets which have worked well for years, but can also benefit from newer development techniques such as sprite-based images. Rounded-corner dialogs including alpha-transparent drop-shadows are two prime examples, obvious subjects for optimization. There is some risk in modifying code which may be used in hundreds of places, so minimizing changes to the HTML structure is important.

In this particular case, legacy table-based code was used to do layout for a rounded-corner dialog, and a separate table element was used to position a drop-shadow underneath it. Both elements used separate images for each edge of the box, resulting in eight HTTP requests for the dialog and another eight for the shadow.

Screenshot: Saving HTTP requests with CSS and sprites

Dialog and drop shadow performance optimizations (border-radius + CSS sprites) save up to 15 HTTP requests, in this example (adding a note to a photo on Flickr.)

Retrofitting legacy code in a low-risk way

In modifying the drop-shadow code while maintaining backwards-compatibility, the src property of each corner image was set to a transparent 1×1 GIF image; width and height were already specified on these elements, so the layout is retained as if the original image were being used. Additionally, a CSS class applied the sprite as a background image and a second class provided the background-position offset.

For the rounded corner dialogs, eight image requests were completely eliminated for browsers supporting CSS’ border-radius property. Additionally, the rounded corners are now anti-aliased in several browsers – a further improvement over the old GIF-based “jaggies.”

While not as “clean” as a complete code rewrite, this incremental approach improved performance and did so with minimal chance of introducing bugs in a global component.

Related resources

Some helpful, free tools were used in creating both the sprite and CSS for these performance tweaks.

  • Smushit.com
    Web-based image optimizer, shave (potentially many) bytes off GIF/PNG images
  • CSS sprite generator
    Upload a .zip of images, set some parameters, and get a sprite with the CSS generated automagically! Quite handy.

Pancake Friday

This one goes out to all the other developers that drag themselves into the office on the second day of the year just before a weekend (and those online over the holidays, just incase).

For the record we are going to Rock 2009 and talk about it here, but first time to grab breakfast.

Photo from keithdavisyoung.

Web Ops Visualizations Group on Flickr

Like lots of operations people, we’re quite addicted to data pr0n here at Flickr. We’ve got graphs for pretty much everything, and add graphs all of the time. We’ve blogged about some of how and why we do it.

One thing we’re in the habit of is screenshotting these graphs when things go wrong, right, or indifferent, and adding them to a group on Flickr.

Xmas lights on Front Page

I’ve decided to make a public group for these sort of screenshots, for anyone to contribute to:

http://flickr.com/groups/webopsviz/

You should realize before posting anything here, that you might want to think about if you want everyone in the world to see what you’ve got. I’ve made a quick FAQ on the groups page, but I’ll repeat it here:

Q: What is this?
A: This group is for sharing visualizations of web operations metrics. For the most part, this means graphs of systems and application metrics, from software like ganglia, cacti, hyperic, etc.

Q:Who gets to see this?
A: This is a semi-public group, so don’t post anything you don’t want others to see.

For now, it’ll be for members-only to post and view. Ideally, I think it’d be great to share some of these things publicly.

Q: What’s interesting to post here?
A: Spikes, dips, patterns. Things with colors. Shiny things. Donuts. Ponies.

Q: My company will fire me if I show our metrics!
A: Don’t be dense, and post your pageview, revenue, or other super-secret stuff that you think would be sensitive. Your mileage may vary.

So: you’ve got something to brag about? How many requests per second can your awesome new solid-state-disk database do? You got spikes? Post them!

Machine Tag Hierarchies

Untitled Compass #1228516024

something:somethingelse=somethingspecific

With apologies to Jeremy Keith

If you’re not already familiar with machine tags the easiest way to think of them is being like a plain old tag but with a special syntax that allows users to define additional structured data about that tag. If you’d like to know more, the best place to start is the official announcement we made about machine tags in the Flickr API group.

If you want to know even more, still, take a look at:

Okay! Now that everyone is feeling warm and fuzzy about machine tags: We’ve added (4) new API methods for browsing the hierarchies of machine tags added to photos on the site. These are aggregate rollups of all the unique namespaces, predicates, values and pairs for public photos with machine tags.

For example, lots of people have added exif: related machine tags to their photos but there hasn’t been a way to know what kind of EXIF data has been added: exif:model? exif:focal_length? exif:tunablaster? Or what about all the planespotters who have been diligently adding machine tags to their photos using the aero namespace: What are the predicates that they’re tagging their photos with?

Those are the sorts of things these methods are designed to help you find. Sort of like wildcard URLs but for metadata instead of photos. Uh, sort of.

Untitled Future #1223665161

Anyway, the new methods are:

flickr.machinetags.getNamespaces

This returns a list of all the unique namespaces, optionally bracketed by a specific predicate. For example, these are all the namespaces that have an airport predicate:

# ?method=flickr.machinetags.getNamespaces&predicate=airport

<namespaces predicate="airport" page="1" total="2" perpage="500" pages="1">
	<namespace usage="1931" predicates="1">aero</namespace>
	<namespace usage="3" predicates="1">geo</namespace>
</namespaces>
		

flickr.machinetags.getPredicates

Like the getNamespaces method this returns a list of all the unique predicates, optionally bracketed by a specific namespace. For example, these are all the predicates that use the dopplr namespace:

			
# ?method=flickr.machinetags.getPredicates&predicate=dopplr

<predicates namespace="dopplr" page="1" total="4" perpage="500" pages="1">
	<predicate usage="4392" namespaces="1">tagged</predicate>
	<predicate usage="1" namespaces="1">traveller</predicate>
	<predicate usage="7780" namespaces="1">trip</predicate>
	<predicate usage="4269" namespaces="1">woeid</predicate>
</predicates>
		

flickr.machinetags.getValues

At this point, the pattern should be pretty straightforward. This method returns all the unique values for a specific namespace/predicate pair. For example, these are some of the values associated with the aero:tail machine tag (yes, really, airplane tail models!):

# ?method=flickr.machinetags.getValues&namespace=aero&predicate=tail
		
<values namespace="aero" predicate="tail" page="1" total="1159" 
	perpage="500" pages="3">
	<value usage="1">01-0041</value>
	<value usage="1">164993</value>
	<value usage="2">26000</value>
	<value usage="1">4k-az01</value>
	<value usage="1">4l-tgl</value>
	<value usage="1">4r-ade</value>
	<!-- and so on... -->
</values>
		

flickr.machinetags.getPairs

Finally, the getPairs method returns the list of unique namespace/predicate pairs optionally filtered by namespace or predicate.

Rather than including yet-another giant blob of XML, here’s a pretty picture of the metro stations in Munich instead:

ubahn

A few things to note

Certain namespace/predicate pairs have been special-cased to return a single value. As of this writing they are:

  • geo:lat (and variations)
  • geo:lon (and variations)
  • file:name
  • file:path
  • anything:md5

If people have particular reasons for needing or wanting these we’re open to the idea but otherwise the cost of storing all the variations and the dubious uses for returning them in the first place made us decide to exclude them.

Now what?

Some people take sandcastles pretty seriously...

photo by Daveybot

Well, that’s what we’re hoping you’ll tell us. Machine tags have been chugging away quietly since we announced them almost two years ago and despite being a bit nerd-tastic and awkward to explain we’ve been thrilled to see how people have been finding their own use for them.

And the list goes on.

The trick with machine tags has always been to make them both invisible (or at least barely visible) to those people who don’t care about them but also as easy as tags to pick up and use for those people who do or who wonder whether they might be the tool they were looking for. One thing we didn’t do very well, though, until the release of the machine tag hierarchy APIs was give people a way to learn about machine tags. The only way to find out which machine tags people were using was to hop-scotch your way around people’s photostreams or to be part of a larger community having a discussion about which tags to use. Oops.

Which is why it was extra-fantastic when a few short days after we announced the machine tag hierarchy methods on the API list, the ever prolific and awesome Paul Mison wrote back and said:

The obvious thing to build on top of these … is some sort of graphical machine tag browser, a bit like the Mac OS X / iPod column view browser. So I did.

http://husk.org/code/machine-tag-browser.html

This is entirely self-contained in one file (except for loading jQuery from Google and (cough) the pulser from Flickr). It uses JavaScript to get a full list of namespaces, giving you the option to drill down into predicates and the values available for that namespace/predicate pair.

We’re hoping that this provides a little more raw material to play with and maybe find some magic and that you’ll tell us what comes next.

Yay!

Oh yeah, the actual API methods

Enjoy!

In the coming weeks we’ll also try to gather most of the blog posts and other writings about machine tags and put them with the rest of the API documentation.

[change log] Add as contact link

Adding a contact from the Buddy Icon dropdown
Change happens. On Flickr it happens quite a lot, although it tends to be just the large stuff that gets the coverage, see Improved Contacts Management over on the main Flickr blog for example.

I’ve started [change log] posts as an attempt to highlight some of the smaller changes that go on.

And… the new Add as a contact link seems like a good start.

For quite a while now you could add someone from any page that featured their buddy icon. Hovering over it and clicking the down arrow opens up a whole selection of options, including adding them as a contact. It’s still a great way of getting to pages about that user or adding them as a contact from places such as group discussions.

As of last week Eric dropped a slightly more obvious Add so-and-so as a contact link onto the photostream page.

Add as contact link

With the extra bonus of also saying what their current contact status is should you ever decided to change it.

Sometimes it’s the little things that make a world of difference, so yay Eric!

Lightweight Layout Management in Flash

Resizable apps are groovy. Users want content to fill the screen, and as the variance in screen resolutions has grown this has become more and more important. To make this happen you can either write manual positioning code to place and size items on window resize, or you can define a hierarchy, some properties on elements of the hierarchy and have a layout engine do the work for you. Up until a few years ago, there were no good pre-built solutions to do this in Flash so the only option was to roll your own framework. Nowadays there are several floating around, including this one from the flash platform team at Yahoo!, and of course there is the Flex/MXML route.

XML based UI markup seems to be all the rage these days, but what I found personally through my experience building the jumpcut tools is that this “responsible” approach has drawbacks. A hierarchy of HBoxes and VBoxes can be a lot of overhead for simple projects, and can become unwieldy and rigid when you are trying to develop something quickly. If you are doing anything weird in your UI (overlapping elements, changing things at runtime) it can also feel like trying to fit a square peg in a round hole.

The Best of Both Worlds

So is there any way to get at a sweet-spot, something quick and flexible, yet powerful? The first thing that probably comes to mind if you are a web developer is the CSS/HTML model. In fact, I think would be a rather good option if not for one missing piece, the unsung hero of the web: the HTML rendering engine. If anyone wants to implement Gecko or WebKit in ActionScript I think that would be a fun project, but because the web ecosystem evolved with the markup user and not the layout engine implementer in mind it’s rather complicated. There is also the fact that this model is based on the assumption of a page of fixed width but infinite height which doesn’t exactly suit our purposes.

What we can borrow though is the philosophy of good defaults and implicit rather than explicit layout specification. When you are building out a layout with HTML/CSS you can just drop stuff on the page and things more or less work and resize as you’d expect, then selectively you can move things around / add styling and padding and so on. To do something similar we can take advantage of one of the nice new features of Actionscript 3, the display hierarchy, which allows us to easily traverse, observe, and manage all the DisplayObjects at run time, without keeping track of them on our own as they are created. This way we can recurse through the tree, look for the appropriate properties, and apply positioning and sizing as necessary. If the properties don’t exist we’ll just leave things alone. When you start your project you can just write code to position stuff manually, then as things get more complex and start to solidify you can begin using layout management where it makes sense. I’m a bit of a commitment-phobe myself, so this really appeals to me; what’s especially nice is that you easily plug this in to any existing codebase.

Some Code

So, to get this working all we really need is a singleton that listens to the stage resize event, traverses the tree and places/sizes objects. Mine looks something like this:

  
/*..*/
stage.addEventListener(Event.RESIZE, this.onResize);
/*..*/
 
function onResize(e:Event){
	this.traverse(root);
}
 
function traverse(obj:DisplayObject){

	for (var i=0; i  < obj.numChildren; i++){

		var child = new LayoutWrapper(obj.getChildAt(i));
		if(child.no_layout)
			continue;
	   
		//layout code to determine how to place and size children
		//child.x = something
		//child.y = something


		child.setSize(W, H); // pre-order





		this.traverse(child);

		child.refresh(); // post-order
	   
	}
}


Two things of note here.
First is the LayoutWrapper which I use to set defaults for properties and functions that don’t necessarily exist on all objects, so sort of like slipping in a base class without having to alter any code. I use flash_proxy to make this easier, but you don’t have to do that. One thing you will want to do is store dummy values for width and height (something like _width and _height maybe as a throwback) so you can assign and keep track of the width and height of container elements without actually scaling the object. You’ll also want to decide how you want to handle turning on / off the layout management and what defaults make sense for you. In the above code I’m doing opt-out, using a no_layout variable to selectively turn off layout management where I don’t need it.
Second, I have two hooks for sizing, setSize on the way down, and refresh on the way up. This is useful in some cases because what you want to do at higher levels may depend on the size of children at lower levels which can change on the way down.
The layout code itself is outside the scope of this blog post, but I’ve found that a vertical stack container, a horizontal stack container, padding, and spacing is all I’ve needed to accomplish just about any UI. Of course, you can and should implement any layout tools that are useful to you; different projects may call for different layout containers and properties.

Some Fun

One thing that we are doing by using this approach is pushing the specification of the display tree into run-time. The analogy with HTML here is the part of the DOM that is generated by visible source vs the part of the DOM that is generated by JavaScript.
Going all cowboy like this does have its drawbacks. Specifically it can get a bit confusing as to what the tree looks like, or why this or that is not showing up, where is this Sprite attached, etc. Web developers have the Firebug inspect tool for a very similar problem, which I’ve found rather handy. We can get a poor-man’s version of something like this by writing some code that pretty prints the tree under an object on rollover or use the Flex debugger, but, depending on the size of your project, this can get unwieldy rather quickly.

As an excuse to play with the groovy new flare visualization library recently I tried pointing it at the display tree. The result is rather useful for debugging, especially if you have a lot of state changes / masking / weirdness (plus you look like you are using a debugging tool from the friggin future). Some nice things I’ve added: on roll over the object gets highlighted, I can see an edit some basic properties and see them update live. There’s quite a bit of room here to do something Firebug like, with drag and drop reparenting and the like (but be careful, not too interesting or you’d wind up reimplementing the Flash IDE).

If you’d like to try this yourself, this is the code to populate the flare data structure with the display tree:

public function populateData(){
	// create data and set defaults
	this.graph = new Tree();
	var n = this.graph.addRoot();
	n.data.label = String(root);
	this.populateDataHelper(n, root);
}

 
public function populateDataHelper(n, s){

	try{ // need this in case of TextFields
		var num_children = s.numChildren;
	}
	catch(e:Error){
		return;
	}
	
	//limit in case you have extremely long lists 
	//which can dominate the display
	num_children = num_children < 20 ? num_children : 20; 

	for(var i=0; i  < num_children; i++){
		var child = s.getChildAt(i);

		if(
		child != this //don't show the layout visualizer itself!
		&& child.visible //don't show items that are not visible
		){
			var c = this.graph.addChild(n);
			c.data.sprite = child;
			c.data.label = String(child);
			this.populateDataHelper(c, child)
		}
	}
  
}

Then just apply a visualization to this tree; I've been simply using the demo examples which has been sufficient for my uses, but obviously you can write your own as well.

Here it is in action on the flickr slideshow:

Boundaries, a tool to explore Flickr’s shapefiles

A few weeks ago we released our shapefiles via the API, and while most people were excited, some folks were a bit confused about what it all meant. Which is why Tom Taylor’s beautiful Boundaries application is so exciting. It helps you visualize the Flickr community’s twisty changing complex understanding of place.

You throw in a place name and it’ll attempt to find that place’s neighbours, using Yahoo GeoPlanet. For each of those neighbours, it uses the WOE ID to fetch and render the polygon from shapefile that Flickr returns.Tom Taylor: Flickr boundaries, now free to play with

Boundaries

5 Questions for Gustavo

Following our second interview Kellan snuck in 5 Questions for Paul Mison before I managed to tap Jim’s suggestion of Gustavo.

I think Drift Words sums Gustavo up nicely with “Although he’s far too clever, he makes up for it by using his polymath powers for good.” … and charts and stunning graphs. It’s always a pleasure to see what Gustavo comes up with.

And on that note …

1. What are you currently building that integrates with Flickr, or a past favorite that you think is cool, neat, popular and worth telling folks about? Or both.

Gustavo: First of all I must say: I’m not really a flickr tool developer. The only real tools I created are the quite minimalistic FlickRandom and Contact Crossing (very kindly hosted by Jim Bumgardner). Rather, I use flickr’s API to mine flickr’s database for interesting information, usually leading to some visualizations or at the very least a bunch of graphs.

The FlickrVerse, April 2005 poster: flickr's social network

One such visualization led to a browser for exploring related groups. Not a full-fledged tool since it explores static and outdated data.

In other words, what I do with flickr’s API is not tool development but analyses, trying to figure out the structure and dynamics of flickr’s social network, group content and participation, etc. The most recent analysis I performed attempted to understand how exposure to photo content flows through the contact network. Starting from a basic pattern combining three types of network relations, I quantified the importance of a user’s social network in determining his or her exposure to content of interest.

2. What are the best tricks or tips you’ve learned working with the Flickr API?

Gustavo: Cache, cache, cache. Depending on what you want to do, you might find yourself retrieving certain bits repeatedly, and you’ll definitely want to build a good local cache. Be careful though, as it’s all too easy to underestimate the size and complexity of the data. Your cache might unexpectedly morph from a speed asset to a sluggish monster.

Another potential problem from underestimating the size and complexity of the data is that you might suddenly discover that your code issued many more API calls than you expected. Remember to play nice, and keep an eye on your API key usage graphs.

Expect unexpected responses: It’s always good to make the code smart enough to realize the server’s response is not exactly what was expected, be it in completeness, format, special characters in free text, etc.

Finally, still on the theme of unexpected responses: Grow a thick skin, as there will always be someone who will misinterpret your motives for developing a tool that “uses” their information.

3. As a Flickr developer what would you like to see Flickr do more of and why?

Gustavo: Let me suggest two: one very simple, the other very complex.
The simple one is a “no special perms” authentication level. For some purposes, as a developer I’d like to have the user authenticate for the sole purpose of knowing who they are – I might want to give certain results for their eyes only, or someone else might want to allow certain actions to happen just once per user, etc. At the moment, the minimal level of authentication requires that the user entrust the developer with access to private data. Many users will rightfully decline giving such access, and as a developer I’d rather not have to even request and disclaim, when I don’t need it.

The more complex one requires a bit of background. Almost four years ago, there were various discussion threads on the topic of finding interesting content. Back then I took a stab at using network information to discover interesting photos – first looking for people who post similar stuff, then looking for people who share faved photos (which striatic called neighbors) and finally, actual photo suggestions.

neighbors

As an aside, the algorithm worked quite well even for people with a hundred or so favorites (I had just 21 faves!); people today have many thousand favorites – a very rich data set to start from. I went on to produce suggestion lists for many people, but that was always me, manually running scripts. Since then, other developers created interesting tools for content-driven content discovery, including Flickr Cross-Recommendations, inSuggest and flexplore.

flexplore 2.1 beta

The reason I never tried to create a stand-alone tool for this is that I quickly realized that to make it work reasonably fast, I essentially needed to replicate flickr’s database. If you take all my favorites, then list all the people that faved those photos, and finally enumerate all their favorites… and you then repeat this exercise for whoever requests it, you either need to use many thousand API calls per visitor (and wait!), or you need to create a huge cache, covering unpredictably disparate segments of flickr’s database. A massive cache that would grow stale very quickly unless people used the tool continuously. If you try to include extra information into the scoring scheme (say tags, color data, group membership…) the network use and storage requirements grow even worse. In other words, this project appears to be completely unsuitable for a developer without direct access to flickr’s database… and flickr’s content suggestion system (Explore) doesn’t provide any personalization tools: we all see the same.

The only effective solution I can suggest (beyond flickr developing a personalized Explore, or flickr sandboxing third party developers) is the creation of high level API calls, embodying some method of complex querying into the database. For example: A higher level API call could accept a list of users as query, and return a list of photos sorted by the number of people (from the query list) who faved them. With the current API, one has to retrieve the full favorites list for each person, collate the results, and discard the vast majority of the information that was transferred. A higher level API method could make such intermediary results invisible and save much bandwidth, latency and data replication.

Hey, you asked. ;)

4. What excites you about Flick and hacking? What do you think you’ll build next or would like someone else to build so you don’t have to?

Gustavo: There’s a number of things that attract me to flickr hacking. There’s of course the vast and ever-growing amounts of data, and the fact that it’s not “more of the same”: The diversity of data types (photos, users, metadata, groups, etc) and relations make this complex system fascinating to study. There is also the human side: People really care about their stuff. People get excited, for example, when they recognize their user icon in a visualization of the social network, and immediately want to explore around. Last but not least, I really appreciate the openness of the flickr API. I’m amazed that such wealth of information is shared so freely!

Free association, first page

As for “what next”, I have a couple of ideas. I’ll only hint that the word association analysis was a fun first step. :)

5. Besides your own, what Flickr projects and hacks do you use on a regular basis? Who should we interview next?

Gustavo: dopiaza as Utata‘s architect.

Dan: Thank you, Gustavo. Next up (unless Kellan gets in first!) dopiaza.

Images from GustavoG,
striatic and malanalars.