5 Questions for David Wilkinson

Me, by Sue

On the last 5 Questions I posted, I ended with “Thank you, Gustavo. Next up (unless Kellan gets in first!) dopiaza.” Well Kellan did indeed get in first, with both Jonnie Hallman and Fraser Speirs. So now, without any more procrastination, I present David Wilkinson, “Utata’s architect” as described by Gustavo …

1. What are you currently building that integrates with Flickr, or a past favorite that you think is cool, neat, popular and worth telling folks about? Or both.

David: I guess one of the most visible things that I’ve built so far is the Utata web site (http://www.utata.org/) and the tools that surround that. Utata, the Flickr group, is a pretty lively community, with over 16,000 members and a pool of over 270,000 photos. In Utata, we like to encourage people to participate, and our web site expands on the group pool and discussion threads on Flickr with a variety of projects for members to participate in, blogs that feature photos picked from the pool, and a wide variety of articles, notices and links. The key thing we’ve tried to do in Utata is to ensure that participation is simple and straightforward. Anyone can come along and join in – the only requirement is that you have a Flickr account.

Utata

Some of the most popular parts of Utata are undoubtedly the group projects – there are typically about three or four group projects in progress at any time, each with their own theme. To take part in a project, all members need to do is tag their photo with the specified project tag and it will then automatically get included in the project pages on the Utata web site. We do all of the hard work of keeping track of which photos are in which project, and presenting the project pages to the world. It is participation made easy – “tag and run”.

To make all that happen, there’s a whole pile of stuff in place behind the scenes. We have a sizeable set of PHP classes, a bunch of perl scripts and a MySQL database, that between them keep track of who all the Utata members are and which photos have been added to which project. The end result of all of this is a pretty smooth and largely automated system. A bunch of cron scripts keep the project database up to date and we have a private administration site that allows Utata administrators to quickly and easily perform a wide variety of tasks: they can set up and publish new projects, create polls, update member details, post new blog entries and so on. We also have HAL, the UtataBot, who keeps a watchful eye on the group pool, making sure that members abide by the rules and that we don’t just become a general dumping ground [RevDanCatt: here’s another pool cleaning bot, hipbot]. There are also the Utata badges that show off the latest contributions to the group pool. Actually, it’s only now, as I sit and write this list, that I realise just how much code there is sitting there on the Utata servers.

Project

One of the key parts of the system is the list of projects and their entries. Utata is a popular site, and it’s clearly not feasible to go back to Flickr to search for project entries each time a page is viewed, so a good deal of the code that runs Utata is devoted to maintaining our local copy of that data – keeping the cached data up to date and retrieving less commonly used elements from Flickr on demand as various pages are viewed. Given the number of projects held on Utata, and the size of the membership, it’s important to make efficient use of the cache, and I’ve tried a number of different strategies to minimise the number of API calls required. The biggest thorn in my side here is the limitation in flickr.photos.search that you can only ever get back the first 4000 photos for a tag-based search. I have some work-arounds to get around that, but if that limitation was ever increased I could make the whole thing far more efficient.

An interesting, but little known fact about Utata, is that it has its own API, which if you were to take a close look at, might well look rather familiar – it uses the Flickr API as its role model. There are only has a limited number of methods right now, but the framework allows more to be added quite easily. The idea behind the API is to enable other interesting applications to be written. There are only a few of these external applications in existence at the moment – there’s a stand-alone project browser, which I don’t think anyone other than me uses. A more interesting one is the Utatascreensaver, which cycles through photos from the Utata project of your choice.

As well as Utata, I dabble with a variety of other projects. The main thing I’m working on right now is a revamp of my Set Manager (http://www.dopiaza.org/flickr/setmgr/). One of the first comments I ever posted to Flickr Ideas was a request for Smart Sets – sets that could dynamically update themselves. You can think of them as the photographic equivalent of SmartPlaylists in iTunes. At the time, I was just starting to experiment with the Flickr API, so I decided to try my hand at building an application to create sets automatically. It’s not especially complicated, it’s really just a simple wrapper around flickr.photos.search, but it has certainly struck a chord with people, with currently over 25,000 authenticated users. It hasn’t changed very much since it was first written, so I figured it was time to give it something of an overhaul.

Set Manager

I’ve recently been playing around with Adobe’s Flex framework and decided that revamping the set manager would give me a good excuse to get to learn more about Flex and understand how it all hangs together – and so Version 2 of the set manager is now well under way. This new version allows you to not only build sets based on tags, interestingness, date and so on, it also allows you build sets based around location – the places API provides a very easy way to get access to all of the photos in a given location. For a more visual approach to set building, I’ve also hooked in Google Maps (sorry, Yahoo!) using their rather nifty Flash component to allow areas of interest to be easily chosen. This new Set Manager also piggy-backs on some of the work I did on Utata – it now exposes an API, built using the same framework as the Utata API, to allow the Flex application to communicate with the server.

Most of the Set Manager revamp has now been completed, and few Flickr stalwarts have been helping test it all out. The plan is to get this new version up and running by the end of the year – all I need is a little more spare time to finish things off…

2. What are the best tricks or tips you’ve learned working with the Flickr API?

David: Gustavo sneaked in ahead of me here and got in first with some of the best answers, but the main one is worth repeating – cache your data. Every API call you make can add several seconds on to the run time of your application or the load time of your web page, so you should really think carefully about whether each API call is necessary. If you think you’re going to need that data again in the near future, store it away somewhere – in a database, in the user’s session, anywhere. A good caching strategy can make the difference between a snappy responsive web site and a painfully slow, treacle-laden disaster.

One thing that seems to catch out a lot of people is the 4000 item limit to searches, which seems to be poorly documented (i.e., not at all). If you’re not aware of this limitation, it can suddenly rear it’s ugly head when you’re least expecting it – and usually when it’s least convenient. I remember the first time a Utata tag exceeded 4000 photos (the utatafeature tag currently returns over 16000 photos) and that certainly caused a few headaches. The only solution to this problem seems to be to construct your searches so that they’re sure to return less than 4000 matches. For the Utata tag searches, I ended up breaking the search down into a set of searches using smaller date ranges and aggregated the results. It’s not perfect, but it seems to be holding up.

And if you build an application or tool and release it out into the world, be prepared for an endless stream of emails from people who want it to do something just a little bit different from what it does right now…

3. As a Flickr developer what would you like to see Flickr do more of and why?

Run Zumi! Run like the wind!

David: Well, more Kitten Tuesdays are clearly an obvious priority. Apart from that? Well, here goes:

One of the biggest things I would like to see is a better way for Flickr members to allow API access to their photos. At the moment, users can either opt in or opt out of third-party API searches – it’s an all or nothing thing. There are good reasons for wanting to opt out of searches. It is all too easy, for example, to get an API key and create a rogue application: one that might display photos in an unwelcome context – on a commercial web site, perhaps. Of course, applications that violateFlickr’s terms of service tend to get shut down pretty quickly, but someone has to spot them and bring them to Flickr’s attention before that can happen. Threads regularly appear in the help forum about this, and many people choose to opt-out of public API searches to avoid having their photos show up in places they don’t expect. The problem with opting out is that members can then no longer make use of some of the applications and services that they would actually like to have access to their photos. Utata is a good example – we have many members who would like to take part in the projects but can’t because they are not willing to opt back in to API searches on a global basis.

I would like to see the current opt-out system expanded to allow members who opt out to then opt back in to API searches for selected applications – an API application white list, if you like. That would allow people to explicitly choose which applications were trusted and so grant them access.

There are a few changes to existing API methods that I’d really like to see:

– That 4000 photo limit I keep mentioning – it would be really nice if that could be lifted, even if only for certain search types, such as time-ordered.

flickr.photos.getInfo returns a whole host of useful information about a photo, but if I want to display that photo somewhere, I also invariably need to know what sizes are available to me. flickr.photos.getSizes gives me that, but that’s now two API calls. I almost always end up calling these two methods as a pair – it would be really nice if flickr.photos.getInfo could optionally return the sizes as well.

– One of the most common feature requests I get for my set manager is the ability to sort by number of favourites, or by views, or by number of comments. Allowing flickr.photos.search to do this would make a lot of people very happy.

– and I know it’s been mentioned quite recently, but searching on EXIF data would be very nice…

4. What excites you about Flick and hacking? What do you think you’ll build next or would like someone else to build so you don’t have to?

David: The thing I love about Flickr is that it’s not just about the photographs. The social network – that jumbled mass of communities that live within Flickr – is really the thing that makes Flickr so compelling. Without those interactions and relationships between people, Flickr would be just another web site full of photos, but as Gustavo’s graphs featured in the last 5 Questions help demonstrate, there’s much more depth to Flickr than a simple on-line photo gallery. It is quite fun building tools to help people perform particular tasks more easily – creating sets, uploading photos, managing group subscription, and so on – but all that’s not a patch the satisfaction gained from building tools that help whole new communities develop. That’s what makes Flickr for me: the people.

What will I build next? I don’t know. There’s always something to tinker with, and you never quite know where that tinkering will lead. The whole shapes thing looks quite interesting, but I haven’t had chance to really have a proper look at that yet. That might well be next on my list to play with – who knows?

5. Besides your own, what Flickr projects and hacks do you use on a regular basis? Who should we interview next?

David: The Flickr tool I’m using the most right now is Jeffrey Friedl’s ‘Export to Flickr’ Lightroom Plug-in (http://regex.info/blog/lightroom-goodies/flickr/). It’s a super little application – it integrates seamlessly with Adobe Lightroom, and does a brilliant job of streamlining the upload process. It’s a very well thought out piece of software – definitely my favourite Flickr tool of 2008.

Testing Jeffrey Friedl’s Lightroom Export Plugin for Flickr

Who to interview next? Sam Judson (http://www.flickr.com/photos/samjudson/). Sam was technical editor on my Flickr Mashups book (did I forget to mention that I wrote a whole book on building Flickr applications? Flickr Mashups, published by Wrox. Buy one now.). Anyway, as I was saying, Sam was technical editor on my book, and did a great job there, helping point some of my most embarrassing errors before they gotcommitted to print. He knows lots about the API and has been using it for even longer than me. He’s the author of the FlickrNet API library (http://www.codeplex.com/FlickrNet) and a regular contributor to both the Flickr API group and the developer mailing list.

Dan: Thank you, David, and sorry about taking forever to post your answers :) Next up Sam Judson.

Photos by dopiaza, Malingering and Axel Bührmann.

5 Questions for Gustavo

Following our second interview Kellan snuck in 5 Questions for Paul Mison before I managed to tap Jim’s suggestion of Gustavo.

I think Drift Words sums Gustavo up nicely with “Although he’s far too clever, he makes up for it by using his polymath powers for good.” … and charts and stunning graphs. It’s always a pleasure to see what Gustavo comes up with.

And on that note …

1. What are you currently building that integrates with Flickr, or a past favorite that you think is cool, neat, popular and worth telling folks about? Or both.

Gustavo: First of all I must say: I’m not really a flickr tool developer. The only real tools I created are the quite minimalistic FlickRandom and Contact Crossing (very kindly hosted by Jim Bumgardner). Rather, I use flickr’s API to mine flickr’s database for interesting information, usually leading to some visualizations or at the very least a bunch of graphs.

The FlickrVerse, April 2005 poster: flickr's social network

One such visualization led to a browser for exploring related groups. Not a full-fledged tool since it explores static and outdated data.

In other words, what I do with flickr’s API is not tool development but analyses, trying to figure out the structure and dynamics of flickr’s social network, group content and participation, etc. The most recent analysis I performed attempted to understand how exposure to photo content flows through the contact network. Starting from a basic pattern combining three types of network relations, I quantified the importance of a user’s social network in determining his or her exposure to content of interest.

2. What are the best tricks or tips you’ve learned working with the Flickr API?

Gustavo: Cache, cache, cache. Depending on what you want to do, you might find yourself retrieving certain bits repeatedly, and you’ll definitely want to build a good local cache. Be careful though, as it’s all too easy to underestimate the size and complexity of the data. Your cache might unexpectedly morph from a speed asset to a sluggish monster.

Another potential problem from underestimating the size and complexity of the data is that you might suddenly discover that your code issued many more API calls than you expected. Remember to play nice, and keep an eye on your API key usage graphs.

Expect unexpected responses: It’s always good to make the code smart enough to realize the server’s response is not exactly what was expected, be it in completeness, format, special characters in free text, etc.

Finally, still on the theme of unexpected responses: Grow a thick skin, as there will always be someone who will misinterpret your motives for developing a tool that “uses” their information.

3. As a Flickr developer what would you like to see Flickr do more of and why?

Gustavo: Let me suggest two: one very simple, the other very complex.
The simple one is a “no special perms” authentication level. For some purposes, as a developer I’d like to have the user authenticate for the sole purpose of knowing who they are – I might want to give certain results for their eyes only, or someone else might want to allow certain actions to happen just once per user, etc. At the moment, the minimal level of authentication requires that the user entrust the developer with access to private data. Many users will rightfully decline giving such access, and as a developer I’d rather not have to even request and disclaim, when I don’t need it.

The more complex one requires a bit of background. Almost four years ago, there were various discussion threads on the topic of finding interesting content. Back then I took a stab at using network information to discover interesting photos – first looking for people who post similar stuff, then looking for people who share faved photos (which striatic called neighbors) and finally, actual photo suggestions.

neighbors

As an aside, the algorithm worked quite well even for people with a hundred or so favorites (I had just 21 faves!); people today have many thousand favorites – a very rich data set to start from. I went on to produce suggestion lists for many people, but that was always me, manually running scripts. Since then, other developers created interesting tools for content-driven content discovery, including Flickr Cross-Recommendations, inSuggest and flexplore.

flexplore 2.1 beta

The reason I never tried to create a stand-alone tool for this is that I quickly realized that to make it work reasonably fast, I essentially needed to replicate flickr’s database. If you take all my favorites, then list all the people that faved those photos, and finally enumerate all their favorites… and you then repeat this exercise for whoever requests it, you either need to use many thousand API calls per visitor (and wait!), or you need to create a huge cache, covering unpredictably disparate segments of flickr’s database. A massive cache that would grow stale very quickly unless people used the tool continuously. If you try to include extra information into the scoring scheme (say tags, color data, group membership…) the network use and storage requirements grow even worse. In other words, this project appears to be completely unsuitable for a developer without direct access to flickr’s database… and flickr’s content suggestion system (Explore) doesn’t provide any personalization tools: we all see the same.

The only effective solution I can suggest (beyond flickr developing a personalized Explore, or flickr sandboxing third party developers) is the creation of high level API calls, embodying some method of complex querying into the database. For example: A higher level API call could accept a list of users as query, and return a list of photos sorted by the number of people (from the query list) who faved them. With the current API, one has to retrieve the full favorites list for each person, collate the results, and discard the vast majority of the information that was transferred. A higher level API method could make such intermediary results invisible and save much bandwidth, latency and data replication.

Hey, you asked. ;)

4. What excites you about Flick and hacking? What do you think you’ll build next or would like someone else to build so you don’t have to?

Gustavo: There’s a number of things that attract me to flickr hacking. There’s of course the vast and ever-growing amounts of data, and the fact that it’s not “more of the same”: The diversity of data types (photos, users, metadata, groups, etc) and relations make this complex system fascinating to study. There is also the human side: People really care about their stuff. People get excited, for example, when they recognize their user icon in a visualization of the social network, and immediately want to explore around. Last but not least, I really appreciate the openness of the flickr API. I’m amazed that such wealth of information is shared so freely!

Free association, first page

As for “what next”, I have a couple of ideas. I’ll only hint that the word association analysis was a fun first step. :)

5. Besides your own, what Flickr projects and hacks do you use on a regular basis? Who should we interview next?

Gustavo: dopiaza as Utata‘s architect.

Dan: Thank you, Gustavo. Next up (unless Kellan gets in first!) dopiaza.

Images from GustavoG,
striatic and malanalars.

5 Questions for Paul Mison

One of the key goals for this blog when we launched it 9 months ago was to be a channel for the voice of the Flickr development community. Most importantly all the amazing developers building on our APIs. Which is by way of introducing our third developer interview here at code.flickr, our first was with Neil Berkman of Photopholow, while our 2nd, and first in the 5 questions format was with Jim Bumgardner a few weeks ago.

We’ll be posting an interview from GustavoG (as tapped by Jim) soon, but in the mean time I want to post an interview from Paul Mison aka blech. Paul is an active participant in the Flickr API group, and I’ve personally been using and loving his new project SnapTrip. And he likes rainbows, which makes him good by us.

A Trip Underground

1. What are you currently building that integrates with Flickr, or a
past favorite that you think is cool, neat, popular and worth telling
folks about? Or both.

Paul: My current main project is snaptrip, which works with Dopplr, a website for sharing personal travel information, as well as with Flickr. It helps you to find photos associated with a trip listed in Dopplr, to label them so that Dopplr can find them easily later on, and also lets you geotag them (if you haven’t already). I’ve also written a Greasemonkey user script so that you can see when other people’s photos were taken during a Dopplr trip.

Flickr photos in snaptrip

In the past I’ve mainly written command line tools; I used a script to migrate images out of my previous home-brewed image hosting to Flickr, and I have another that applies machine tags for EXIF properties (since tags make it easy to find photos; here’s all my photos with a focal length of 50mm). My great lost project was a web application called groupr that would let you see all the photos in groups you’re a member of, but unfortunately the platform I built it on was withdrawn, and I should rebuild it elsewhere (possibly on Google’s App Engine, like snaptrip; possibly not).

2. What are the best tricks or tips you’ve learned working with the
Flickr API?

Paul: The API Explorer is wonderfully useful, and it should be everyone’s first port of call when developing an application. Beyond that, I’d advise picking an API framework that does the tricky things (notably, user authentication – I’ve handcoded it myself, and it can be fiddly), but otherwise gets out of your way.

snaptrip uses Beej’s Python Flickr API, and I’ve previously used flickraw (for Ruby) and Flickr::API (for Perl). All three are nice and minimal, so that when new API methods are added, you don’t have to update your library. flickraw uses JSON internally, too, which is very nice if (like me) you lean towards dynamic, rather than static, languages.

For Greasemonkey scripts, this post from mortimer? about using API calls in GM scripts is great.

3. As a Flickr developer what would you like to see Flickr do more of
and why?

Paul: Well, a relatively minor request would be for JSON in the API Explorer. More seriously, there’s an entire class of methods I wish existed for groups. For example, the only way to track conversations at the moment is the group RSS feed, which isn’t segregated by thread. There’s no way to find out a group’s administrators or moderators. A final example is that the queue of photos awaiting approval isn’t exposed to the API. While I’m not sure I’d have used all of these in groupr, some of them would have been very handy.

Another small request would be for more extras, especially a few method-specific ones. In particular, I’d love to see ‘favedate’ on flickr.favorites.getPublicList.

4. What excites you about Flickr and hacking? What do you think you’ll
build next or would like someone else to build so you don’t have to?

Paul: For all that my answer to the last question was a demand for more methods, Flickr is exciting both because of the wealth of photography there, and the richness of the methods of getting at it. The geographical data, and access to it, that has emerged over the last year is really interesting, and I’d love to do something with it (and intend to, somehow, in snaptrip). However, for a complete new project, I’ve been poking for a year or so at making your Flickr favourites look a bit nicer, and maybe within another year I’ll actually have something out publicly. (I really need a simple job queue; anyone?)

5. Besides your own, what Flickr projects and hacks do you use on a
regular basis? Who should we interview next?

Paul: Well, as a Mac user, I’m a fan of Fraser Speirs Flickr Export for getting my images onto the site in the first place, and if I upgrade to an iPhone I look forward to playing more with Exposure. On the web, Dario Taraborelli’s Group Trackr is very nice, and I use fd’s Flickr Scout every now and again to see if anything’s hit Explore.

However, I think the most impressive recent project I’ve seen is a desktop app written using Adobe’s Air, called Destroy Flickr, by Jonnie Hallman. There were a lot of subtle UI techniques to hide the latency inherent in talking to a network service, so I think he’d be a great choice for an interview.

Kellan: Thanks Paul! And y’all be looking out for an interview with Jonnie and his oddly named project.

5 Questions for Jim Bumgardner

We’ve been keeping a careful eye on our Sister Blog to see what they’re up to. Something that’s particularly caught our eye is "5 Questions ", asking the same 5 questions to the Flickrverse, with the last question being who we should ask next. And so, we hope, it goes on and on.

Banana-bub

This is our version, asking questions of those that develop, hack and fiddle with Flickr in new and interesting ways. Of course we couldn’t start with anyone else but KrazyDad (aka Jim Bumgardner).

Jim founded the Flickr Hacks group back in the day, a great place to hang out and ask question if you want to learn how to bend Flickr to your will. In 2006 he also coauthored the Flickr Hacks book for O’Reilly and happily for us he hasn’t stopped tinkering with Flickr yet.

So, without any further ado, 5 Questions for Jim Bumgardner:

1. What are you currently building that integrates with Flickr, or a past favorite that you think is cool, neat, popular and worth telling folks about? Or both.

Jim: It seems like I’m always building something that integrates Flickr. A recent favorite is this interactive mosaic that shows the most interesting photos of the week.

The photos are arranged to form a spiral, a form that appears quite frequently in my work.

www.coverpop.com/pop/flickr_interesting
Coverpop: Most Interesting Photos of the Week

I have a "cron job" which runs on one of my computers at home, which updates this mosaic every week, so the photos in it are always fresh. Incidentally, I prefer to call this process a "cron joy." Oh, nerd humor…

2. What are the best tricks or tips you’ve learned working with the Flickr API?

Jim: I think every Flickr hackr should have access to a powerful high level
graphics library. My library of choice is ImageMagick combined with the Perl programming language (it also works nicely with Ruby), but the GD library, which works with various languages, and PIL, for Python, are also good.

I not only use ImageMagick for building mosaics and graphs, but also for "under the hood" kinds of things, like measuring the average colors of photos for the Colr Pickr (see below).

3. As a Flickr developer what would you like to see Flickr do more of and why?

Jim: One of the very first Flickr hacks I made was the Colr Pickr

www.krazydad.com/colrpickr/

…which allows photos to be selected by color. Since that appeared, I’ve worked on, and seen some fancier variations on the concept, that allow larger quantities of Flickr photos to be selected using multiple colors. But all these systems, require that thousands or even millions of thumbnails be downloaded and analyzed for color. This is because Flickr does not supply "average color" information in its APIs, and cannot provide the color search functionality that this data would enable.

flickr Colr Pickr

I would like to see Flickr provide, via it’s APIs, the three most common colors in each photo (using cluster analysis), and provide a way to search for photos which match one, two, or three colors. These parameters, similar to geocode searches, would need to be combined with some other search parameters, such as tags, to narrow
the field down.

A feature like this would be a godsend to designers. I’ve got sample code for the color analysis, if anyone’s interested… :)

4. What excites you about Flick and hacking? What do you think you’ll build next or would like someone else to build so you don’t have to?

Jim: One thing that excites me is the ability to access large quantities of photos that contain valuable metadata, such as the time the photo was taken, or the geocoded location. I used the ‘date taken’ data to construct this very cool graph of sunsets:

A year of sunsets

While most digital cameras store the time within photos, these days, not enough of them automatically store the location. We have to rely on photographers adding the geocoded information manually, and sadly, not enough of them are geeky enough to do it. I’m looking forward to the day, a few years from now, when most of the new photos on Flickr will also contain geocoded information, as this will enable me to make apps which enable a kind of instant photo-journalism of heavily photographed events, such as rallies and parades. We’re seeing the beginnings of these kind of apps now, but we’re barely scratching the surface.

5. Besides your own, what Flickr projects and hacks do you use on a regular basis? Who should we interview next?

mc-50 map of FlickrLand: flickr's social network

Jim: GustavoG has made some amazing graphs which exploit and illustrate the Flickr social network.

Dan: Thank you, Jim. Next up for our thrilling installment of 5 Questions, GustavoG .

Images from krazydad / jbum , earthhopper and GustavoG.