About Daniel Bogan

I do the stuff on the Internets.

Pleiades: A guest post

I recently asked our friend Sean from Pleiades (which I will *never* be able to spell correctly) to write up a lil’ guest post on how we did something cool with Flickr machine tags and ancient sites of the world – and here it is!


Intro

I’m Sean Gillies, a programmer at ISAW, the Institute for the Study of the
Ancient World at New York University. I’m part of the Digital Programs team,
which develops applications for researchers of ancient civilizations. Most of
my work is on a gazetteer and graph of ancient places called Pleiades. It
identifies and describes over 34,000 places in antiquity and makes them
editable on the web. A grant from the U.S. National Endowment for the
Humanities (NEH) running through April 2013 is allowing Pleiades to bulk up on
ancient world places and develop features that can support ambitious
applications like the digital classics network called Pelagios.

Background

In August of 2010, Dan Pett and Ryan Baumann suggested that we coin Flickr
machine tags in a "pleiades" namespace so that Flickr users could assert
connections between their photos and places in antiquity and search for photos
based on these connections. Ryan is a programmer for the University of
Kentucky’s Center for Visualization and Virtual Environments and
collaborates with NYU and ISAW on Papyri.info. Dan works at the British
Museum and is the developer of the Portable Antiquities Scheme’s website:
finds.org.uk. At about the same time, ISAW had launched its Flickr-hosted
Ancient World Image Bank and was looking for ways to exploit these images,
many of which were on the web for the first time. AWIB lead Tom Elliott,
ISAW’s Associate Director for Digital Programs, and AWIB Managing Editor Nate
Nagy started machine tagging AWIB photos in December 2010. When Dan wrote "Now
to get flickr’s system to link back a la openplaques etc." in an email, we all
agreed that would be quite cool, but weren’t really sure how to make it happen.

As AWIB picked up steam this year, Tom blogged about the machine tags. His
post was read by Dan Diffendale, who began tagging his photos of cultural
objects to indicate their places of origin or discovery. In email, Tom and Dan
agreed that it would be useful to distinguish between findspot and place of
origin in photos of objects and to distinguish these from photos depicting the
physical site of an ancient place. They resolved to use some of the predicates
from the Concordia project, a collaboration between ISAW and the Center for
Computing in the Humanities at King’s College, London (now the Arts and
Humanities Research Institute), jointly funded by the NEH and JISC. For
findspots, pleiades:findspot=PID (where PID is the short key of a Pleiades
place) would be used. Place of origin would be tagged by pleiades:origin=PID.
A photo depicting a place would be tagged pleiades:depicts=PID. The original
pleiades:place=PID tag would be for a geographic-historic but otherwise
unspecified relationship between a photo and a place. Concordia’s original
approach was not quite RDF forced into Atom links, and was easily adapted to
Flickr’s "not quite RDF forced into tags" infrastructure.

I heard from Aaron Straup Cope at State of the Map (the OpenStreetMap annual
meeting) in Denver that he’d seen Tom’s blog post and, soon after, that it was
on the radar at Flickr. OpenStreetMap machine tags (among some others) get
extra love at Flickr, meaning that Flickr uses the machine tag as a key to
external data shown on or used by photo pages. In the OSM case, that means
structured data about ways ways and nodes, structured data that surfaces on
photo pages like http://flickr.com/photos/frankieroberto/3396068360/ as "St
George’s House is a building in OpenStreetMap." Outside Flickr, OSM
users can query the Flickr API for photos related to any particular way or
node, enabling street views (for example) not as a product, but as an
grassroots project. Two weeks later, to our delight, Daniel Bogan contacted Tom
about giving Pleiades machine tags the same kind of treatment. He and Tom
quickly came up with good short labels for our predicates and support for the
Pleiades machine tags went live on Flickr in the middle of November.

The Pleiades machine tags

Pleiades mainly covers the Greek and Roman world from about 900 BC – 600 AD. It
is expanding somewhat into older Egyptian, Near East and Celtic places, and
more recent Byzantine and early Medieval Europe places. Every place has a URL
of the form http://pleiades.stoa.org/places/$PID and it is these PID
values that go in machine tags. It’s quite easy to find Pleiades places through
the major search engines as well as through the site’s own search form.

The semantics of the tags are as follows:

pleiades:depicts=PID
The PID place (or what remains) is depicted in the photo
pleiades:findspot=PID
The PID place is where a photo subject was found
pleiades:origin=PID
The PID place is where a photo subject was produced
pleiades:where=PID
The PID place is the location of the photo subject
pleiades:place=PID
The PID place is otherwise related to the photo or its subject

At Pleiades, our immediate use for the machine tags is giving our ancient
places excellent portrait photos.

On the Flickr Side

Here’s how it works on the Flickr side, as seen by a user. When you coin a new,
never before used on Flickr machine tag like pleiades:depicts=440947682 (as
seen on AWIB’s photo Tombs at El Kab by Iris Fernandez), Flickr fetches the
JSON data at http://pleiades.stoa.org/places/440947682/json in which the
ancient place is represented as a GeoJSON feature collection. A snippet of
that JSON, fetched with curl and pretty printed with python

  $ curl http://pleiades.stoa.org/places/440947682/json | python -mjson.tool

is shown here:

  {
    ...
    "id": "440947682",
    "title": "El Kab",
    "type": "FeatureCollection"
  }

[Gist: https://gist.github.com/1488270]

The title is extracted and used to label a link to the Pleiades place under the
photo’s "Additional info".

https://i2.wp.com/farm8.staticflickr.com/7161/6522002861_537ca823d4_b_d.jpg

Flickr is in this way a user of the Pleiades not-quite-an-API that I blogged
about two weeks ago.

Flickr as external Pleiades editor

On the Pleiades end, we’re using the Flickr website to identify and collect
openly licensed photos that will serve as portraits for our ancient places. We
can’t control use of tags but would like some editorial control over images,
so we’ve created a Pleiades Places group and pull portrait photos from its pool.
The process goes like this:

https://i0.wp.com/farm8.staticflickr.com/7172/6522275377_bbda2a70ac_o_d.png

We’re editing (in this one way) Pleiades pages entirely via Flickr. We get a kick
out of this sort of thing at Pleiades. Not only do we love to see small pieces
loosely joined in action, we also love not reinventing applications that already
exist.

Watch the birdie

This system for acquiring portraits uses two Flickr API methods:
flickr.photos.search and flickr.groups.pools.getPhotos. The guts of it
is this Python class:

  class RelatedFlickrJson(BrowserView):

      """Makes two Flickr API calls and writes the number of related
      photos and URLs for the most viewed related photo from the Pleiades
      Places group to JSON like

      {"portrait": {
         "url": "http://flickr.com/photos/27621672@N04/3734425631/in/pool-1876758@N22",
         "img": "http://farm3.staticflickr.com/2474/3734425631_b15979f2cd_m.jpg",
         "title": "Pont d'Ambroix by sgillies" },
       "related": {
         "url": ["http://www.flickr.com/photos/tags/pleiades:*=149492/"],
         "total": 2 }}

      for use in the Flickr Photos portlet on every Pleiades place page.
      """

      def __call__(self, **kw):
          data = {}

          pid = self.context.getId() # local id like "149492"

          # Count of related photos

          tag = "pleiades:*=" + pid

          h = httplib2.Http()
          q = dict(
              method="flickr.photos.search",
              api_key=FLICKR_API_KEY,
              machine_tags="pleiades:*=%s" % self.context.getId(),
              format="json",
              nojsoncallback=1 )

          resp, content = h.request(FLICKR_API_ENDPOINT + "?" + urlencode(q), "GET")

          if resp['status'] == "200":
              total = 0
              photos = simplejson.loads(content).get('photos')
              if photos:
                  total = int(photos['total'])

              data['related'] = dict(total=total, url=FLICKR_TAGS_BASE + tag)

          # Get portrait photo from group pool

          tag = "pleiades:depicts=" + pid

          h = httplib2.Http()
          q = dict(
              method="flickr.groups.pools.getPhotos",
              api_key=FLICKR_API_KEY,
              group_id=PLEIADES_PLACES_ID,
              tags=tag,
              extras="views",
              format="json",
              nojsoncallback=1 )

          resp, content = h.request(FLICKR_API_ENDPOINT + "?" + urlencode(q), "GET")

          if resp['status'] == '200':
              total = 0
              photos = simplejson.loads(content).get('photos')
              if photos:
                  total = int(photos['total'])
              if total < 1:
                  data['portrait'] = None
              else:
                  # Sort found photos by number of views, descending
                  most_viewed = sorted(
                      photos['photo'], key=lambda p: p['views'], reverse=True )
                  photo = most_viewed[0]

                  title = photo['title'] + " by " + photo['ownername']
                  data['portrait'] = dict(
                      title=title, img=IMG_TMPL % photo, url=PAGE_TMPL % photo )

          self.request.response.setStatus(200)
          self.request.response.setHeader('Content-Type', 'application/json')
          return simplejson.dumps(data)

[Gist: https://gist.github.com/1482469]

The same thing could be done with urllib, of course, but I’m a fan of httplib2.
Javascript on Pleiades place pages asynchronously fetches data from this view
and updates the DOM. The end result is a "Flickr Photos" section at the bottom
right of every place page that looks (when we have a portrait) like this:

https://i1.wp.com/farm8.staticflickr.com/7012/6522002865_350997d652_o_d.jpg

We’re excited about the extra love for Pleiades places and can clearly see it
working. The number of places tagged pleiades:*= is rising quickly – up 50%
just this week – and we’ve gained new portraits for many of our well-known
places. I think it will be interesting to see what developers at Flickr, ISAW,
or museums make of the pleiades:findspot= and pleiades:origin= tags.

Thanks

We’re grateful to Flickr and Daniel Bogan for the extra love and opportunity to
blog about it. Work on Pleiades is supported by the NEH and ISAW. Our machine tag
predicates come from a NEH-JISC project – still bearing fruit several years later.

A lil’ time with… Stephen

IMG_9625

Who are you, and what do you?

I’m Stephen Woods and I’m a frontend engineer at Flickr. Along with the rest of our team I am responsible for the bits that make the UI work, which mostly means html, css and javascript. So I write code, but I also make sure the design of the site looks and works the way the designers want it to.

What hardware are you using?

In terms of computers I have 15″ MacBook Pro with a 24″ monitor, as well as the standard ipad and iphone. I don’t have a desktop anymore and I do any hacking or side projects on a slicehost image. I used to have a pretty large collection of random computers and desktops, but over the years I have really cut back on hardware, it just doesn’t seem necessary anymore. Like a lot of people here I also enjoy photography. I have a digital rebel xsi and a someone worse for wear 4×5 view camera, neither of which I use as much as I would like.

And what software?

I use TextMate (and sometimes vim) for coding. My development browser is mostly Safari, with Firefox/firebox for some things. I use Charles proxy a lot for HTTP debugging. VMWare fusion is indispensable. I have a portable hard drive with a little collection of VM images to test all the various browsers out there.

I am also completely dependent on Evernote. I use it for everything. Frequently I remember something I need to do or figure out a problem while lying in bed, so I just grab my ipad, write down my notes and they are on my laptop when I need them.

What would be your dream setup?

My dream setup would be the MacBook Pro with the good video so I could actually play games on it. Otherwise I am pretty much living the dream.

A lil’ time with… mroth

Mroth Hotdesks

Who are you, and what do you?

Ahoy! I’m Matthew Rothenberg. Most people here call me mroth.

I’m the head of product strategy / management at Flickr, which means I’m ultimately responsible for figuring out what Flickr needs to build and why. I work with all of our teams during the development of Flickr features to help make sure they meet our goals and vision, as well as do boring manager-y things. In reality, that means I spend most of my life in meetings (formal or informal).

In a broader sense, I’m responsible for trying to make Flickr the world’s best place to share your photos with the people who matter to you.

What hardware are you using?

I have a 15” MacBook Pro with 4GB of RAM (which I insist on still calling a PowerBook). I have a 24” Dell display both at work and at home, but I almost never bother to plug into them anymore.

Mobile, oy–I have way too many phones. My primary phones that I carry every day are a BlackBerry 9650 (writing email, making phone calls I want to stay connected for more than 60 seconds), and an iPhone 4 (browsing the web and my primary camera). I also have a Palm Pre (used to be my main phone, now I use it mostly for development), HTC Evo 4G (development and testing) and a Nokia N95-3 (which I use as the world’s most expensive alarm clock). With all those phones, I’m a heavy Google Voice user for bouncing calls back and forth.

For running all my side projects and home backup, I have a homebuilt Linux (Ubuntu Server 10.04.1 LTS) server with an Atom chipset sitting in a MSI Wind enclosure, and a Seagate “Green” hard drive that spins down to lower speeds to conserve power.

My loft’s stereo is wired up for network play via AirTunes and Airport Express, but unfortunately the software for it is so terrible (no streaming from iOS) it doesn’t get much use currently — I’m hoping AirPlay in iOS 4.2 will finally resolve this. More successfully I do stream video via the 802.11n network from the Linux server to my Sony PS3 which is hooked up to the television (Sony 40” flatscreen of some sort, if anyone actually cares).

The most transformative tech hardware I’ve purchased in years has been the Kindle (2nd gen), which completely changed my relationship with books. It perfectly embodies the design principle of doing one thing exceptionally well, and has resulted in me reading approximately 350% more books since I purchased it.

Finally, and most importantly, the best productivity-enhancing hardware I own is a La Marzocco GS/3 espresso machine.

And what software?

I live in email — I currently use Mail.app for its live search folders but I’m not entirely satisfied with it. There is a definite market for a real “power user” email app that hasn’t quite been filled yet.

Skitch is another “couldn’t live without it” application for me, since I spend a lot of time reviewing not-quite-ready-for-primetime features, it lets me super quickly capture and annotate notes for the team.

I’ve been a UNIX nerd for a long long time, so I spend a lot of time in Terminal.app, and sadly even do most of my basic navigation file system operations in there. For actually hacking on code, I use TextMate.

Most of my side projects are coded in Python, but lately I’ve been dabbling in Ruby/Rails.

Github completely changed the way I hack on side projects. If you haven’t read Anil Dash’s recent blog post on it, you should. If you ever develop software for fun and you aren’t using it, you’re sorely missing out.

I occasionally have to build presentations for my job, so when I do, I insist on using Keynote.

What would be your dream setup?

A mythical iPhone with a physical QWERTY keyboard (jailbroken, of course).

A lil’ time with… Chris

CJ Martin

Who are you, and what do you?

Hello, I’m Chris Martin, one of the resident nerds here at Flickr who keep everything running smoothly. I’m from Atlanta, Georgia but I don’t have a southern accent and this seems to disappoint people pretty regularly, I apologize for that.

My focus at Flickr is on mobile engineering and making sure people can get their lovely photos out of their pockets and on to Flickr; and vice versa. In real life I’m a professional travel writer/photographer. I’ve never written anything and my photos are sub-par, but I have to justify traveling around the world somehow.

What hardware are you using?

At work I use a 15″ MacBook Pro hooked up to two 24″ Dell displays, one for code and one for browser windows etc., they are both quite useful for creating a barrier to hide behind during the regular FlickrHQ foam dart wars. I also have a variety of mobile devices that we use to test our own code, 3rd party integrations, and API apps. My Flickr Moleskine and stack of post-it notes are also integral parts of my work hardware setup.

At home I have a pretty assorted collection of hardware, the most interesting of which is probably a 450MHz G4 Cube that sits quietly on my desk and still does a great job as a file server. Rounding out my fanboy worthy collection of Apple products are a unibody 15″ MacBook Pro, an iPad, and an iPhone 4 that replaced my 1st gen iPhone which I’ve used for the past 3 years (and still do for testing). I have two “hackintosh” machines; one desktop with a couple terabyte drives hooked up to my Samsung TV as a media center, and a 12″ ASUS 1201n which I find myself using more than my MacBook Pro because I like it’s small size when I’m out, and I hook up to a 24″ display when I’m at my desk at home.

In my “retired” collection are a linux PC that used to be a MythTV media center, an original eepc 701 that served me well on a 5 month backpacking trip, an ancient original iMac, and an assorted collection of iPods. I have a problem saying goodbye.

It’s probably appropriate to include camera gear as well… I have a Canon 350D which usually wears a 50mm f1.4 lens; it’s starting to show it’s age but has seen and faithfully captured many great trips over the years, a Kodak M1033 which isn’t the greatest camera in the world but it’s built like a tank and was the least obscenely expensive thing I could find in India after leaving a Canon on a train, and a Panasonic Lumix ZS3 that takes excellent video. However, I still take most of my photos with my iPhone.

And what software?

Obviously I’m an OSX guy, ever since I moved from Gentoo linux to a powerbook in college. I love the polish of the Mac GUI, and the raw power of Unix that’s just below the pretty surface.

I use the assortment of standard apps (Safari, Mail.app, iCal, and address book) as my primary browser, mail, and calendar applications. I’ve moved around a bunch in the past in this area, but the apple tools have finally started playing nicely with the google suite of services and they work well enough to keep me happy and in sync with my iPhone.

For development I primarily use TextMate (web-dev) and xCode (Cocoa), with the occasional trip over to Coda when I need to quickly touch something I’m doing any front end development on. I don’t have any special tools I use for source control or file transfer, that’s all svn, git, scp, etc. in terminal.

A few other apps like Linkinus, Adium, and Echofon live in my dock all the time, and of course I pull out photoshop when I need to mock something up.

As for utilities, I absolutely can’t live without Visor, a SIMBL plugin that makes my terminal appear on command from the top of the screen, teleport is a nifty tool for sharing one keyboard and mouse between multiple computers, DeskLickr keeps my desktop beautiful with photos from Flickr, and MenuMeters gives me a quick view of how the computer is doing from the menu bar.

On my phone, I use Echofon for twitter, Reeder is my absolute favorite RSS reader of all time (I wish there were a desktop version), Instapaper, AutoStitch to make panorama photos, iTimelapse for time-lapse video, and of course m.flickr.com and the Flickr app for browsing Flickr.

What would be your dream setup?

As much as I love laptop computers and the amazing mobile devices we have today, I can’t wait until our computing experiences are more thoroughly integrated into all of the other objects we use in daily life. I don’t just mean “connected things”, but more along the lines of augmented reality. I guess my dream setup would be a pair of contact lenses that I could put in in the morning and immediately start seeing extra information in every day life. I’m sure it will be done, but it will be a fine balance between adding to reality, and completely removing ourselves from it; I hope we do it well.

A lil’ time with… Timoni

Monochromatic

Who are you, and what do you?

I am Timoni Grone, former-Nebraska-farmgirl-now-Californian-city-dweller, and I design web interfaces. I’m currently a UI/visual designer at Flickr, which is a totally rad gig.

What hardware are you using?

At work, I have a 2.15Ghz Core 2 Duo 15″ MBP, a 24″ external monitor, the short Mac keyboard and a Mighty Mouse. At home, I have a 2.67GHz 15″ MBP and a Magic Mouse. I forget how much RAM I have, but both computers are fast enough.

I also have an old Mac Mini working as a media center (hooked up to my absurdly ginormous television), a 2TB Lacie Quadra for backups, an Airport for network stuff, a 16GB wifi iPad and an iPhone 4.

I briefly had a Wacom Cintiq 12WX earlier this year. I excitedly used it for a few weeks, then left it untouched for months, and finally sold it. It was too much of a pain to switch back and forth between the tablet and my laptop.

And what software?

For web dev, I like TextMate, though I was on a strict Coda diet for a while when it made sense to have seamless FTP integration. If I’m using TextMate, I use Transmit. Current browser of choice is Chrome, cause it’s super-fast, though I prefer Firebug to Web Inspector for development.

Day to day, I use Photoshop and Illustrator all the time. I’ve been trying to switch to comping in Fireworks for a while, but for some reason it was never installed on my work computer, so I default back to Photoshop. The way it handles smart objects is a pain, but Save For Web in Illustrator can be effing tricky sometimes (also, the way Illustrator renders text below a certain size makes my eyes hurt).

I have Skitch, the Last.fm scrobble app, and Dropbox running at all times on both of my MBPs. I use Dropbox to sync my work folder and one of my iTunes libraries. Works like a charm and it’s only ten bucks a month.

For print work, I use InDesign. For small things like bills & letterhead, I use TextEdit. I try to avoid halfway-done word processors like Pages and Word; the way they handle style sheets drives me batty.

I just started using Notational Velocity for my to-do lists (thanks Daniel!). I use Google Calendar for scheduling, and have SMS reminders sent to my phone (a total lifesaver). When I want to write, I use Ommwriter, the nicest little text editor out there.

On my iPad, I most often use NetNewsWire, Instapaper, the Kindle app, and Autodesk’s Sketchbook Pro.

What would be your dream setup?

We’re at a really fascinating point in hardware development right now, which makes it difficult to answer this question. My knee-jerk answer is that I want the Young Lady’s Illustrated Primer combined with an iPad combined with the Cintiq combined with, you know, a Cray supercomputer or something else equally powerful.

The problem is, really, handwriting recognition; if you’ve ever tried to use the iPad with an external keyboard, you’ll know exactly what I mean. Switching from typing to writing or drawing and back is a fucking pain. Regular notebooks allow you to draw and write without changing your hand position, which doesn’t seem like a luxury until you try actually working on a tablet and then find you need to input text.

SJ may think that styli are inelegant, but the fact is, using a pen to write or draw on paper is both comfortable and easy; it’s just not as fast as typing. Most people are content with inputting data via a keyboard, and this makes sense for a lot of jobs: marketing, business development, finance, and programming, for example. But for the designers, there’s a big gap between starting the creative process and executing the product design *because* it’s much easier to sketch out your ideas on paper, with a pen, than a computer. And this is unfortunate; in the future, we should have computers that allow us to keep contexts for different stages of product development. The iPad and ThinkPads are steps in the right direction, but they’re still awfully clumsy, which is why, in part, people criticize the iPad as a product for mere consumption.

I want a Moleskine that is a blindingly superfast computer. That’s my dream setup.

A lil’ time with… Nolan

This is the first in a new series of interviews with the Flickr staff, asking what tools (hardware, software) they use to get the job done. Our first volunteer is Nolan Caudill, PHP engineer and wearer of beards.

hat hair

Who are you, and what do you?

I’m Nolan Caudill, a software engineer at Flickr (obviously). I’m a recent addition to the company and a new transplant to San Francisco. At Flickr, I work on i18n/l10n, though I like to keep a variety of projects on my plate to keep things interesting.

I’m originally from Wilkes County, North Carolina which is the moonshine capital of the world and the birthplace of NASCAR (and those two facts are indeed related). I moved here from the Research Triangle in North Carolina, going to school in Chapel Hill and residing in Durham. My wife and our cat moved here in May and are both loving this chilly, foggy city with good beer and food on every corner.

What hardware are you using?

I’m currently typing this on my work-issued MacBook Pro hooked up to a 24″ HP monitor, fairly par for the course at Flickr. This is the first Mac I’ve worked on in several years and I’m a big fan. The consistency of all the software is nice to use and things just seem to work.

At home, I use a Lenovo Thinkpad R61i with Ubuntu 10.04 installed. I’ve ran Ubuntu since version 4.10 and really enjoy being able to install about any developer tool under the sun with a one-line command. It’s hard to beat the big Debian/Ubuntu repositories. This is also about my third or fourth Thinkpad laptop. They are built like tanks and are nicely supported out of the box by most Linux distributions.

I use a iPhone 3G and it gets along just fine.

And what software?

As far as software goes, my programming life revolves around programming languages, browsers, and text editors.

At work, I do almost everything in PHP and at home I’m always playing with new languages. I’m currently in the process of learning Haskell and learning how compilers work so I can one day build my own toy language. I’ve also done professional work in Python and that is still my favorite multi-purpose language where I can just get stuff done.

My main browser at work is whatever the stable version of Firefox is. Like most web developers, I use Firefox for extensions like Firebug, Web Developer, and Y!Slow. I use Chrome at home, mainly just because I don’t need a lot of bells and whistles when I’m browsing and with a smaller laptop screen, the minimal chrome really frees up some real estate.

At work, I use TextMate to write code. This was a big departure from what I normally used. I’ve always used vim and have those keyboard bindings hard-wired into my muscle memory. I don’t think TextMate is as powerful or as quick to use as vim (at least for me), but that’s not necessarily a bad thing. TextMate shines at its intended purpose, which is to just get some code out. The project navigation and function browse is really nice and having it integrated in the Mac style is a plus. I still use vim for my off-hours hacking projects though.

Other random pieces of software include: iTunes, Adium for both IM and IRC, Thunderbird for email, and Tweetie for keeping up with Twitter.

What would be your dream setup?

As far as dream setup goes, I’ve never been that picky and happy with a minimal setup. As long as I have a PC that keeps up with my work, a good keyboard, a mouse that glides, a good-sized second monitor, and a comfortable chair, I’m golden.

How the contact cache was won

You say ‘cash’, I say ‘kaysh’

Flickr has a lot of users. A lot. And most of those users have contacts, family, friends; somewhere between none and a bajillion. Or tens of thousands, anyhow. That’s a lot of relationships flying hither and yon, meaning we can’t just cache this stuff on the fly whenever the need strikes us. And strike it did.

Thus, Bo Selecta.

This project was designed to grab up a person’s contacts from anywhere in Flickrspace, and it had to be usable in bits of the site we hadn’t even designed yet. But it also had to not suck, and it had to be fast.

Supafast.

Luckily for us, we have at our disposal a shipping crate in the basement full of terribly clever little robots wearing suitable, fleshy attire and having names like Ross and Paul and Cal.

Walking into the river

As Rossbot has already covered, we spent a lot of time back-and-forthing on how we’d seed this aggregated cache all over the damned place without compromising on speed or our own general sexual attractiveness. Plus, I just wanted to use big words like ‘aggregated’ and ‘seed’.

As I’ve already mentioned above, making this magic happen at request time was not an option, so we turned to our (somewhat) trusty offline tasks system. These tasks munge and purge and generally do all sorts of wonderful data manipulation on boxes separate to the main site, in a generally orderly fashion, and do it in the background.

Offline tasks do it in the background

First up, we needed to work out what data we’d actually want to cache, which ended up being a minimal chunk useful enough for Rossbot to do whatever it is he does with Javascript that makes the ladies throw their panties on stage, and not a single byte more. We ended up with something that looks like this:

You got me.

Oh, you’re a clever one. That’s actually a picture of a fish. We really ended up with something like this:

NSIDaemail@address.comacharacter_nameareal nameaicon serveraicon farmapath aliasais_friendais_familyamagic_dust

Thus, we’re generating a bunch of contact data separated by designated control characters, and ultimately stored in a TEXT field in a database. The first time your cache is built, we actually walk your entire contact list and generate one of these chunks for each person you’re affiliated with. On subsequent updates, we use a bit of regular expression hoohah pixie dust to only change the necessary details from individuals, and write those changes back to the DB.

Big ups to Mylesbot for his help with making these tasks as efficient and as well-oiled as he is.

Speaking of updates, clearly we have to make sure we catch any changes you or your contacts make, so we have various spots around the site that fire off these offline tasks – when you update your various profile details, when you pick a named URL on Flickr for the first time, or when change your relationship with someone.

These updates have been carefully honed to work in the context of what’s changing – again, to squeeze out as much speed as we can. F’instance, there’s no need for us to tell all of your contacts that your relationship with SexyBabe43 has progressed to ‘Friend’. Unless that’s your sort of thing, but really, let’s leave that as an exercise for the reader.

All of this attention to detail has ultimately helped us eck out as much speed as possible. Seeing a theme here? So any time you’re sending a Flickrmail, searching for a contact or sharing a photo, think of the robots, and smile that secret little smile of yours, knowingly.

Trickr, or Humanising the Developers (Part 2)

Back for more, my wee little tin miners? As we say ’round these parts, Onwards!

But first thanks to Rafe, and Stephen for playing along.

[flickr_staff_buddy_icon name=mylesdgrant]
Myles, POWER USER!
Testing on dev is for the unconfident.

Macbook Pro, Textmate, scp, Firefox, Firebug, ack, Quicksilver, iTerm, vi, Mail.app.

[flickr_staff_buddy_icon name=norby]
Norby, Ops Succubus
Sleep! What is it with you people and sleep?

  • Y!-issue MBP (upgraded to Leopard w/ spaces)
  • Terminal, ssh, Safari, Firefox, Nagios, vi
  • tunnels >> VPN, have more than one RSA key if you work remotely :)

[flickr_staff_buddy_icon name=laloyd]
Paul L, Roaming (not Roman) Yeti
Gngghhhghghhh

MacBook Pro, JDK 1.6, IntelliJ Idea 7.03, iTerm, one reliable crazy Canadian-Russian.

[flickr_staff_buddy_icon name=schill]
Schill, Lil’ Javascript charmer
[ This space left intentionally blank ]

MBP, TextMate, A-grade (Fx/Safari/Opera) browsers, IE 6/7/8 + MS Script debugger via Parallels. GIMP for the odd image edit. Dell 24″.

Fun stuff: Beyerdynamic DT-880 headphones, iTunes + last.fm + SOMA FM for muzak. Finger rocket defense system. Grande dark roast in the AM.

[flickr_staff_buddy_icon name=sm]
Serguei, Ex-KGB Field Agent
Comments are lying, code tells the truth!

PC laptop. JDK 1.6, IntelliJ Idea 7.03, FAR manager, SecureCRT ssh client. kill, especially in its most radical form kill -9. And tail -f , I can watch logs for hours, it’s better than TV.

[flickr_staff_buddy_icon name=hitherto]
Simon, Totally bi(linguisticalated)
Um, no, that won’t work in French

  • MBP
  • Textmate
  • Firefox 3
  • Safari
  • Parallels (for testing in IE)
  • Apple Terminal
  • Apple Mail
  • Quicksilver

grep, vi, perl, dozens of bash shortcuts. and my personal favourite for code review : “cvs diff | mate”

Trickr, or Humanising the Developers (Part 1)

We busy little nerds of Flickr may act and smell like a bunch of psychotic monkey-bots, but beneath our filth-drenched metal exteriors beat the fleshy hearts of a thousand delicate human flowers. We feel, we love.

Ever wondered what keeps us well-oiled? I did, so I asked people, “What do you use to get the job done?”

And this, my dubious friends, is how we (rick)roll.

[flickr_staff_buddy_icon name=straup]
Aaron, Ce sera mauvais français parce que j’ai utilisé l’Internet
You’re still wrong

  • Emacs (dired-mode and shell-mode and M-x goto-line)
  • Glimpse (and alias grep grep -n -r -e)
  • Tabs and virtual workspaces

[flickr_staff_buddy_icon name=bees]
Cal, Baconmeister
Fuck off and die

  • shitty pc laptop w/ xp pro
  • twin 20″ monitors
  • noted
  • explorer
  • ff & thunderbird
  • putty / pageant / plink
  • winscp
  • cygwin
  • msys & mingw
  • wireshark
  • paint shop pro 5
  • miranda
  • calc & chamap
  • tortoise cvs/svn
  • beyond compare
  • apache/mysql/php
  • ms office w/ visio
  • itunes

[flickr_staff_buddy_icon name=revdancatt]
Dan, The Rev.
I don’t do quotes

2 Machine setup;

MacBook Pro for writing code, TextWrangler (off-white Lucida Grande 11pt font on blue background, for reduced eye-strain), Safari & Opera for (final) testing.

PC for testing, with IE6,7,blah + MS Script debugger. Most testing takes place in Firefox + Firebug (cannot live without firebug). Monitor rotated 90 degrees to give Firebug more real estate for hacking around the dom, editing js script on the fly, etc.

Extras:

[flickr_staff_buddy_icon name=dunstan]
Dunstan, He’s like, got a dog and stuff
Mistakenly included on the engineering mailing list for 505 days and counting

  • Macbook Pro
  • Textmate
  • Photoshop
  • Transmit
  • Safari
  • Firefox+Firebug
  • IE (in Parallels)
  • Quicksilver

[flickr_staff_buddy_icon name=eric]
Eric, Teenage Mutant Ninja Scripta
Please, just work

Powerbook, BBEdit, Perl scripts to manage scp+cvs+Flex+compression, Firefox w/ Firebug and Webdev toolbar, Flex 3 SDK, Terminal, nano, Parallels.

[flickr_staff_buddy_icon name=kellan]
Kellan, Rebellious off-worlder
I’d rather be building cloud castles

MBP, a hot-rodded version of Textmate, QuickSilver+Terminal.app (what’s the Finder?), Thunderbird + keyconfig for threading and archiving, SSHKeychain, grep, awk, tree, QuickProxy for Firefox (2.x), WordPress.com, last.fm, Pandora, and Adium. used to use PHPfi, but less lately.

[flickr_staff_buddy_icon name=murphy_slaw]
Murphy, Secret ops mole
… –force –yes –quiet > /dev/null 2>&1

  • MacBook Pro
  • iTerm
  • vim, kill, screen, awk, rsync, mtr, nmap, strace, gdb
  • Wireshark
  • SSH Agent
  • Thunderbird + Enigmail
  • Firefox + Firebug + SwitchProxy + Nagios Checker
  • Adium
  • Caffeine
  • Home Zone

Coming soon: more responses!