allspaw sez “We ate 10 fuckin terabytes yesterday in uploads. Not bad.“
Monthly Archives: December 2008
Web Ops Visualizations Group on Flickr
Like lots of operations people, we’re quite addicted to data pr0n here at Flickr. We’ve got graphs for pretty much everything, and add graphs all of the time. We’ve blogged about some of how and why we do it.
One thing we’re in the habit of is screenshotting these graphs when things go wrong, right, or indifferent, and adding them to a group on Flickr.
I’ve decided to make a public group for these sort of screenshots, for anyone to contribute to:
http://flickr.com/groups/webopsviz/
You should realize before posting anything here, that you might want to think about if you want everyone in the world to see what you’ve got. I’ve made a quick FAQ on the groups page, but I’ll repeat it here:
Q: What is this?
A: This group is for sharing visualizations of web operations metrics. For the most part, this means graphs of systems and application metrics, from software like ganglia, cacti, hyperic, etc.Q:Who gets to see this?
A: This is a semi-public group, so don’t post anything you don’t want others to see.For now, it’ll be for members-only to post and view. Ideally, I think it’d be great to share some of these things publicly.
Q: What’s interesting to post here?
A: Spikes, dips, patterns. Things with colors. Shiny things. Donuts. Ponies.Q: My company will fire me if I show our metrics!
A: Don’t be dense, and post your pageview, revenue, or other super-secret stuff that you think would be sensitive. Your mileage may vary.
So: you’ve got something to brag about? How many requests per second can your awesome new solid-state-disk database do? You got spikes? Post them!
Machine Tag Hierarchies
something:somethingelse=somethingspecific
With apologies to Jeremy Keith
If you’re not already familiar with machine tags the easiest way to think of them is being like a plain old tag but with a special syntax that allows users to define additional structured data about that tag. If you’d like to know more, the best place to start is the official announcement we made about machine tags in the Flickr API group.
If you want to know even more, still, take a look at:
Okay! Now that everyone is feeling warm and fuzzy about machine tags: We’ve added (4) new API methods for browsing the hierarchies of machine tags added to photos on the site. These are aggregate rollups of all the unique namespaces, predicates, values and pairs for public photos with machine tags.
For example, lots of people have added exif:
related machine tags to their photos but there hasn’t been a way to know what kind of EXIF data has been added: exif:model? exif:focal_length? exif:tunablaster? Or what about all the planespotters who have been diligently adding machine tags to their photos using the aero namespace: What are the predicates that they’re tagging their photos with?
Those are the sorts of things these methods are designed to help you find. Sort of like wildcard URLs but for metadata instead of photos. Uh, sort of.
Anyway, the new methods are:
flickr.machinetags.getNamespaces
This returns a list of all the unique namespaces, optionally bracketed by a specific predicate. For example, these are all the namespaces that have an airport predicate:
# ?method=flickr.machinetags.getNamespaces&predicate=airport <namespaces predicate="airport" page="1" total="2" perpage="500" pages="1"> <namespace usage="1931" predicates="1">aero</namespace> <namespace usage="3" predicates="1">geo</namespace> </namespaces>
flickr.machinetags.getPredicates
Like the getNamespaces
method this returns a list of all the unique predicates, optionally bracketed by a specific namespace. For example, these are all the predicates that use the dopplr namespace:
# ?method=flickr.machinetags.getPredicates&predicate=dopplr <predicates namespace="dopplr" page="1" total="4" perpage="500" pages="1"> <predicate usage="4392" namespaces="1">tagged</predicate> <predicate usage="1" namespaces="1">traveller</predicate> <predicate usage="7780" namespaces="1">trip</predicate> <predicate usage="4269" namespaces="1">woeid</predicate> </predicates>
flickr.machinetags.getValues
At this point, the pattern should be pretty straightforward. This method returns all the unique values for a specific namespace/predicate pair. For example, these are some of the values associated with the aero:tail machine tag (yes, really, airplane tail models!):
# ?method=flickr.machinetags.getValues&namespace=aero&predicate=tail <values namespace="aero" predicate="tail" page="1" total="1159" perpage="500" pages="3"> <value usage="1">01-0041</value> <value usage="1">164993</value> <value usage="2">26000</value> <value usage="1">4k-az01</value> <value usage="1">4l-tgl</value> <value usage="1">4r-ade</value> <!-- and so on... --> </values>
flickr.machinetags.getPairs
Finally, the getPairs method returns the list of unique namespace/predicate pairs optionally filtered by namespace or predicate.
Rather than including yet-another giant blob of XML, here’s a pretty picture of the metro stations in Munich instead:
A few things to note
Certain namespace/predicate pairs have been special-cased to return a single value. As of this writing they are:
- geo:lat (and variations)
- geo:lon (and variations)
- file:name
- file:path
- anything:md5
If people have particular reasons for needing or wanting these we’re open to the idea but otherwise the cost of storing all the variations and the dubious uses for returning them in the first place made us decide to exclude them.
Now what?
Well, that’s what we’re hoping you’ll tell us. Machine tags have been chugging away quietly since we announced them almost two years ago and despite being a bit nerd-tastic and awkward to explain we’ve been thrilled to see how people have been finding their own use for them.
-
It turns out we have photos for 31, 594 unique Last.FM events all because people added lastfm:event machine tags to their photos.
-
Researchers at Lewis & Clark College, in Portland, have been “developing an educational collection of contemporary ceramics images using the photo sharing site Flickr” and machine tags.
-
Shortly after the launch of Google’s Street View feature Mikel Maron “took GeoRSS feeds from Upcoming, grabbed lat/long and associated that with a panoid via the RESTful ‘api’ and pulled down the images … and then uploaded the panos to flickr with machine tags“.
-
The Utata Collective uses a combination of groups on Flickr and machine tags to collect photos for their projects.
And the list goes on.
The trick with machine tags has always been to make them both invisible (or at least barely visible) to those people who don’t care about them but also as easy as tags to pick up and use for those people who do or who wonder whether they might be the tool they were looking for. One thing we didn’t do very well, though, until the release of the machine tag hierarchy APIs was give people a way to learn about machine tags. The only way to find out which machine tags people were using was to hop-scotch your way around people’s photostreams or to be part of a larger community having a discussion about which tags to use. Oops.
Which is why it was extra-fantastic when a few short days after we announced the machine tag hierarchy methods on the API list, the ever prolific and awesome Paul Mison wrote back and said:
The obvious thing to build on top of these … is some sort of graphical machine tag browser, a bit like the Mac OS X / iPod column view browser. So I did.
http://husk.org/code/machine-tag-browser.html
This is entirely self-contained in one file (except for loading jQuery from Google and (cough) the pulser from Flickr). It uses JavaScript to get a full list of namespaces, giving you the option to drill down into predicates and the values available for that namespace/predicate pair.
We’re hoping that this provides a little more raw material to play with and maybe find some magic and that you’ll tell us what comes next.
Yay!
Oh yeah, the actual API methods
- flickr.machinetags.getNamespaces
- flickr.machinetags.getPredicates
- flickr.machinetags.getValues
- flickr.machinetags.getPairs
Enjoy!
In the coming weeks we’ll also try to gather most of the blog posts and other writings about machine tags and put them with the rest of the API documentation.
[change log] Add as contact link
Change happens. On Flickr it happens quite a lot, although it tends to be just the large stuff that gets the coverage, see Improved Contacts Management over on the main Flickr blog for example.
I’ve started [change log] posts as an attempt to highlight some of the smaller changes that go on.
And… the new Add as a contact link seems like a good start.
For quite a while now you could add someone from any page that featured their buddy icon. Hovering over it and clicking the down arrow opens up a whole selection of options, including adding them as a contact. It’s still a great way of getting to pages about that user or adding them as a contact from places such as group discussions.
As of last week Eric dropped a slightly more obvious Add so-and-so as a contact link onto the photostream page.
With the extra bonus of also saying what their current contact status is should you ever decided to change it.
Sometimes it’s the little things that make a world of difference, so yay Eric!