Lightweight Layout Management in Flash

Resizable apps are groovy. Users want content to fill the screen, and as the variance in screen resolutions has grown this has become more and more important. To make this happen you can either write manual positioning code to place and size items on window resize, or you can define a hierarchy, some properties on elements of the hierarchy and have a layout engine do the work for you. Up until a few years ago, there were no good pre-built solutions to do this in Flash so the only option was to roll your own framework. Nowadays there are several floating around, including this one from the flash platform team at Yahoo!, and of course there is the Flex/MXML route.

XML based UI markup seems to be all the rage these days, but what I found personally through my experience building the jumpcut tools is that this “responsible” approach has drawbacks. A hierarchy of HBoxes and VBoxes can be a lot of overhead for simple projects, and can become unwieldy and rigid when you are trying to develop something quickly. If you are doing anything weird in your UI (overlapping elements, changing things at runtime) it can also feel like trying to fit a square peg in a round hole.

The Best of Both Worlds

So is there any way to get at a sweet-spot, something quick and flexible, yet powerful? The first thing that probably comes to mind if you are a web developer is the CSS/HTML model. In fact, I think would be a rather good option if not for one missing piece, the unsung hero of the web: the HTML rendering engine. If anyone wants to implement Gecko or WebKit in ActionScript I think that would be a fun project, but because the web ecosystem evolved with the markup user and not the layout engine implementer in mind it’s rather complicated. There is also the fact that this model is based on the assumption of a page of fixed width but infinite height which doesn’t exactly suit our purposes.

What we can borrow though is the philosophy of good defaults and implicit rather than explicit layout specification. When you are building out a layout with HTML/CSS you can just drop stuff on the page and things more or less work and resize as you’d expect, then selectively you can move things around / add styling and padding and so on. To do something similar we can take advantage of one of the nice new features of Actionscript 3, the display hierarchy, which allows us to easily traverse, observe, and manage all the DisplayObjects at run time, without keeping track of them on our own as they are created. This way we can recurse through the tree, look for the appropriate properties, and apply positioning and sizing as necessary. If the properties don’t exist we’ll just leave things alone. When you start your project you can just write code to position stuff manually, then as things get more complex and start to solidify you can begin using layout management where it makes sense. I’m a bit of a commitment-phobe myself, so this really appeals to me; what’s especially nice is that you easily plug this in to any existing codebase.

Some Code

So, to get this working all we really need is a singleton that listens to the stage resize event, traverses the tree and places/sizes objects. Mine looks something like this:

  
/*..*/
stage.addEventListener(Event.RESIZE, this.onResize);
/*..*/
 
function onResize(e:Event){
	this.traverse(root);
}
 
function traverse(obj:DisplayObject){

	for (var i=0; i  < obj.numChildren; i++){

		var child = new LayoutWrapper(obj.getChildAt(i));
		if(child.no_layout)
			continue;
	   
		//layout code to determine how to place and size children
		//child.x = something
		//child.y = something


		child.setSize(W, H); // pre-order





		this.traverse(child);

		child.refresh(); // post-order
	   
	}
}


Two things of note here.
First is the LayoutWrapper which I use to set defaults for properties and functions that don’t necessarily exist on all objects, so sort of like slipping in a base class without having to alter any code. I use flash_proxy to make this easier, but you don’t have to do that. One thing you will want to do is store dummy values for width and height (something like _width and _height maybe as a throwback) so you can assign and keep track of the width and height of container elements without actually scaling the object. You’ll also want to decide how you want to handle turning on / off the layout management and what defaults make sense for you. In the above code I’m doing opt-out, using a no_layout variable to selectively turn off layout management where I don’t need it.
Second, I have two hooks for sizing, setSize on the way down, and refresh on the way up. This is useful in some cases because what you want to do at higher levels may depend on the size of children at lower levels which can change on the way down.
The layout code itself is outside the scope of this blog post, but I’ve found that a vertical stack container, a horizontal stack container, padding, and spacing is all I’ve needed to accomplish just about any UI. Of course, you can and should implement any layout tools that are useful to you; different projects may call for different layout containers and properties.

Some Fun

One thing that we are doing by using this approach is pushing the specification of the display tree into run-time. The analogy with HTML here is the part of the DOM that is generated by visible source vs the part of the DOM that is generated by JavaScript.
Going all cowboy like this does have its drawbacks. Specifically it can get a bit confusing as to what the tree looks like, or why this or that is not showing up, where is this Sprite attached, etc. Web developers have the Firebug inspect tool for a very similar problem, which I’ve found rather handy. We can get a poor-man’s version of something like this by writing some code that pretty prints the tree under an object on rollover or use the Flex debugger, but, depending on the size of your project, this can get unwieldy rather quickly.

As an excuse to play with the groovy new flare visualization library recently I tried pointing it at the display tree. The result is rather useful for debugging, especially if you have a lot of state changes / masking / weirdness (plus you look like you are using a debugging tool from the friggin future). Some nice things I’ve added: on roll over the object gets highlighted, I can see an edit some basic properties and see them update live. There’s quite a bit of room here to do something Firebug like, with drag and drop reparenting and the like (but be careful, not too interesting or you’d wind up reimplementing the Flash IDE).

If you’d like to try this yourself, this is the code to populate the flare data structure with the display tree:

public function populateData(){
	// create data and set defaults
	this.graph = new Tree();
	var n = this.graph.addRoot();
	n.data.label = String(root);
	this.populateDataHelper(n, root);
}

 
public function populateDataHelper(n, s){

	try{ // need this in case of TextFields
		var num_children = s.numChildren;
	}
	catch(e:Error){
		return;
	}
	
	//limit in case you have extremely long lists 
	//which can dominate the display
	num_children = num_children < 20 ? num_children : 20; 

	for(var i=0; i  < num_children; i++){
		var child = s.getChildAt(i);

		if(
		child != this //don't show the layout visualizer itself!
		&& child.visible //don't show items that are not visible
		){
			var c = this.graph.addChild(n);
			c.data.sprite = child;
			c.data.label = String(child);
			this.populateDataHelper(c, child)
		}
	}
  
}

Then just apply a visualization to this tree; I've been simply using the demo examples which has been sufficient for my uses, but obviously you can write your own as well.

Here it is in action on the flickr slideshow:

Boundaries, a tool to explore Flickr’s shapefiles

A few weeks ago we released our shapefiles via the API, and while most people were excited, some folks were a bit confused about what it all meant. Which is why Tom Taylor’s beautiful Boundaries application is so exciting. It helps you visualize the Flickr community’s twisty changing complex understanding of place.

You throw in a place name and it’ll attempt to find that place’s neighbours, using Yahoo GeoPlanet. For each of those neighbours, it uses the WOE ID to fetch and render the polygon from shapefile that Flickr returns.Tom Taylor: Flickr boundaries, now free to play with

Boundaries

5 Questions for Gustavo

Following our second interview Kellan snuck in 5 Questions for Paul Mison before I managed to tap Jim’s suggestion of Gustavo.

I think Drift Words sums Gustavo up nicely with “Although he’s far too clever, he makes up for it by using his polymath powers for good.” … and charts and stunning graphs. It’s always a pleasure to see what Gustavo comes up with.

And on that note …

1. What are you currently building that integrates with Flickr, or a past favorite that you think is cool, neat, popular and worth telling folks about? Or both.

Gustavo: First of all I must say: I’m not really a flickr tool developer. The only real tools I created are the quite minimalistic FlickRandom and Contact Crossing (very kindly hosted by Jim Bumgardner). Rather, I use flickr’s API to mine flickr’s database for interesting information, usually leading to some visualizations or at the very least a bunch of graphs.

The FlickrVerse, April 2005 poster: flickr's social network

One such visualization led to a browser for exploring related groups. Not a full-fledged tool since it explores static and outdated data.

In other words, what I do with flickr’s API is not tool development but analyses, trying to figure out the structure and dynamics of flickr’s social network, group content and participation, etc. The most recent analysis I performed attempted to understand how exposure to photo content flows through the contact network. Starting from a basic pattern combining three types of network relations, I quantified the importance of a user’s social network in determining his or her exposure to content of interest.

2. What are the best tricks or tips you’ve learned working with the Flickr API?

Gustavo: Cache, cache, cache. Depending on what you want to do, you might find yourself retrieving certain bits repeatedly, and you’ll definitely want to build a good local cache. Be careful though, as it’s all too easy to underestimate the size and complexity of the data. Your cache might unexpectedly morph from a speed asset to a sluggish monster.

Another potential problem from underestimating the size and complexity of the data is that you might suddenly discover that your code issued many more API calls than you expected. Remember to play nice, and keep an eye on your API key usage graphs.

Expect unexpected responses: It’s always good to make the code smart enough to realize the server’s response is not exactly what was expected, be it in completeness, format, special characters in free text, etc.

Finally, still on the theme of unexpected responses: Grow a thick skin, as there will always be someone who will misinterpret your motives for developing a tool that “uses” their information.

3. As a Flickr developer what would you like to see Flickr do more of and why?

Gustavo: Let me suggest two: one very simple, the other very complex.
The simple one is a “no special perms” authentication level. For some purposes, as a developer I’d like to have the user authenticate for the sole purpose of knowing who they are – I might want to give certain results for their eyes only, or someone else might want to allow certain actions to happen just once per user, etc. At the moment, the minimal level of authentication requires that the user entrust the developer with access to private data. Many users will rightfully decline giving such access, and as a developer I’d rather not have to even request and disclaim, when I don’t need it.

The more complex one requires a bit of background. Almost four years ago, there were various discussion threads on the topic of finding interesting content. Back then I took a stab at using network information to discover interesting photos – first looking for people who post similar stuff, then looking for people who share faved photos (which striatic called neighbors) and finally, actual photo suggestions.

neighbors

As an aside, the algorithm worked quite well even for people with a hundred or so favorites (I had just 21 faves!); people today have many thousand favorites – a very rich data set to start from. I went on to produce suggestion lists for many people, but that was always me, manually running scripts. Since then, other developers created interesting tools for content-driven content discovery, including Flickr Cross-Recommendations, inSuggest and flexplore.

flexplore 2.1 beta

The reason I never tried to create a stand-alone tool for this is that I quickly realized that to make it work reasonably fast, I essentially needed to replicate flickr’s database. If you take all my favorites, then list all the people that faved those photos, and finally enumerate all their favorites… and you then repeat this exercise for whoever requests it, you either need to use many thousand API calls per visitor (and wait!), or you need to create a huge cache, covering unpredictably disparate segments of flickr’s database. A massive cache that would grow stale very quickly unless people used the tool continuously. If you try to include extra information into the scoring scheme (say tags, color data, group membership…) the network use and storage requirements grow even worse. In other words, this project appears to be completely unsuitable for a developer without direct access to flickr’s database… and flickr’s content suggestion system (Explore) doesn’t provide any personalization tools: we all see the same.

The only effective solution I can suggest (beyond flickr developing a personalized Explore, or flickr sandboxing third party developers) is the creation of high level API calls, embodying some method of complex querying into the database. For example: A higher level API call could accept a list of users as query, and return a list of photos sorted by the number of people (from the query list) who faved them. With the current API, one has to retrieve the full favorites list for each person, collate the results, and discard the vast majority of the information that was transferred. A higher level API method could make such intermediary results invisible and save much bandwidth, latency and data replication.

Hey, you asked. ;)

4. What excites you about Flick and hacking? What do you think you’ll build next or would like someone else to build so you don’t have to?

Gustavo: There’s a number of things that attract me to flickr hacking. There’s of course the vast and ever-growing amounts of data, and the fact that it’s not “more of the same”: The diversity of data types (photos, users, metadata, groups, etc) and relations make this complex system fascinating to study. There is also the human side: People really care about their stuff. People get excited, for example, when they recognize their user icon in a visualization of the social network, and immediately want to explore around. Last but not least, I really appreciate the openness of the flickr API. I’m amazed that such wealth of information is shared so freely!

Free association, first page

As for “what next”, I have a couple of ideas. I’ll only hint that the word association analysis was a fun first step. :)

5. Besides your own, what Flickr projects and hacks do you use on a regular basis? Who should we interview next?

Gustavo: dopiaza as Utata‘s architect.

Dan: Thank you, Gustavo. Next up (unless Kellan gets in first!) dopiaza.

Images from GustavoG,
striatic and malanalars.

On UI Quality (The Little Things): Client-side Image Resizing

Make it good.

As front-end engineers, our job is ultimately to produce and deliver the front-end “experience.” That is, in addition to the core service (eg., a photo sharing site) which you are providing, you are also responsible for replicating and maintaining the quality of the visual aesthetic, including attention to detail in your UI.

Making it really good.

It’s “the little things” – small layout changes, single-pixel margin tweaks and color fiddling done as part of this process – that can sometimes seem overly nit-picky and risk being overlooked or sacrificed in order to meet deadlines, but the result of this effort has value: It’s quite obvious when UI polish has been done, and it’s something everyone can appreciate – even if they don’t know it.

Getting the fine details down “just so” can take extra time and effort, but in my experience has always been justified – and whenever your work is under a microscope, attention to detail is a wonderful opportunity to make it shine.

An exercise in nit-picking

Resized images between different browsersWith a recent home page redesign on Flickr, we found ourselves needing to show thumbnails for photos at a 48×48 pixel square size, smaller than the standard 75×75 square (currently the smallest available from the servers.) Interestingly (and not surprisingly), browsers differ in rendering images at non-native sizes, with some providing noticeably better results than others. Internet Explorer 6 in particular looked to be the least smooth when scaling images down, but some interesting workarounds are available.

Tweaking image resampling in IE

IE 7 supports high-quality bicubic image resampling, producing results akin to what you’d see in most image editing programs. It’s disabled by default, but you can enable it using -ms-interpolation-mode:bicubic;. It’s probably safest to apply this only to images you are explicitly showing at non-native sizes.

IE 6 is a riskier proposition, but can show improved image resizing when the AlphaImageLoader CSS filter is applied, the same filter commonly used for properly displaying PNGs with alpha transparency. For example, filter:progid:DXImageTransform.Microsoft.AlphaImageLoader (src='/path/to/image.jpg', sizingMethod='scale');. While there is no transparency to apply here, the resizing method applied gives a higher-quality result.

Further investigation

Resized images between different browsersAfter looking at IE 6 and 7, I was curious to see how other browsers handled image resizing. Opera is a little rough around the edges relative to others, but overall the quality of scaled images is pretty good.

Findings

UI quality is an ongoing process, the result of a shared interest between design and engineering groups in nit-picking, always trying to make things better. This particular image resizing example came out of a home page redesign, but was by no means the first use of it on the site. Despite the occasional limitations and peculiarities of the client-side, one of its upsides is its flexibility: You can often find creative solutions for just about any quirk.

EXIF, Machine Tags, Groupr, and more Paul Mison

After we published Paul’s interview last week, he wrote in to let us know Groupr had been dusted off and was alive and well again. And followed it up with a post on machine tags, and automated EXIF extraction, two of our favorite topics here at Code.Flickr:

Why bother with such a thing? Flickr will extract EXIF metadata, but it won’t allow you to do any aggregate queries on it. By extracting all the data from my photos into machine tags (and a local SQLite database), it becomes possible to point people at all the photos taken at the wide end of my widest lens, or those taken with a particular make of camera (and to do more complex queries locally).Flickr, EXIF, Machine Tags

5 Questions for Paul Mison

One of the key goals for this blog when we launched it 9 months ago was to be a channel for the voice of the Flickr development community. Most importantly all the amazing developers building on our APIs. Which is by way of introducing our third developer interview here at code.flickr, our first was with Neil Berkman of Photopholow, while our 2nd, and first in the 5 questions format was with Jim Bumgardner a few weeks ago.

We’ll be posting an interview from GustavoG (as tapped by Jim) soon, but in the mean time I want to post an interview from Paul Mison aka blech. Paul is an active participant in the Flickr API group, and I’ve personally been using and loving his new project SnapTrip. And he likes rainbows, which makes him good by us.

A Trip Underground

1. What are you currently building that integrates with Flickr, or a
past favorite that you think is cool, neat, popular and worth telling
folks about? Or both.

Paul: My current main project is snaptrip, which works with Dopplr, a website for sharing personal travel information, as well as with Flickr. It helps you to find photos associated with a trip listed in Dopplr, to label them so that Dopplr can find them easily later on, and also lets you geotag them (if you haven’t already). I’ve also written a Greasemonkey user script so that you can see when other people’s photos were taken during a Dopplr trip.

Flickr photos in snaptrip

In the past I’ve mainly written command line tools; I used a script to migrate images out of my previous home-brewed image hosting to Flickr, and I have another that applies machine tags for EXIF properties (since tags make it easy to find photos; here’s all my photos with a focal length of 50mm). My great lost project was a web application called groupr that would let you see all the photos in groups you’re a member of, but unfortunately the platform I built it on was withdrawn, and I should rebuild it elsewhere (possibly on Google’s App Engine, like snaptrip; possibly not).

2. What are the best tricks or tips you’ve learned working with the
Flickr API?

Paul: The API Explorer is wonderfully useful, and it should be everyone’s first port of call when developing an application. Beyond that, I’d advise picking an API framework that does the tricky things (notably, user authentication – I’ve handcoded it myself, and it can be fiddly), but otherwise gets out of your way.

snaptrip uses Beej’s Python Flickr API, and I’ve previously used flickraw (for Ruby) and Flickr::API (for Perl). All three are nice and minimal, so that when new API methods are added, you don’t have to update your library. flickraw uses JSON internally, too, which is very nice if (like me) you lean towards dynamic, rather than static, languages.

For Greasemonkey scripts, this post from mortimer? about using API calls in GM scripts is great.

3. As a Flickr developer what would you like to see Flickr do more of
and why?

Paul: Well, a relatively minor request would be for JSON in the API Explorer. More seriously, there’s an entire class of methods I wish existed for groups. For example, the only way to track conversations at the moment is the group RSS feed, which isn’t segregated by thread. There’s no way to find out a group’s administrators or moderators. A final example is that the queue of photos awaiting approval isn’t exposed to the API. While I’m not sure I’d have used all of these in groupr, some of them would have been very handy.

Another small request would be for more extras, especially a few method-specific ones. In particular, I’d love to see ‘favedate’ on flickr.favorites.getPublicList.

4. What excites you about Flickr and hacking? What do you think you’ll
build next or would like someone else to build so you don’t have to?

Paul: For all that my answer to the last question was a demand for more methods, Flickr is exciting both because of the wealth of photography there, and the richness of the methods of getting at it. The geographical data, and access to it, that has emerged over the last year is really interesting, and I’d love to do something with it (and intend to, somehow, in snaptrip). However, for a complete new project, I’ve been poking for a year or so at making your Flickr favourites look a bit nicer, and maybe within another year I’ll actually have something out publicly. (I really need a simple job queue; anyone?)

5. Besides your own, what Flickr projects and hacks do you use on a
regular basis? Who should we interview next?

Paul: Well, as a Mac user, I’m a fan of Fraser Speirs Flickr Export for getting my images onto the site in the first place, and if I upgrade to an iPhone I look forward to playing more with Exposure. On the web, Dario Taraborelli’s Group Trackr is very nice, and I use fd’s Flickr Scout every now and again to see if anything’s hit Explore.

However, I think the most impressive recent project I’ve seen is a desktop app written using Adobe’s Air, called Destroy Flickr, by Jonnie Hallman. There were a lot of subtle UI techniques to hide the latency inherent in talking to a network service, so I think he’d be a great choice for an interview.

Kellan: Thanks Paul! And y’all be looking out for an interview with Jonnie and his oddly named project.