(Flickr is hiring! Check out our open job postings and what it’s like to work at Flickr.)
Web workers are awesome. They’ll change the way you think about JavaScript.
Chris posted an excellent writeup on how we do client-side Exif parsing in the new Uploader, which is how we can display thumbnails before uploading your photos to the Flickr servers. But parsing metadata from hundreds of files can be a little expensive.
In the old days, we’d attempt to divide our expensive JS into smaller parts, using setTimeout to yield to the UI thread, crossing our fingers, and hoping that the user could still scroll and click when they wanted to. If that didn’t work, then the feature was simply too fancy for the web.
Since then, a lot has happened. People started using better browsers. HTML got an orange logo. Web workers were discovered.
So now we can run JavaScript in separate threads (“parallel execution environments”), without interrupting the standard UI stuff the browser is always working on. We just need to put our job code in a separate file, and instantiate a web worker.
Without YUI
For simple, one-off tasks, you can just write some JavaScript in a new file and upload it to your server. Then create a worker like this:
var worker = new Worker('my_file.js');
worker.addEventListener('message', function (e) {
// do something with the message from the worker
});
// pass some data into the worker
worker.postMessage({
foo: bar
});
Of course, the worker thread won’t have access to anything in the main thread. You can post messages containing anything that’s JSON compatible, but not functions, cyclical references, or special objects like File references.
That means any modules or helper functions you’ve defined in your main thread are out of bounds, unless you’ve also included them in your worker file. That can be a drag if you’re accustomed to working in a framework.
With YUI
Practically speaking, a worker thread isn’t very different from the main thread. Workers can’t access the DOM, and they have a top-level self object instead of window. But plenty of our existing JavaScript modules and helper functions would be very useful in a worker thread.
Flickr is built on YUI. Its modular architecture is powerful and encourages clean, reusable code. We have a ton of small JS files—one per module—and the YUI Loader figures out how to put them all together into a single URL.
If we want to write our worker code like we write our normal code, our worker file can’t be just my_file.js. It needs to be a full combo URL, with YUI running inside it.
An aside for the brogrammers who have never seen modular JS in practice
In development, we have one JS file per module. Let’s say photo.js, kitten.js, and puppy.js.
A page full of kitten photos might require two of those modules. So we tell YUI that we want to use photo.js and kitten.js, and the YUI Loader appends a script node with a combo URL that looks something like this:
<script src="/combo.php?photo.js&kitten.js">.
On our server, combo.php finds the two files on disk and prints out the contents, which are immediately executed inside the script node.
C-c-c-combo
Of course, the main thread is already running YUI, which we can use to generate the combo URL required to create a worker.
That URL needs to return the following:
YUI.add()statements for any required modules. (Don’t forget yui-base)YUI.add()statement for the primary module with the expensive code.YUI.add()statement to execute the primary module.
Ok, so how do we generate this combo URL? Like so:
//
// Make a reference to our original YUI configuration object,
// with all of our module definitions and combo handler options.
//
// To make sure it's as clean as possible, we use a clone of the
// object from before we passed it into YUI.
//
var yconf = window.yconf; // global for demo purposes
//
// Y.Loader.resolve can be used to generate a combo URL with all
// the YUI modules needed within the web worker. (YUI 3.5 or later)
//
// The YUI Loader will bypass any required modules that have
// already been loaded in this instance, so in addition to the
// clean configuration object, we use a new YUI instance.
//
var Y2 = YUI(Y.merge(yconf));
var loader = new Y2.Loader({
// comboBase must be on the same domain as the main thread
comboBase: '/local/combo/path/',
combine: true,
ignoreRegistered: true,
maxURLLength: 2048,
require: ['my_worker_module']
});
var out = loader.resolve(true);
var combo_url = out.js[0];
Then, also in the main thread, we can start the worker instance:
// // Use the combo URL to create a web worker. // This is when the combo URL is downloaded, parsed, // and executed. // var worker = new window.Worker(combo_url);
To start using YUI, we need to pass our YUI config object into the worker thread. That could have been part of the combo URL, but our YUI config is pretty specific to the particular page you’re on, so we need to reuse the same object we started with in the main thread. So we use postMessage to pass it from the main thread to the worker:
//
// Post the YUI config into the worker.
// This is when the worker actually starts its work.
//
worker.postMessage({
yconf: yconf
});
Now we’re almost done. We just need to write the worker code that waits for our YUI config before using the module. So, at the bottom of the combo response, in the worker thread:
self.addEventListener('message', function (e) {
if (e.data.yconf) {
//
// make sure bootstrapping is disabled
//
e.data.yconf.bootstrap = false;
//
// instantiate YUI and use it to execute the callback
//
YUI(e.data.yconf).use('my_worker_module', function (Y) {
// do some hard work!
});
}
}, false);
Yeah, I know the back-and-forth between the main thread and the worker makes that look complicated. But it’s actually just a few steps:
- Main thread generates a combo URL and instantiates a Web Worker.
- Worker thread parses and executes the JS returned by that URL.
- Main thread posts the page’s YUI config into the worker thread.
- Worker thread uses the config to instantiate YUI and “use” the worker module.
That’s it. Now get to work!
