The session began with demo of the "usability kit" which was developed by Máirín Duffy and Ray Strode which records the keyboard, screen, and the user's face. The box contains a DVR-style device which records all four video inputs. It's using embedded Linux. Total cost of all hardware including VGA scan converter: $800.
A set of tasks are given by the test giver to be accomplish tester. The analysis is done after-the-fact where she fills in the user's behavior when given the task to complete. The video provides actual documentation between the lag of a user click and the responsiveness of the UI (especially helpful on a web site design).
She tests a wide range of users: developers and regular users.
Máirín recommends narrowing the scope down to a very specific set of tasks. Don't just say, "I'm going to test GNOME." Narrow it down to "File Management" and then come up with a list of tasks related to that and measure those.
Ray finished the session by showing how they use gstreamer to stitch the four separate video feeds in to a single video.
The floor was opened to discussion on how to do constructive user-army sourcing of input in a way that is useful for improving usability.