Content Services for Mozilla examines the delicate balance between power and efficiency.
Firefox users don’t necessarily want more features on New Tab. They want more value.
Although I had always suspected as much, this was our key finding in two back-to-back studies conducted with nearly 200 participants. The first study sought to determine the degree to which users wanted to personalize their overall New Tab experience. A second was then launched to investigate how much organization users wanted, and how much direct control they preferred.
The results are equally revealing and worth sharing.
Study #1 – New Tab Controls Menu
For this we study we tested the current New Tab Controls menu against two other variations which offered either limited or unlimited customization.
Today, the New Tab Controls menu offers users a strange choice: Everything or nothing. By default, New Tab displays a search bar, the user’s top sites on the Web, and sites suggested by Firefox. In this view, the only “control” the user has is to disable those suggestions. Choosing the other menu option removes absolutely everything but the cog icon.
The first variation of the New Tab Controls menu simplified the display to reflect one New Tab page with options to show or hide both a user’s top sites and content suggested by Firefox.
A second version added to the new layout by including options that allowed a user to change the background color of New Tab, or upload a personal image.
Two groups of 48 were sourced from usertesting.com, totaling 96 participants. The first group was shown the current New Tab Controls menu against Version A; whereas the second group was shown Version B instead.
Initially, we ran a qualitative un-moderated usability study with 8 participants in the United States and United Kingdom to observe how they interacted with the new features available in the New Tab prototype. Testers were asked to think out loud as they explored the various menu options, while inputting answers to specific questions for measurable data points.
Afterwards, we ran the same study with 40 more participants (without video analysis) to supplement data gathered from the original 8. Combined, a total of 48 responses were collected.
Download the final report: UserStudy_NT_Page_Controls_V1 (782 KB)
Study #2 – Filters vs. Groups
In this study, we showed testers two different ways to organize their sites on New Tab.
The first prototype offered a menu, labeled “Your Top Sites” by default. Clicking the text launched a drop-down that allowed testers to sort fictional browsing history by recent interest category.
Selecting any of the categories displayed just the sites in the user’s history related to that interest.
The drop-down menu also provided the option to view all “top 100 recent sites”.
A second prototype showed a group of logos titled “travel.” Testers could click any logo to view the destination page in another tab, expand the group to see larger logos and page thumbnails, or edit the group contents.
Once again, two groups of 48 were invited to participate in the study, for a total of 96. Group A were shown Filters on New Tab first for a usability study, and then then Groups for sentiment analysis and ultimate preference. Group B were show Groups on New Tab first, and then Filters.
8 individuals from the United States, United Kingdom and Canada were enlisted for an un-moderated usability study. As with the previous test, participants were asked to think out loud as they tried the various features available in each prototype, while answering specific questions for data inputs. 40 additional participants provided supplemental data, totaling 48 responses altogether.
Download the final report: UserStudy_NT_Filters_vs_Groups_V1 (1.5 MB)
Study #1 – New Tab Controls Menu
Although few usability issues were observed, an interesting patter emerged almost immediately. Nearly all participants indicated that the current New Tab Controls menu was “sufficient”, and many had only a few quibbles with the functionality or presentation.
Then they were presented with either more simplified, “limited” controls, or additional controls that allow for “unlimited” customization of the New Tab background…
1. Users don’t always know what they want
Even though most testers thought the original controls were sufficient, three out of four preferred either one of the alternative versions. In other words, testers explicitly stated they wouldn’t change anything, until they saw something different-but-better, and then they wanted more… or less.
Lesson learned: People lie! They just don’t always know it. It’s our job as experience professionals to see through blanket declarations to uncover the true sentiment. (But sometimes people do tell the truth… Which is we we test things.)
2. Rethink rethinking convention
Today, a blue check-mark indicates which view of New Tab a user has selected. But it isn’t a check-box. Several testers tried to click the blue arrow to deselect, instead of rolling over to click the other option. Of course, there is a check-box underneath “Show your top sites,” so this behavior is not without precedent.
Those who indicated that they preferred either of the alternative menus probably did so in response to the restyled controls that were actual check-boxes.
Lesson learned: As a designer, it is always tempting to challenge or replace old conventions with new variations. We must be especially careful, however, because what seems very clear to us may be vague to others. “bespoke controls” are better suited for primary navigation elements because they are often big enough to be understood as single-click destinations to something important. For menus, settings, and discrete functions, it’s often better to err on the side of convention.
3. Nothing should ever be all-or-nothing
If there was one thing testers disliked the most, it was the “Show a blank page” menu option. Choosing that view removed every element from New Tab, save the New Tab Controls cog icon. In the words of one testers,
“Not sure why anyone would want that.”
Every single participant observed specifically asked for the search bar to be added to this otherwise empty page. This is maybe why so many participants loved the custom Background Image feature offered, because when they deselected “Show top sites” and “Allow suggested content,” they still saw a pretty picture.
“Background image – that’s more interesting to me.”
Lesson learned: Even taking away the custom Background Color options would afford any user literally infinite possibilities for customization. This example reinforces the sheer power one, well-conceived feature can allow limitless variety; additional controls are not always necessary to enable a satisfying experience.
Study #2 – Filters vs. Groups
This is where is gets really interesting.
Unlike other studies, participants were asked at first not to interact with anything, but to describe the first few things they noticed on or about New Tab. We simply wanted to know if testers would find the new features on their own. The responses elicited everything from the affordance of the search bar, styling of the “tiles”, suggested content options, and (sometimes) the features we hoped they would notice. In many instances, individuals described how they actually use Firefox to navigate the Web.
Then they interacted with one prototype, and then the other for comparison. Interpreting the final results was done so through the lens of the experiences mentioned above.
For me, the finding I found most profound was this:
1. A “winner” does not indicate “perfection”
As predicted, Groups was favored by testers – but not by a wide margin. In fact, a handful of participants actually liked both, equally. During video analysis, a few actually wanted to combine the two features in some way.
More importantly, the dominant reason Groups won over Filters always also shed light on the feature’s biggest flaw…
With Filters, a user must click once to launch the menu, click again to select a category, and then a third time after identifying the different, filtered sites shown. That’s terribly inefficient. With a group, a user has to click only once, but from among five additional destinations – all within the same basic tile shape. That is very definition of efficiency, and the reason stated by many as to why they preferred Groups over Filters.
The “expanded” Group view is another story. Even though most enjoyed playing around with the controls for customizing a site’s appearance on New Tab, testers otherwise felt like the expanded view was either redundant or confusing, and far less useful overall.
Understanding what a user values most is what provides the insight necessary to perfecting any feature. In this case, testers had stated that they use New Tab for one primary purpose: navigation. Whichever prototype facilitated navigation better for a particular tester was therefore almost always the version they preferred…
Lesson learned: So, don’t get in a user’s way by loading up New Tab with lots of fancy stuff. Keep it simple, light, and purposeful. Then let users decide if and how they use the features provided to make New Tab their own.
Naturally, all the features we tested were intended to invoke a noticeable reaction. I just didn’t expect the degree to which users actually enjoyed trying out the features (especially custom Background Images). At one point or another, nearly all participants made an unsolicited exclamation of joy or delightful surprise. Personally, this affirms that we’re experimenting in the right ways by exploring features that add user value above everything else.
Also, there is still much work to be done. We cannot, should not, overwhelm Firefox users with “enhancements.” Instead, we need to be purposeful and minimal.
Going forward, I believe that Content Services – and Mozilla at large – now have a clear mandate…
Wherever possible, reduce the number of steps a user has to make to make it personal.
Darren Herman, VP of Content Services
Kevin Ghim, Group Product Manager
Justin Terry, Product Manager