Content Services for Mozilla examines the fundamentals.
As New Tab for desktop users continues to evolve, those of us who create features for this core Firefox product must be forever mindful of user sovereignty. Although it is foremost a simple navigation tool, the current experience lacks any form of user customization for individual sites. To this end, the Content Services team recently conducted a study examining both the usability and user sentiment regarding more advanced controls over the sites shown on New Tab.
For this study, prototypes were created to depict new controls that would allow users to add, edit, or delete any site they see on New Tab. These were developed for two primary groups: 1.) current Firefox Desktop users and 2.) new users who have not tried Firefox within the last six months.
In addition, new controls for Suggested Sites and related contextual labels were introduced for current Firefox users (24 of 48 testers). Discrete controls enabled these users to edit their individual category preferences for all related suggestions content on New Tab.
Initially, usertesting.com ran a qualitative un-moderated usability study with 8 participants (4 current Firefox users, 4 new) in the United States and United Kingdom to observe how they interacted with the new features available in the New Tab prototype. Testers were asked to think out loud as they explored enhanced controls for Top Sites and Suggested Sites, while inputting answers to specific questions for measurable data points.
For this first test, we wanted to know:
- Would testers notice these features on New Tab?
- How easy or difficult were they to use?
- Were they at all interested in them as permanent features?
- What roadblocks might users encounter when deleting, editing, and adding a tile, or updating their Suggested Site category preferences?
After completing round one, we ran a second study with 40 additional participants – this time without video analysis – to verify data gathered from the original 8.
Combined, a total of 48 responses were collected and analyzed to ensure that the results were 90% accurate or better.
Because hundreds of millions use Firefox every month, the “audience” for this study was intentionally general. The only requirements we gave usertesting.com participants were as follows:
Current users: People who use Firefox as their primary or secondary browser, at least a few times a week.
New users: People who have not used Firefox for at least 12 months
Locations: US & UK
Languages: English only
Mac users: Mac OS 10.6 or newer
PC users: Windows 6 or newer
Other requirements: Access to a reliable, high-speed internet connection
The original 8 testers provided a number of key insights that were then confirmed by the other 40 responses:
Insight: While many testers rolled over sites shown in the prototype, most did not interact with any new controls on their own.
Analysis: In an effort to simplify the New Tab interface, site-specific controls were made available on rollover by clicking a cog icon. By doing so, however, users may not notice the icon, and will therefore fail to discover any additional controls available to them. Furthermore, when prompted to interact with the menu, several testers explicitly asked for the “X”/delete icon to display in the left corner when rolling over a site, which partially solves the problem. Discover-ability could be further addressed by increasing the contrast of the icons to attract more attention, or showing a tool-tip the first time a user interacts with a site.
Insight: Nearly all testers found the new controls generally easy and intuitive to use.
Analysis: When prompted to do so in the study, nearly all testers were able to add, edit, or delete sites on New Tab on their first attempt, and without much difficulty. In usability terms, we could not have hoped for better results, as this affords us a high degree of confidence that these features would be intuitive for the vast majority of Firefox users if we deployed them today.
Insight: Most testers were interested in seeing these features on Firefox New Tab.
Analysis: In fact, the majority of participants expressed appreciation for being able to choose how a top site or Suggest Site was displayed on New Tab – whether as a logo, the homepage, or destination page. When asked to rate their likelihood of using these controls on their actual New Tab page, 41 responded with “likely” (13) or “very likely” (28). These findings confirm our hypothesis (derived from previous research) that Firefox users want more direct control over their New Tab experience, including how their sites are presented. As such, future experiments should explore the degree of customization users actually want.
Insight: Testers did not find the controls related to the interest category especially intuitive, but useful.
Analysis: Of the 4 initial participants who were shown the prototype for current users (which shows a Suggested Site), 2 of them did not expect the related interested category link to provide more control over their preferences. On the plus side, all testers found the controls to edit their interest categories for Suggested Sites easy to use. When combined with raw data from the second test, 20 of the 24 participants indicated that they were either “somewhat likely” or “very likely” to use these controls on Firefox. We interpret this to mean that users would appreciate more customization of Suggested Sites, even if the user value perceived will depend on the individual.
We already knew Firefox users wanted more control over their New Tab experience. Now we have a strong indication that further control over their top sites and Suggested Sites would be well-received. Together with findings from previous studies for New Tab and other Firefox products, the data shows that users are more receptive to suggested content than we initially thought. We now suspect that users may also want to organize New Tab according to their own preferences, instead of Firefox organizing New Tab on their behalf.
What we don’t know is how much of a connection users expect or want between their natural browsing behavior, and the content recommendations Firefox may show them based on inferred interests. Clearly, more research is required to determine how much New Tab should be a utility, verses a tool for discovery.
Darren Herman, VP of Content Services
Kevin Ghim, Group Product Manager
Patrick Finch, Marketing & Communications Director
Special thanks to Alexandra Michaelides, User Research Specialist at UserTesting.com who, after providing her with a draft test plan, ran the study on their platform and provided the detailed findings this post is based upon.