Thanks to everyone who participated in our recent survey.
I received thousands of responses, and statistically the sample size indicates a 99% confidence level that the quantitative responses (multiple choice) fall within +/- 2.5% of the mean for the entire Web Mail user population. That means the answers should fairly represent most users. There was also great participation in the open-ended questions, where you could describe what you need or would like to see in Web Mail. I added all those answers up too (thousands of open-ended answers and I read every single one) and there were a lot of repeated themes that I'll summarize and rank.
As promised, I'm publishing the full results here. We're still digesting all the information internally and formulating plans to try and better meet the top needs of our users. But you can count on a lot of this information leading to specific solutions.
The survey gathered 3 sets of useful data:
- Web Mail usage by tenure, frequency, connection type, browser, and OS.
- Web Mail satisfaction for reliability, speed, features, storage, and ease-of-use.
- Unmet needs and desired improvements.
We asked 8 required multiple-choice questions for quantitative and ranking data, and 3 optional open-ended questions for additional data on user needs and problems. Here's the results, and some comments summarizing what we learned:
First, the basic questions about Web Mail users.
If we count all the respondents who say they have used Web Mail for over 3 months, AND use Web Mail at least 3 times per week, AND use Web Mail at least as regularly as POP (Outlook or similar software), we get 77% of all the respondents. That's a high percentage of regular and heavy users providing the data. This "self-selection bias" of respondents means we are hearing from engaged users who have an opinion.
28% say they "usually" or "regularly" use Web Mail over dialup. This is a very important group that has specific needs, particularly speed optimization (more on that later).
- 15% use Safari, which is a unique browser shipped on Macs that sometimes has different behavior than IE and Firefox.
- 19% use Mac OSX.
- 9% are using a Windows version older than XP or Vista (at least sometimes, users could choose more than one answer).
These are significant population segments that must be supported.
Next we asked users about general satisfaction.
I've graphed the responses. Here's the results for all respondents combined.
Here's the results for only those users who said they usually use Web Mail over a high-speed connection in question #4.
And for the dialup users:
Both high-speed and dialup users rated "speed" and "features" the lowest overall score of the categories listed. The best overall scores were for "reliability" and "ease-of-use".
High-speed users rated features lowest.
Dialup users rated speed lowest.
High-speed users are generally a little happier about everything, with top 2 boxes averaging 79% vs. 72% for dialup.
Next we asked users to rank the importance of several enhancements we are already considering, in order to help us prioritize them.
The top ranked responses, in order, are:
- "Auto-save while writing messages so the text isn't lost if I lose my connection."
- "Never flag a message as spam if the sender is in my Address Book ."
- "Save my default font settings for writing messages."
- "Put common tasks in a toolbar across the top of the page."
Some others that were highly ranked by users are:
- "Insert pictures inline in the body of a message."
- "Allow more than 3 file attachments on outgoing messages."
- "Allow easy switching between mailboxes while logged in."
- "Improve searching and sorting in the Address Book."
We then asked three open-ended questions:
74% of respondents answered one or more of these, and 52% answered two or more. Here's the top 20 responses for each one. Some of the same general responses show up in each of the questions (we'll combine them in a minute).
|9. Without limiting your answer to the list above, what one new Web Mail feature would be most useful to you?|
|1||More personal control of spam (higher block list limit, allow blocking top-level domains, allow custom filters)|
|2||More storage space|
|3||Auto-save while writing messages so the text isn't lost if I lose my connection|
|4||The feature or preference already exists for what the user wants but they don't know it|
|5||Better rich-text compose (save font settings as default, should work on Safari, make it easier to use)|
|6||Improve the layout (make it easier, toolbar at the top, signout link at top)|
|7||Improve the speed, particularly log-in time|
|8||Allow easy switching or combined viewing of multiple mailboxes, bypassing login screen|
|9||Improve address book sorting, searching, and syncing|
|10||Be able to insert pictures in the body of new messages|
|11||Eliminate the 3-attachment limit in Web Mail|
|12||Remove advertisements or move them out of the way|
|13||Improve the search feature, including message body text|
|14||Add a positive whitelisting feature for addresses to bypass all spam filtering|
|16||Allow compose and/or read message in separate popup window(s) so I can do more than 1 thing|
|17||Improve the spell-check ( better dictionary, inline checking, allow add-to-dictionary)|
|18||Make forwarding easier and more accurate, particularly with inline images and certain message types|
|19||Support mobile devices|
|20||(3-way tie) Login without typing domain; Larger attachment limit; Don't require login every time from same computer|
|10. If we could fix one thing in Web Mail that doesn't work for you today, what would it be?|
|1||Speed, particularly log-in time|
|2||Advertising complaints or trouble caused by ads|
|3||Better control of spam (higher block limit, block TLDs)|
|4||Better rich-text compose (save font settings, support Safari)|
|5||The feature or preference already exists for what the user wants but they don't know it|
|7||Desired messages are being caught as spam|
|8||Storage space is insufficient|
|9||Autosave to prevent composed messages being lost; save text on lost connection|
|10||Forwarding trouble, particularly with inline images and certain message types|
|11||3-attachment limit imposed by Web Mail interface|
|12||User is confused about POP vs. Web Mail, can't find messages, not in sync|
|13||Can't insert pictures inline in compose|
|14||Address book feature/usability improvements|
|15||Allow saving password or keep session open even after closing browser|
|16||Allow compose and/or read message in separate popup windows so I can do more than one thing|
|17||Allow easy switching between accounts, bypassing login|
|18||Need a preference to return to Inbox on delete, rather than display next message automatically|
|19||Mobile device support|
|20||Better searching, including body text|
|11. Please briefly describe any other improvements you would like to see in Web Mail.|
|1||Remove advertisements or move them out of the way; no animated ads|
|2||More storage space at no charge|
|3||The feature or preference already exists for what the user wants but they don't know it|
|4||More personal control of spam (higher block list limit, allow blocking TLDs, allow custom filters)|
|5||Improve the speed, particularly log-in time|
|6||Improve the layout (make it easier, toolbar at the top, signout link at top)|
|7||Better rich-text compose with more fonts and features; save font settings as default|
|9||Support mobile devices|
|10||Better customer support via phone and chat|
|11||Add a positive whitelisting feature for addresses to bypass all spam filtering|
|12||Improve address book sorting, searching, and syncing|
|13||Auto-save while writing messages so the text isn't lost if I lose my connection|
|14||Improve the spell-check ( better dictionary, inline checking, allow add-to-dictionary)|
|15||Allow larger file attachments|
|16||Better searching, including body text|
|17||Allow saving password or keep session open even after closing browser (so I don't have to log in every time)|
|18||Allow compose and/or read message in separate popup window(s) so I can view more than 1 thing|
|20||Provide IMAP access|
Now let's put it all together, combining the feature questions with the open-ended responses to come up with a top 10 list of what most users want.
Before I get to the top 10, here are a couple things worth noting. In the open-ended responses, a number of users said don’t change anything, they like it a lot just like it is.
A significant number were concerned that "changes" or "improvements" might mean making things fancier at the expense of speed and simplicity, like they’ve seen some other major email providers do. Most users want Web Mail to be fast and simple, rather than trying to imitate a desktop email program inside a web browser.
The most common comparison made to a "good" example, was to Gmail, which is full-featured but still clean and uncluttered. Only a very few users would like Web Mail to be more "like Outlook" or more "client-like" and attempt to behave like desktop software (which tends to hurt performance a lot in a browser-based application). I agree, and we aren't planning to go down that path.
Also note that in all three open-ended questions, a number of users requested a feature or setting that already exists. We released an update in April (right before the survey ran) that has a completely reorganized Preferences section, making it much easier to find the setting you are looking for. If you haven't looked at it lately, click Preferences and see what is available there.
Here's the overall top 10 user needs or concerns:
- Advertising - in the way; slows me down; animation interferes; I object to it in general.
- Speed - particularly login time; top issue for dialup users; ads perceived as a contributor too.
- More personal control of spam - higher block list limit; allow blocking top-level domains; allow custom filters.
- Storage space is insufficient.
- Improve the rich text (HTML) compose - save compose font settings between sessions; don't require a toggle; don't lose text when toggling; make it work on Safari; allow inserting pictures inline.
- Improve the layout - including a toolbar at the top with major tasks.
- Auto-save text during compose and allow inline re-login if session is lost so no composed text is lost.
- Improve overall reliability.
- Allow whitelisting - so that email from my contacts never goes to Known Spam.
- Improve the Address Book - including searching, sorting, ease-of-use, and syncing.
From the list above I can tell you that we are working on doing something about all of them, and I can make a few specific comments about some upcoming changes.
- There are several changes coming soon that will improve speed. One involves a quicker login step due to a change in how our servers are set up, and that will happen in the next few weeks. We're also doing some things to make sure advertisements don't slow you down. Ads already load last on the page (except login page) for users of any up-to-date browser, so that the pages are ready to use before ads begin loading and you should be able to click anything you want without waiting. A few older browsers don't obey this, so update your browser if you can. We're also working hard to enforce file sizes on banners, and using a stricter standard for dialup users, so that they load quickly. And, we have a new layout coming later this year that should display most pages quicker.
- Advertising - other than the things mentioned above, we're also about to change the main banner placement to one many users have suggested. Up until about a year ago the banner was at the top of the page, and we moved it to the right side. That is being moved back in early June. Not only is the top placement seen as a lesser evil by many users (not as in the way), it is also smaller and will load faster. And finally, the Ad-Free Web Mail option I've mentioned before is almost ready, and it will be available to subscribers in June. More details will be provided soon.
- Items 3,5,6,7 and 10 are on our roadmap for this year and should show up in updates over the coming months. Some other requested items that didn't make the top 10 are also going to be included. We're still working on #4 (storage) but I don't have any update for now. We're working on #8 (reliability) all the time. And I'm trying to get #9 (whitelisting) implemented and should have more news on that soon.
Thanks again for your participation in the survey. You're welcome to comment here on the results or the preliminary plans I've mentioned.