As of February, Microsoft reported 1.2 billion Office users, which translates to roughly 500 million to 600 million Outlook users. Even with a growing iOS user base, Outlook has a significant market share, especially within the corporate space. With Outlook 2016 arriving soon, and the failure of the Fix Outlook campaign resulting in Outlook continuing to use Word instead of WebKit for email rendering, email QA remains a necessary evil. Though Outlook often gets picked on for its quirks and nonconformity to other email client standards, it is hardly the only email client that is difficult to code and test for.
Many things can affect the end user experience for email — from platform and email client, to screen resolution and app versions. The possible permutations that exist out in the wild make it unreasonable to support everyone, so concessions and compromises must be made and priorities must be set. The process can be cumbersome as a number of factors, such as metrics, target audience, technological restrictions and resources, must be accounted for.
While internal metrics on target audiences should take precedence, Litmus provides a good starting point for what should take priority in support and testing. Litmus, along with other email blogs like Email on Acid, Campaign Monitor and Email Monks, all provide regular updates to changes within the email industry. If you don’t know which email clients to support, it can be helpful to have the dev and QA teams work together to test functionality. Campaign Monitor also provides a decent guide to the basics.
When deciding on support and testing priority, OS version should also be factored in. As of April 2015, 81% of Apple devices are on the latest iOS 8, which is great because, in general, more recent versions indicate greater support for new features and CSS. Adoption rates can be slow, though. More than 41% of Android devices are still using 4.4, even though Android Lollipop (5.0) has been out since November 2014. And many devices cannot be upgraded to 5.0 yet (or ever, in some cases). Supporting and testing older combinations can be time-consuming for a shrinking population of users, so think carefully about supporting something like Internet Explorer 7 or 8.
These are not the only aspects to weigh when deciding support and testing standards. Any number of issues can influence QA. Making sure everyone on the team knows the reasoning behind QA decisions is tough, but necessary for a satisfied client and an excellent end user experience.