(Last Updated on )
That nice Marcy Sutton asked me to test and give feedback about a new product she’s involved with called Evinced, which describes itself as an “Enterprise grade digital accessibility platform for modern software development teams”. Quite what “enterprise grade” means is beyond me, but it’s basically software that can crawl a website from a root domain and check its code against some rules and report back. There are similar tools on the market, and I’ve recently been working with a Client to integrate Tenon into their workflow, so wanted to compare them.
“Now hang on!”, I hear you say. “automated tools are terrible!”. Well, yes and no. Certainly, overlays etc that claim to automatically fix the problems are terrible, but tools that help you identify potential problems can be very useful.
It’s also true that automated tools can’t spot every single accessibility error; they can tell you if an image is missing alternate text, but not that <img src=dog.png alt=”a cat”> has useless alt text. Only a human can find all the errors.
However, many errors are machine-findable. The most-common errors on WebAIM’s survey of the top million homepages are low contrast text, missing alternative text, empty links, missing form input labels, empty buttons and missing document language, all of which were found by running automated tests on them (which, presumably, the developers never did before they were pushed to production).
I personally feel that a good automated scanner is a worthwhile investment for any large site to catch the “lowest hanging fruit”. While some things can’t be automatically tested, other things can, and other aspects live in a grey area depending on the rigour of the test.
For example, a naive colour contrast test might compare CSS color with background-color, and give a pass or fail; a less naive test will factor in any CSS opacity set on the text and ignore off-screen/ hidden text. A sophisticated contrast test could take a screenshot of text over an image or gradient and do a pixel-by-pixel analysis. To do a test on a screenshot would require actually rendering a page. Like Tenon, Evinced doesn’t just scan the HTML, but renders the pages in a headless browser, which allows the DOM to be tested (although I don’t believe it tests colour contrast in this way).
Evinced uses Axe Core, and open-source library also used by Google Lighthouse. It also contains other (presumably proprietary secret-source) tests so that “if interactable components are built with divs, spans, or images – Evinced will detect if they are broken”.
The proof of the pudding with automated site scanners is how well they report errors they’ve found. It’s all very well reporting umpty-squllion errors, but if it’s not reported in any actionable way, it’s not helpful.
Like all the automated scanners I’ve tried, errors are grouped according to severity. However, if those categories correspond with WCAG A, AA and AAA violations, that’s not made clear anywhere.
It’s a fact of corporate life that most organisations will attempt to claim AA compliance, so need to know the errors by WCAG compliance.
One innovative and useful reporting method is by what Evinced calls component grouping: “Consolidates hundreds of issues to a handful of broken code components”.
With other scanners, it takes a trained eye to look through thousands of site-wide errors and realise that a good percentage of them are because of one dodgy piece of code that is repeated on every page. Evinced analyses pages and identifies these repeated components for you, so you know where to concentrate your efforts to get gross numbers down. (We all know that in the corporate world, a quick fix that reduces 10,000 errors to 5,000 errors buys you time to concentrate on the really gnarly remaining problems.)
There’s a vague suggestion that this grouping is done by Artificial Intelligence/ Machine Learning. The algorithm obviously has quite clever rules, and shows me a screenshot of areas on my pages it has identified as components. It’s unclear whether this is real Machine Learning, eg whether it will improve as its corpus of naughty pages gets larger.
I don’t recall signing anything to allow my data to be used to improve the corpus; perhaps a discount for not-for-profits/ time-limited demos could be offered to organisations allowing their data to be added to the training data, if indeed that’s how it “learns”.
Tenon has some clunkers in its web interface (for example, it’s hard to re-run a previously defined scan) because it’s most commonly accessed via its API rather than web back-end.
Evinced makes it easy to re-run a scan from the web interface, and also promises an API (pricing is not announced yet) but also suffers from its UI. For example, one of my pet peeves is pages telling me I have errors but not letting me easily click to see them, requiring me to hunt. The only link in this error page, for example, goes to the “knowledge base” that describes the generic error, but not to a list of scanned pages containing the error.
(After I gave feedback to the developers, they told me the info is there if you go to the components view. But that requires me to learn and remember that. Don’t make me think!)
There’s also terminology oddities. When setting up a new site for testing, the web interface requires a seed URL and to press a button marked “start mapping”, after which the term “mapping” is no longer used, and I’m told the system is “crawling”. Once the crawl was complete, I couldn’t see any results. It took a while for me to realise that “crawling” and “mapping” are the same thing (getting a list of candidate URLs) and after the mapping/ crawling stage, I need to then do a “scan”.
A major flaw is the ability to customise tests. In Tenon I can turn off tests on an adhoc basis if, for example, one particular test is giving me false positives, or I only want to test for level A failures. This is unavailable in Evinced’s web interface.
Another important but often-overlooked UI aspect of these “Enterprise” site scanners is the need to share results across the enterprise. While it’s common to sell “per-seat” licenses, it’s also necessary for the licensee to able to share information with managers, bean-counters, legal eagles and the like. Downloading a CSV doesn’t really help; it’s much more useful to be able to share a link to the results of a run and let the recipient investigate the reports and issues, but not let them change any configuration or kick off any new scans. This is missing in both Evinced and Tenon.
The system is currently in Beta and definitely needs some proper usability testing with real target users and UI love. One niggle is the inaccuracy of its knowledge base (linked from the error reports). For example, about the
footer element, Evinced says
Since the <footer> element includes information about the entire document, it should be at the document level (e.g., it should not be a child element of another landmark element). There should be no more than one ><footer> element on the same document
This is directly contradicted by the HTML specification, which says
The footer element represents a footer for its nearest ancestor sectioning content or sectioning root element… Here is a page with two footers, one at the top and one at the bottom… Here is an example which shows the footer element being used both for a site-wide footer and for a section footer.
I saw no evidence of this incorrect assumption about the footer element in the tests, however.