In the latest A List Apart article, Testability Costs Too Much, Gian
Sampson-Wild points out why she believes WCAG 2.0’s tenet of testability to be fundamentally flawed.
The article’s example of how Guideline 1.1.1. (relating to text alternatives to non-text content) was watered down
for testability purposes is interesting. However - and maybe I’m missing the point here - I don’t quite follow why the form that’s now in the latest working draft is supposed to be bad:
All non-text content has a text alternative that presents equivalent information, except for the situations listed below.
Gian points out the apparent problem of consistently applying testability throughout the entirety of the guidelines:
There are many instances in WCAG2 where success criteria are actually not testable—and the Working Group knows it. In Bugzilla, the Working Group’s issue tracking system, there is a tracked issue lodged by three Working Group members that reads: “In particular, the current wording [of WCAG2] does not seem testable. Words such as, “key,†“consistent,†“predictable,†“inconsistent,†and “unpredictable†are subjective.†Yet these terms have been used throughout WCAG2—there’s even an entire guideline that rests one of these subjective, non-testable terms:
Make Web pages appear and operate in predictable ways.
Now, despite the use of subjective terms, I’d still maintain that those success criteria are actually testable in the context of human testing:
Reliably Human Testable: The technique can be tested by human inspection and it is believed that at least 80% of knowledgeable human evaluators would agree on the conclusion.
The WCAG Samurai Errata are mentioned as an example of a set of guidelines that don’t rely on testability but do give developers clear instructions on how to comply with relevant requirements
. Let’s look at two examples from the Samurai:
Guideline 2. Don’t rely on colour alone [...] For actual content, do not set one confusable colour on top of another.
The Samurai mention confusable colour
, but then just give an example of red type on a green or black background
. Who says what is and isn’t a confusable colour combination? And, without a clause on testability, are the Samurai errata not more ambiguous and open to interpretation?
Guideline 3. Use markup and stylesheets and do so properly [...] Requirement to use HTML semantics [...] For ambiguous cases, you must use the elements and attributes that most closely fit your content even if other authors would disagree. [...] Other authors’ disagreement with your choices is not relevant to these errata. (emphasis added)
This almost sounds like “anti-testability”…but how is that a clear instruction
?
Don’t get me wrong, Gian’s point that, because of testability, many valid guidelines and techniques - particularly for users with cognitive disabilities - did not make it into, or were removed from, WCAG 2.0 is indeed valid. The fact that, at least in part, testability was probably pushed through in the WCAG Working Group by those members (stakeholders?) involved in producing authoring/testing tools is deplorable, and certainly shows where some of the members’ priorities and (not so hidden) agendas lie. However, I personally can’t see how that leads to Gian’s conclusion:
With the publication of the WCAG Samurai Errata, the web community finally has a choice—and if WCAG 2.0 continues to be unworkable, developers will simply turn to another set of guidelines
…particularly since even the Samurai document (just like the current WCAG 2.0 Working Draft) acknowledges that:
Compliance with WCAG+Samurai cannot be a claim of full accessibility to people with cognitive disabilities.
Maybe it’s just me, but somehow I don’t see how guidelines can realistically be implemented and enforced (say in the context of a large university with multiple authors), unless there are actually testable sets of guidelines and success criteria / checkpoints?
So says Matt May
Hold on. Testability wasn’t pushed through by anyone. It was a guiding principle of WCAG 2, since WCAG 1 was decried by implementers (along with Section 508) as not testable enough.
When Gian proposed removing testability as a requirement for success criteria in April of 2005, she was voted down unanimously, in two consecutive meetings. There were 19 participants at the 29 April meeting, including 5 current and former WCAG editors, 5 reps from accessibility advocacy groups, 2 from assistive technology vendors, 2 reps (sharing one vote) from IBM, and one each from Microsoft and Adobe.
I was there the entire time this unfolded. This was not, by any stretch of the imagination, foisted upon the Web community by the evaluation tool vendors. It was rejected after much discussion (indeed, after many, many discussions) by all types of stakeholder groups, who thought it was a bad idea. When the WCAG WG can agree on something that readily, that should be a pretty good sign that nobody’s arm was getting twisted.
Added June 27, 2007 at 5:20 am
So says Matt May
Not true. I was a WG participant at the time, and no such thing happened. Testability was a guiding principle of WCAG 2 because WCAG 1 was (along with Section 508) decried by implementers for not being testable enough.
In the case Gian cites, she was voted down unanimously – twice. On the second occasion, 19 working group participants voted against removing testability. Of those, 2 were reps from evaluation tool vendors, and 4 were from large companies (2 from IBM, who share one vote, and one each from Microsoft and Oracle). In attendance were 5 current and former WCAG editors (including Michael Cooper, who at the time was representing Watchfire, which bought Bobby), and 5 reps from accessibility advocacy groups.
At the time, you couldn’t get 19 people in the WCAG WG to agree that the sun rises in the east. So when they can agree so consistently on this one point, what is the most logical assumption to draw: that, after rehashing this same issue over and over and over again, they had thoughtfully considered the implications of both sides and determined that a standard that you can’t test isn’t worth the angle brackets it’s written with; or that a vast conspiracy was unhatched by the makers of Bobby specifically to disenfranchise Gian?
What really gets me is that Gian is going to get people to comment because they think non-testability will help them to dilute the conformance requirements to next to nothing, when what she really wants is to increase those requirements dramatically.
Added June 27, 2007 at 6:53 am
So says Joe Clark
At this point, I think anyone who works in accessibility and doesn’t understand that confusable colours are red and green and the replacements seen by colour-deficient people hasn’t been paying attention.
And anyone who continually harps on one minor deficiency of a draft document has an axe to grind.
Added July 3, 2007 at 1:36 pm
So says Patrick H. Lauke
am i “harping on”? no, i mentioned a nitpick i have, concurrently with my comment on the ALA article. and, as i mentioned in my reply to you over at ALA as well, it’s not meant as a criticism of WCAG+Samurai, but only mentioned because Gian holds this document up as a set of clear instructions which are so much better than WCAG 2.
http://www.alistapart.com/comments/testability?page=4#37
Added July 4, 2007 at 10:39 am