Measurement Methods

Christophe Strobbe, K.U.Leuven, Belgium

What Basis for Measurements?

Example from web accessibility:

One can distinguish between several types of documents, each with a slightly different purpose:

guideline

general advice on what a developer or author should do or avoid; e.g. Don't rely on color alone

success criterion

specific requirement formulated as a testable statement; e.g. Captions are provided for live synchronized media

technique

concrete technical procedure that a developer or author can follow when implementing a guideline or to fulfill a success criterion

test (for evaluation)

concrete technical procedure that an evaluator can follow to check whether specific content or functionality meets the requirements of a guideline or a success criterion

law

(can be generic or technically precise)

Examples: BITV in Duitsland is gebaseerd op een vertaling van WCAG 1.0 (BITV in HTML, BITV in PDF); Section 508 in the USA was based on a working draft of WCAG 1.0 (see Jim Thatcher: Side by Side WCAG vs. 508); in some countries the law requires that websites are accessible, without defining criteria for accessibility; other countries rely on general anti-discrimination legislation.

In this context, these different types of documents are focused on different types of users:

• guidelines, best practices and techniques focus on the authors or developers (those who provide the information or application);
• success criteria (in the case of WCAG 2.0) focus on the user of the information or application;
• tests focus on evaluators of web sites or applications.

Evaluation: Steps / Structure

Scope and sampling

• manual sampling of pages
• automatic sampling of pages

= independent of UWEM

Not in UWEM:

• scenarios for end-users

Evaluation

Tests:

• unambiguous, based on existing criteria (WCAG 1.0)
• pass/fail
• fail (one or more) = non-conformance

Test execution:

• expert evaluation (all tests)
• automatic evaluation (only fully automatable tests)

Test section = "pluggable"

Not in UWEM:

• end-user evaluation

User satisfaction?

• accessibility = baseline
  • perceivable
  • operable
  • understandable
  • robust
• WCAG = one step removed from user experience

Reporting

• conformance claim (relevant outside WCAG/UWEM?)
• text-based report
• machine-readable report (relevant outside WCAG/UWEM?)
• scorecard report