General Questions to Consider When Comparing or Evaluating VPAT® Documents
A VPAT® is NOT an Accessibility Audit
It is important to distinguish an accessibility audit from the Voluntary Product Accessibility Template (VPAT® ) used to report the findings of the Accessiblility Conformance Report (ACR).
Accessibilty audits are typically highly granular and cover each state of a software application, page of a website, or feature of a hardware product.
While it is typically necessary to complete an accessibility audit to develop and deliver an accurate VPAT®, not every vendor will have completed an accessibility audit in advance of preparing their VPAT®. Do not expect the VPAT®/ACR to fully address the extent of the acccessibility issues in the product or the potential risk exposure of publically deploying a less than accessible product.
Was Testing or an Accessibility Audit Undertaken?
Given the voluntary character of the report, it is possible no formal audit or little or no accessibilty testing was undertaken prior to completion of the VPAT®/ACR.
Typically, details related to any testing or audit undertaken is found in the Notes sections of the VPAT®.
A document missing notes on the related testing or audit proceedures used to prepare the VPAT® should raise a red flag and be followed up immediately with a request for supplemental information on the tools/methods, platforms, operating systems, browsers, assistive technologies, test plans, and testing results assertained in creating the ACR. An NDA may be required in order for the vendor to release this information.
Is Every VPAT® Accurate?
Highly unlikely.
A VPAT® is a voluntary disclosure on part of the vendor, manufacturer, developer, etc. It is not a certification. In practical terms it may, or may not, be:
- Complete
- Thorough
- Accurate
- Up-to-date
- Written by an independent third-party
- Reflect testing by a independent third-party
- Based on the findings of an accessibilty audit
- Include thorough or even appropriate testing methodologies
- Include testing by users with a variety of disabilities
- Reflective of the use of the product with a variety of assistive technologies
- Reflective of your planned usage or implementation of the product
That said, consider VPAT®/ACR a good faith effort to report on the accessibility capabilities and shortfalls embodied in the product—at least until proven otherwise.
“Trust, but verify,” as they say.
Procurement Process Red Flags Related to VPAT®/ACR Documents
Was an ACR Provided Initially or at All?
Did the vendor have an ACR prepared and was it already included in the bid? Perhaps the ACR is missing, but instead the vendor provided a general, vague accessibility statement. The latter might point to the lack of a substantive accessibility audit or consideration of accessibilty in the development of the product.
Has the Vendor Provided the Correct VPAT®/ACR?
Is the VPAT®/ACR for the Correct Product?
Make certain the product described at the beginning of the report is the one you are considering purchasing. Closely read the name and description. Double check product versions, etc.
Has the vendor provided mulitple, fundamentally identical ACR documents for different products? This might point to reuse of a single document without an actual accessibility audit, which increases the likelihood of hidden accessibility issues.
Similiarly, has a vendor provided similar ACR for products from different manufacturers. This might point to a vendor (reseller) that was unable or unwilling to have the actual manufacturer provide an accurate VPAT/ACR and instead has rubberstamped results from one product for all.
Do the Documents Cover All Products Offered?
Double check that the VPAT®/ACR documents cover all products under consideration.
For example, a software or hardware system may come in a variety of configurations. If the “product” you are being offered is really three separate products that work in unison, it is common for each product to have a separate VPAT®/ACR. Make sure you have all the appropriate VPATs®/ACRs.
Is the Document Current?
Accessibility audits require effort. A vendor may complete an accessibilty audit and develop a VPAT®, but fail to update the audit findings and related reports as revisions are made to the product to time. This can lead to inaccuracies. If the VPAT®/ACR is more that a couple years old, inquire if you have the most up-to-date version of the document and whether the document is truly reflective of the current state of the product.
As noted above, verify the version of the software or hardware listed is the version under consideration for purchase. Technology moves quickly.
Have You Been Provided a VPAT® of the Correct Version?
There are four versions of the current VPAT®, plus older versions still in use (covered in a later section below).
The current versions and their usage are as follows:
- VPAT 2.4 508: Revised Section 508 standards – the U.S. Federal accessibility standard
- VPAT 2.4 EU: EN 301 549 – the European Union’s “Accessibility requirements suitable for public procurement of ICT products and services in Europe”
- VPAT 2.4 WCAG: WCAG 2.1 or ISO/IEC 40500 – W3C/WAI’s recently updated Web Content Accessibility Guidelines
- VPAT 2.4 INT: Incorporates all three of the above standards
Note: The Web Content Accessibilty Guidelines version (VPAT 2.4 WCAG), uses the same tiered criteria (A, AA, AAA) found in the WCAG itself. It is common to report compliance with the first two tiers of criteria: A & AA. However, if you have a mandate to meet some or all of the AAA criteria, such a report will fall short of the information you require to determine the product’s compliance.
Older VPAT® Versions
There are older versions of the VPAT® still distributed by many vendors. These may not fulfill your current requirements.
Section 508 was updated in January 2017 as part of what is termed “the Section 508 refresh.” Prior to that VPAT®/ACR documents had different reporting criteria most noticably lacking reference to the WCAG 2.0 criteria commonly used for websites, web-based applications, mobile applications and other software.
Many product vendors are slow to update their reporting documents. You may find yourself in a situation where competiting vendors report using different guidelines making an apples-to-apples comparison very difficult.
Does the VPAT® Appear Complete?
Remember, vendors differ in their willingness and ability to develop accessible products, perform regular accessibility audits, and prepare VPAT®/ACR. Because of this the quality of reporting varies widely. Furthermore, vendors might not have a VPAT® available at the time of your request and scramble to complete one for your procurement process potentially leading to errors or incomplete reporting.
Are All Sections Present?
Check that the document contains all sections that should exist for the version of the VPAT® provided.
Does the Report Cover the Correct Product Under Consideration?
Make certain the product described at the beginning of the report is the one you are considering. Check versions and configurations of the software that are available. Be wary for an out of date report that covers a previous or incorrect version of the hardware, software or other technology.
Does the Report Cover the Full Product?
Does the ACR cover all of the product? For example a course software vendor may have audited the course software as populated with typical course content, but failed to take into account all the supporting functions related to everyday use of the software by students, such as, registration, login, password reset, account management, etc. Likewise, they have have excluded the authoring system used by faculty or other course content creators. The report should cover all aspects of the product’s use.
Does the VPAT®/ACR use the Correct Conformance Level Terms?
Typically, conformance to each criterion is reported as one of the following:
Supports: The functionality of the product has at least one method that meets the criterion without known defects or meets with equivalent facilitation.
Partially Supports: Some functionality of the product does not meet the criterion.
Does Not Support: The majority of product functionality does not meet the criterion.
Not Applicable: The criterion is not relevant to the product.
Note: The “not evaluated” phrase may only be used for the Level AAA table since this is the only success criteria table that is not required to be completed.
If other conformance criteria are reported, it may point to hidden accessibility issues in the product or below necessary understand of accessibility concerns on the part of the vendor.
Lastly, make certain Not Applicable is not over reported in the document and when given a statement as to why it is Not Applicable is made.
Does the VPAT®/ACR Cover the Correct Accessibility Guidelines?
Know which Laws and Regulations Apply
You will need to know which laws and regulations determine applicable accessibility guidelines for your situation.
Federal, state and local laws may all apply as may specific policies.
Does Your Organization Have a Specific Procurement Policy?
Your institution, business or organization may have adopted additional internal accessibility guidelines as a matter of general procurement policy or specifically for this procurement.
Final Thoughts: Little Details Are a Canary in the Coalmine
Failure to Remove the Initial Sections of the VPAT®
The VPAT® itself states that the initial sections of the document be removed when posting/providing the final ACR:
“When publishing your Accessibility Conformance Report, be sure to remove the entire first 9 pages of this document, including the table of contents, introductory information and instructions.”
Use the presence of these intial pages as the first item in reviewing a VPAT®—it’s a dead canary, similar to the infamous Van Halen “No Brown M&Ms” contract rider, showing me the vendor may not have been overly focused on details (or accuracy) in providing their report or the information therein.
Failure to Report Conformance for ALL Elements of the Product
Typically, any VPAT®/ACR version that includes WCAG criteria will require conformance reporting for not only the product itself, but its accompanying documentation and support features. Documentation may take the form of a website or PDF that will both have reportable accessiblity requirements. Likewise, support may include a website, a built in help feature of the product, or even the phone support proces. Absence of accessibilility conformance reporting for one or more of these elements may point to incompleteness in auditing, unstanding compliance requirements, or a general lack of familiarity with accessibility standards on the part of the vendor.
The ACR Document Itself is not Accessible
The ACR document itself, commonly a PDF, should also be accessible. If you have a copy of Adobe Acrobat, you can use its built in accessibility tool (All Tools >> Prepare for Accessibility >> Accessibility Check) to spot check the document’s comformance. A large number of accessibility issues in the document itself may point to other issues in the ACR or potentially the product.
Have Questions?
Feel free to contact me questions or thoughts on any of the above. Leave a question if you have something to add or have experienced unique issues with VPAT/ACRs as a vendor or procurement officer.
