We prepared the answers to the submitted questions for the Call for Expert Audit and Baseline Metrix (Milestone 1a).
Many questions regarding the scope have reached as so that we decided to provide more details which we will update on this website until Monday, June 8th, 2026 latest.
Below you find the questions that have reached us.
Current status: 7 answers missing.
Is the audit scope limited to Joomla core, the Atum admin template, and the Cassiopeia front end template, or are any core extensions, modules, plugins, sample data, or optional components also included? Deliverable 1
Will be answered as soon as possible, latest until Monday June 8th, 2026
Ref. No.: MS1a-1
Will OSM provide a definitive list of core admin views to be included in the axe core baseline scans, or should the contractor define this list as part of the methodology? Deliverable 2
Will be answered as soon as possible, latest until Monday June 8th, 2026
Ref. No.: MS1a-2
Which Joomla version or code branch should be treated as the audit baseline at kick off? Deliverable 1
The audit baseline is Joomla 6.1, the current stable release at the time of publication. OSM will confirm the exact version at kick-off if a newer patch release has shipped. The contractor must document the version explicitly in all deliverables.
Ref. No.: MS1a-3
Will OSM provide a prepared test environment with sample content, users, permissions, media, and representative configuration, or should the contractor create this environment? Deliverable 1
OSM will provide a standard Joomla 6.1 installation as defined in Q1, with representative content and user accounts covering the required roles. Contractors may use their own local environment for testing, provided it matches this reference configuration. A configuration checklist will be agreed at kick-off, and OSM will require access to the contractor's environment or equivalent documentation for verification during review.
Ref. No.: MS1a-4
Which user roles should be included in the admin interface testing, for example Super User, Administrator, Editor, Author, or others? Deliverable 1
At minimum: for the backend a Super User and an administrator, for the frontend: a registered user and an author/publisher. Contractors may propose additional roles (e.g. Editor, Author) if their methodology identifies them as relevant to the scope. The final role matrix will be confirmed at kick-off.
Ref. No.: MS1a-5
For the assistive technology task completion benchmarks, will OSM provide the key user journeys, or should the contractor define them? If the contractor should define them, is there an expected maximum number of journeys? Deliverable 2
Will be answered as soon as possible, latest until Monday June 8th, 2026
Ref. No.: MS1a-6
How should ATAG be interpreted for this milestone? Should the audit assess only the authoring interface, or also Joomla’s support for authors creating accessible output? Deliverable 1
The ATAG audit covers Part A only (accessibility of the authoring tool itself) as the primary conformance target. Part B (support for producing accessible content) is generally out of scope for this milestone. However, Joomla includes a built-in accessibility checker ("Accessibility Check"-Button) in the editor, which partially addresses ATAG Part B — most directly B.3.1 and B.3.2. Contractors are invited to propose a lightweight Part B gap analysis as an optional deliverable, identifying which Part B criteria are already met, partially met, or not yet addressed by the current implementation. This would serve as an informational supplement to the Part A audit and as input for future milestones. Any such proposal must include a clear scope description and cost breakdown.
Ref. No.: MS1a-7
Should task completion benchmarks include timing, error rates, blocker counts, qualitative observations, or another preferred measurement model? Deliverable 2
The contractor is expected to propose a measurement model. The benchmarks established in this milestone will be re-run at different milestones and at the project close to measure progress. The contractor is expected to propose a measurement model that can be executed identically at multiple points. The model will be confirmed at kick-off.
Ref. No.: MS1a-8
What format is preferred for findings and baseline metrics? For example spreadsheet, HTML report, PDF, Markdown, GitHub issues, or a combination. Deliverables 1, 2
No single format is mandated. The report must be suitable for public publication and structured so findings can be imported into the Joomla GitHub issue tracker for Milestone 1b. Contractors are free to use the format that best fits their normal audit practice, provided deliverables are handed over in open, non-proprietary formats. The exact format will be agreed at kick-off.
Ref. No.: MS1a-9
The brief asks for 2 examples of comparable published accessibility audits. Would redacted or anonymised examples be acceptable where previous complex audits were delivered under NDA? General / Application
Yes. Redacted or anonymised examples are acceptable.
Ref. No.: MS1a-10
How many review cycles should the contractor allow for before final acceptance? Contract
Contractors should allow for two structured review cycles per deliverable. OSM and the Project Manager will provide consolidated feedback after each submission. Contractors should factor both cycles into their timeline.
Ref. No.: MS1a-11
Since payment is listed as full payment upon acceptance, will there be a defined acceptance period after submission of the final deliverables? Contract
Yes. OSM will confirm acceptance or provide consolidated feedback within a defined acceptance period, which will be formalised in the service agreement. Upon acceptance, payment to the contractor will follow within 30 calendar days of that invoice.
Ref. No.: MS1a-12
The brief mentions access to a Joomla community maintainer. Will there be an expected response time or allocated number of support hours for this? General
A response time of under 72 hours applies. This support covers technical and contextual questions about Joomla's architecture.
Ref. No.: MS1a-13
WCAG 2.x as such does not foresee graduated assessments as audit outcomes: it only allows binary statements (PASS / FAIL) under specific conditions ². Conformance statements must, for example, refer to a WCAG conformance level and apply only to complete pages and processes. Introducing any form of categorisation (e.g. critical, major, minor) is not supported by WCAG (nor by ATAG) and is inherently subjective — it would be an editorial choice of the auditors rather than a property of the standard. What purpose is the severity categorisation meant to serve, and who is its intended audience? Are there predefined criteria for the assignment of severity levels that we are expected to apply, or shall we propose and document our own? Expert WCAG 2.2 AA Audit
The severity categorisation (critical / major / minor) is an editorial classification for remediation prioritisation and not a conformance claim under WCAG. Its audience is the OSM project team and contractors working on subsequent milestones. All findings are also mapped to WCAG 2.2 success criteria, which provides the conformance dimension independently. There are no predefined severity criteria; contractors are expected to propose and document their own model. OSM will confirm it at kick-off.
Ref. No.: MS1a-15
We assume the audit is primarily intended as the basis for subsequent remediation (rather than for demonstrating conformance from a standing start, which is unlikely). Given Joomla’s modular architecture, we also assume that remediation will be component-driven. A report structured by modules or components (Editor, Forms, Dialogs, Tables, Media manager, Installer, Authentication, etc., as listed in the RFP scope) would arguably be more useful for the downstream work than a strictly page-/view-based presentation, even though that would deviate from WCAG’s formal page/process scoping referenced above. Which structure is preferred for the report — applied to both the WCAG 2.2 AA and the ATAG findings: component-driven (likely more useful for remediation) or page-driven (more closely aligned with formal WCAG conformance reporting)? Expert WCAG 2.2 AA Audit
Will be answered as soon as possible, latest until Monday June 8th, 2026
Ref. No.: MS1a-16
axe-core ³ (and any tool derived from it, as well as automated accessibility tools in general) is widely held in the expert community (example ⁴) to detect only a fraction of digital barriers. Even Deque, the makers of axecore, share this assessment ⁵. Tools like axe-core are therefore a useful starting point, but neither a reliable nor a sufficient basis for an overall picture of an application’s accessibility. The majority of barriers can only be found and judged through manual testing with human expertise — which is inherently subjective, only partially quantifiable, not uniform, and only repeatable at significant effort. axe-core output is also prone to false positives and requires qualified interpretation. For which audience are the axe-core baseline figures intended, and to what extent does that audience have the expertise needed to interpret them correctly and to evaluate the additional qualitative measures that the numbers cannot capture? Baseline Metrics
The axe-core baseline figures are intended primarily for reporting — to provide a quantifiable, reproducible before/after metric across the project. OSM and the STF are aware that automated tools detect only a subset of accessibility barriers and that output requires qualified interpretation. The baseline figures are one quantitative input alongside an expert manual audit; they are not intended as a complete picture of Joomla's accessibility.
Ref. No.: MS1a-18
Regarding the assistive-technology user-journey tests, we need clarification on a number of points before we can scope and price them: Which concrete user journeys are to be tested, and who defines them? Is the contractor expected only to identify the relevant journeys and define the test methodology, or also to perform the initial round of tests? Baseline Metrics
The contractor is expected to propose the user journeys, define the methodology, and perform the initial round of tests as part of MS1a. OSM will confirm the journeys at kick-off.
Ref. No.: MS1a-19
Who performs the tests: end-user participants with assistive-technology experience, or digital-accessibility experts? Baseline Metrics
The contractor is expected to propose their testing approach, including who performs the tests. If end-user participants with AT experience are proposed instead of or in addition to expert testers, any cost difference must be clearly stated in the proposal.
Ref. No.: MS1a-20
Are the tests to be repeated? At which points in time, and by whom (especially for repeat runs)? The RFP states that the metrics must be re-runnable identically at project close — is that re-run also part of Milestone 1a, or is it a later milestone’s obligation? Baseline Metrics
The initial run is part of MS1a. The re-run at project close is MS10's obligation and may be carried out by a different contractor, provided they have accessibility expertise. The methodology must therefore be documented in sufficient detail for re-execution by someone with no prior involvement in MS1a.
Ref. No.: MS1a-21
How is “task completion” defined, and what does “benchmark” mean here? Which metrics are in scope — whether a participant can complete a task at all, how quickly they do so, which workarounds are admissible? Repeat runs with the same participants are difficult to evaluate because subsequent passes are necessarily skewed by familiarity. Overall, speed, success rate, satisfaction and other outcomes are highly sensitive to the individual participants, their skills, the assistive technology used and the concrete test environment. We consider this form of testing to be poorly suited to repeated execution and would like to flag the considerable effort involved.Why are only screen readers used for the user-journey tests, and no other assistive technologies (e.g. screen magnifiers, voice-control software, switch input)? Baseline Metrics
The contractor defines task completion in their methodology; see other answers above. Screen reader combinations are specified as the minimum baseline. Contractors may propose extending the AT matrix (e.g. voice control, screen magnification) with rationale and cost breakdown.
Ref. No.: MS1a-22
On what basis were the browser/screen-reader combinations chosen? Why, for example, is Firefox + NVDA — a combination in widespread use ⁶ — not part of the matrix? Baseline Metrics
Will be answered as soon as possible, latest until Monday June 8th, 2026
Ref. No.: MS1a-23
What is the motivation behind the requirement to define which assistive technologies and browser combinations are “in scope”? Deliverable 3: Accessibility Test Protocols and Acceptance Criteria
The RFP scope is WCAG 2.2 AA and ATAG. Contractors who consider EN 301 549 coverage necessary for a complete picture should flag this in their proposal with scope and cost implications. Any extension beyond WCAG 2.2 AA / ATAG must be agreed before contract signature.
Ref. No.: MS1a-25
Why is the goal not full WCAG conformance independent of specific technology pairings?
The AT/browser matrix defines the testing environment for reproducible, comparable baseline and regression metrics, it does not limit the conformance target. Full WCAG 2.2 AA conformance remains the goal. The matrix ensures that test runs at project open and project close use identical conditions, enabling the before/after comparison required for STF reporting.
Ref. No.: MS1a-26
Which understanding of “acceptance criteria” — beyond the success criteria of WCAG (and ATAG) themselves — underlies the requirement? Acceptance criteria for whom and for what? Are they meant to be a deliberately reduced set of criteria used during remediation and further development of Joomla? Where is the cut to be drawn if WCAG as a whole is not the yardstick? (Example: WCAG defines minimum text-contrast ratios; should sufficient text contrast be named as an explicit acceptance criterion for every Joomla component, or is it assumed throughout — and if so, who guarantees it?) Deliverable 3: Accessibility Test Protocols and Acceptance Criteria
The acceptance criteria defined in Deliverable 3 are operational criteria that make WCAG requirements actionable in a development workflow across all subsequent milestones. They are not a reduced alternative to WCAG. Their audience is developers, reviewers, and the automated CI pipeline. WCAG requirements that apply universally (e.g. contrast ratios) should be expressed as standing criteria rather than repeated per component. The contractor is expected to propose format, granularity, and structure.
Ref. No.: MS1a-27
Who is the target audience for the acceptance criteria — designers, developers, manual testers, or automated test pipelines? Deliverable 3: Accessibility Test Protocols and Acceptance Criteria
Developers, manual testers, and the automated CI pipeline. The criteria must be actionable in all three contexts.
Ref. No.: MS1a-28
In which format are the acceptance criteria to be expressed (free text, Gherkin, structured machine-readable form, etc.) and at what level of granularity? Deliverable 3: Accessibility Test Protocols and Acceptance Criteria
The contractor is expected to propose the format. Formats considered appropriate include structured Gherkin, machine-readable YAML, or well-organised Markdown. The key requirement is that criteria are unambiguous and usable by both manual testers and automated tooling and reproducible.
Ref. No.: MS1a-29
What is the relationship between the acceptance criteria and the explicitly listed AT/browser matrix? In particular, are criteria to be expressed per AT/browser combination, or independently of it? Deliverable 3: Accessibility Test Protocols and Acceptance Criteria
Criteria should be expressed independently of specific AT/browser combinations where possible. AT-specific variants should be documented separately where behaviour genuinely diverges across combinations.
Ref. No.: MS1a-30
Should the test protocols and acceptance criteria reflect EN 301 549 (and/or other regulatory standards relevant to Joomla’s user base), or are they to be strictly scoped to WCAG 2.2 AA and ATAG as named in Deliverable 1? Deliverable 3: Accessibility Test Protocols and Acceptance Criteria
Strictly scoped to WCAG 2.2 AA and ATAG as named in Deliverable 1. See Ref No. MS1a-25 for the rationale.
Ref. No.: MS1a-31
The RFP also asks for a “regression test approach”. Is this expected to be a written methodology only, or are concrete test artefacts (e.g. an axe-core configuration, scripted user journeys, CI integration patterns) part of the scope of Milestone 1a? Deliverable 3: Accessibility Test Protocols and Acceptance Criteria
Concrete, reusable artefacts are required, not a written methodology only. At minimum: an axe-core configuration, documented scripted user journey protocols, and a reference CI integration pattern. These artefacts must be handed over to OSM and be usable by the MS8a contractor.
Ref. No.: MS1a-32
What is the substantive content of this deliverable that is not already part of Deliverables 1 and 2? Deliverable 4: Public Publication of Audit Report and Baseline Metrics
Deliverable 4 is the act of publication. Its purpose is to make the audit findings and baseline metrics publicly available before the remediation plan (MS1b) is drafted, ensuring community transparency and enabling feedback before implementation priorities are set.
Ref. No.: MS1a-34
What is meant by “Public Publication”? Are the test results to be handed over to OSM only, or actually published at a defined location? If the latter, where is the publication expected to take place, and who is responsible for the publication channel (OSM, the contractor, or jointly)? Deliverable 4: Public Publication of Audit Report and Baseline Metrics
Publication will take place on Joomla's GitHub and/or community channels. OSM is responsible for the publication channel. The contractor is responsible for delivering publication-ready materials. The exact channel will be confirmed at kick-off.
Ref. No.: MS1a-35
Are there constraints on the publicly released versions of the audit report and baseline metrics — licence, formats, anonymisation of test participants, redactions, embargo period? Deliverable 4: Public Publication of Audit Report and Baseline Metrics
The report will be published immediately upon acceptance. The audit report and all deliverables will be published under a Creative Commons CC BY 4.0 licence. Any personal data must be anonymized and handled in accordance with GDPR.
Ref. No.: MS1a-36
What does “establish the programme documentation framework” mean concretely? Is it a deliverable that the subsequent milestones are intended to reuse? Since the scope of those later milestones is not yet known to us, we cannot judge what is required for the framework to support them. Deliverable 5: Documentation and Milestone Report
OSM will provide a documentation framework at kick-off covering document structure, naming, versioning, and handover formats. All MS1a deliverables must conform to this framework.
Ref. No.: MS1a-38
Can we freely choose the format of our audit report — as it corresponds to our normal audit practice — or must we comply with specific requirements? If the latter, which ones, and will they be available in time for proposal preparation rather than only at kick-off? Deliverable 5: Documentation and Milestone Report
Contractors may use the format that best fits their normal audit practice. Accepted file formats include PDF, Microsoft Office, LibreOffice, Google Files and markdown formats. The documentation framework provided at kick-off will specify any additional structure requirements.
Ref. No.: MS1a-39
General: Is there an overview over the remaining 18 milestones to help with understanding the requirements for this first one and to ensure that we can take them into consideration for a holistic beginning of the project?
The milestone descriptions including deliverables and durations are available in the STF application document and will be shared with the contractor.
Ref. No.: MS1a-50
General: If we were selected as a supplier for this or another RFP connected to the programme, would that affect our eligibility to also be selected as one of the two project managers in the “Questions - RFP - Call for (2) Project Managers for the Joomla! Accessibility Project”? Is there any potential conflict of interest or restriction we should be aware of?
To avoid conflicts of interest or appearance of such, a Project Manager or their associated companies can not apply for any project Milestones.
Ref. No.: MS1a-51
Deliverable 1: What is the ATAG conformance target? For WCAG, a conformance target of AA is set. Should we assume ATAG 2.0 AA for the proposal?
Yes. The conformance target is ATAG 2.0 Level AA, Part A only. As noted in Q7, Joomla's built-in accessibility checker partially implements aspects of ATAG Part B (specifically B.3.1 and B.3.2). A full Part B conformance audit is not required. However, contractors may optionally propose a Part B gap analysis - an informational review of which Part B criteria are currently met, partially met, or absent - as an additional deliverable. If proposed, it must be scoped and priced separately and will not affect Part A acceptance.
Ref. No.: MS1a-52
Deliverable 2: What exactly is meant by “establish … measurements”? Are we supposed to develop how to measure the metrics or should the first round of axe core and user testing also be carried out?
Both. The contractor is expected to define the measurement methodology and execute the first round of axe-core scans and user journey tests as part of MS1a. The results of that first round constitute the baseline. See also Q19 and Q8.
Ref. No.: MS1a-53
Deliverable 3: How does this relate to the standardized WCAG 2.2/ATAG 2.0 protocols and acceptance criteria? Do you want to generate Joomla-specific test procedures?
Both. The foundation is WCAG 2.2 AA and ATAG 2.0 AA as the conformance standard. Deliverable 3 asks the contractor to make those standards operational for Joomla's specific architecture and development workflow thereby defining which components, views, and user journeys are in scope, and producing test artefacts (axe-core configuration, scripted journeys, CI integration pattern) that can be used directly by subsequent milestone contractors and the CI pipeline. See also Q25, Q26, and Q32.
Ref. No.: MS1a-54
Deliverable 4: Are there any requirements for the public publication of the report?
The report must be publication-ready upon handover to OSM. Requirements: open non-proprietary file formats, CC BY 4.0 licence, GDPR-compliant handling of any participant data. OSM is responsible for the publication channel (Joomla GitHub and/or community channels/websites). The exact channel will be confirmed at kick-off. See also Q35 and Q36.
Ref. No.: MS1a-55
Deliverable 5: Is there public documentation for the “programme documentation framework” and “milestone completion report”?
No existing public documentation exists for either. OSM will provide both at kick-off. The documentation framework will cover document structure, naming, versioning, and handover formats. The milestone completion report template will be shared at that point. Accepted file formats are listed in Q39. (See also Q38.)
Ref. No.: MS1a-56
On Proposal material, specifically: examples of comparable published accessibility audits: Since many of our client engagements are covered by NDAs, we are somewhat limited in what kinds of example reports and case materials we can share. Could you clarify how incoming application documents are handled from Joomla’s side in terms of confidentiality and information management?
Proposals submitted to OSM are treated as confidential and reviewed only by the selection committee (OSM board members, the Project Manager, Commitee members). They are not published or shared outside that group. As confirmed in Q10, redacted or anonymised audit examples are acceptable where the original engagement was covered by an NDA.
Ref. No.: MS1a-57
Please clarify "published" accessibility audits For the proposal you require "At least two examples of comparable published accessibility audits". Usually our audit reports are not published by our customers. Results are published as an accessibility statement. Would it be sufficient to provide two "internal" audit reports?
"Published" is not limited to publicly available reports. Internal audit reports are acceptable as comparable examples, provided they demonstrate the methodology, findings structure, and reporting format the contractor typically applies. Redacted or anonymised versions are explicitly acceptable where the original engagement was covered by an NDA or client confidentiality agreement. See also Q10.
Ref. No.: MS1a-67
Please clarify "reference from a contracted entity" For the proposal you require "At least one reference from a contracted entity where an audit was done". What's your expectation here? Do your required a contact from the client?
A reference from a contracted entity means a contact person at a previous client organisation who can confirm that an accessibility audit was delivered under contract. A name, role, and email address is sufficient. The reference will only be contacted if required during the selection process. No formal reference letter is required.
Ref. No.: MS1a-68
Scope of backend testing required Can you give us an idea of the scope of testing required for the backend (Joomla admin)? How many user stories, screens and components do you expect to be tested? This is a main driver for the efforts required.
Will be answered as soon as possible, latest until Monday June 8th, 2026
Ref. No.: MS1a-69
Mobile testing We assume that mobile testing is required for frontend and backend?
Will be answered as soon as possible, latest until Monday June 8th, 2026
Ref. No.: MS1a-70
Funding It's unclear to us, if there is funding for this project already, of if you are still in progress to get the funding. Can you please clarify the situation?
STF already publicly declared their willing to fund us: https://ted.europa.eu/en/notice/-/detail/347501-2026. OSM is currently in the process of finalising the contract with the Sovereign Tech Fund. The funding amount and project scope are already defined.
Ref. No.: MS1a-71
Expected budget Can you indicate an estimated or expected budget?
The total programme budget is allocated across all 19 milestones, project management, and legal reserves. OSM is not in a position to disclose budget allocations per milestone. Contractors are expected to define a fair and reasonable price based on their own assessment of the scope and effort required. Proposals must include a clear breakdown of costs and a rationale for the fee.
Ref. No.: MS1a-72
Changes to the this document
June 4th, 2026: Added 15 answers