Accessibility scanner
Scanner Improvement Report
What we changed in the accessibility scanner between August and October 2026, how the scanner now works, how we validated the results, and what improved.
1. Summary
The goal of this work was that the scanner must not report a problem that is not there. A report is only useful if a developer can check any finding in it and find it true.
In August that was not the case. When we re-tested the August version's findings on real websites, only about 4 in 10 were true, and it missed most of the real problems it should have found.
We changed three things:
- How a page is tested. Four specialised testers now check every page at the same time in a real browser, each measuring the page the way a person testing for that group of users would (section 3).
- How we know whether a finding is right. Every finding a client would see was re-tested independently in a real browser, with a measurement as evidence, on 34 real websites. Automated test suites check the scanner after every change (section 4).
- What happens when a finding is wrong. We traced each false alarm back to the reason it happened and fixed that reason, instead of only hiding the result (section 6).
Section 9 sets out what is still not perfect. The figures come from small samples, and making the scanner more careful also means it shows fewer of the real problems it can detect.
2. Where we started
At the end of August we evaluated the scanner as it was then. We scanned 11 real websites and re-tested every finding. Three problems stood out:
- Most findings were false alarms. About 38–43% of what it reported was true.
- It missed most real problems. On a public test suite of 89 known accessibility failures, it found 36. It found only 5 of the 15 keyboard failures, even though keyboard testing is what sets this scanner apart.
- Sometimes it scanned the wrong page. On 2 of the 11 websites it clicked something, such as a link inside a cookie banner, and then reported on whatever page it landed on, without any warning.
How a scan worked in August
Everything ran in one browser, one step after another. A keyboard robot pressed keys through the page, then axe (the industry-standard free accessibility engine, in an old version) ran its rules, then a basic colour-contrast check, and optionally an AI assistant looked over the page. There was no dedicated testing for screen reader users, people with low vision, or readability: those parts of the report were mostly whatever axe and the AI happened to notice. Nothing was available until the whole scan had finished, and there was no systematic way to tell whether a finding was right.
3. How a scan works now
This is what happens, in order, from the moment someone asks the scanner to check a website until the report is ready. The scanner uses a real Chrome browser throughout, so it sees the page exactly as a visitor would, after all scripts have run.
- Choose the pages
For a single page, the scanner uses the address it was given. For a whole website, it reads the site's page index and follows its links (without a browser, so this is fast), ignores duplicates and pages that do not work, and offers a list of pages to choose from. For pages behind a login, it can sign in first and look for pages only signed-in users can reach.
- Check that the page can be tested
Before testing anything, the scanner checks that it has really reached the page. If the website shows a bot-protection page instead (such as a "checking your browser" screen), it waits up to 15 seconds for it to clear by itself. It never tries to get around one. If the page stays blocked, the scan stops and the report says why, instead of reporting problems on the wrong page.
- Prepare the page as a visitor would
The scanner answers cookie banners and closes pop-ups, choosing the buttons a visitor would use ("Accept", "Godta alle", "Ja") and never following a link away from the page. If login details were provided, it signs in. Login details are never shown to the AI, written to logs or stored. It then scrolls through the page so that content which loads on scroll is present, and waits for animations to finish.
- Four testers check the page at the same time
The keyboard tester works in its own browser because it changes the page as it presses keys. The other three share a second browser with an exact copy of the same session (cookies and login), so they see the same page. Each tester is described in detail below.
- Run axe
axe's 105 rules run in the same browser once animations have settled. Where axe and one of our own checks report the same problem on the same element, the finding is kept once: the one with the better evidence.
- Combine and sort the findings
Repeats of the same problem (for example the same unlabelled icon on 20 product cards) are grouped into one finding with a count. Each finding is linked to the WCAG requirement it breaks and to a topic such as "Contrast" or "Names and labels". It then gets its reliability group (shown, "Needs review" or hidden, see section 6) and its severity: critical, serious, moderate or minor, using axe's four levels.
- Write the report
Each finding carries its evidence: a screenshot of the element, the element's location in the page, and for keyboard findings the exact keys that were pressed. An AI model writes the plain-language title, explanation and fix advice from that evidence. It does not decide whether a problem exists or how severe it is; those come from the measurements. A final step checks the wording against a fixed set of rules, so every report reads consistently.
- Show results as they come in
Each tester saves its results as soon as it finishes, so the live view fills in while the scan runs. During the scan it shows "possible issues so far", because the final step removes findings that are not reliable enough. For a multi-page scan, the scanner signs in once and copies the session to each page, then adds findings that span pages, such as navigation that changes from page to page.
What each tester does, and how
Keyboard
Uses the page only with the keyboard, the way people who cannot use a mouse do.
- Presses Tab through every stop, forwards and backwards, and records where focus lands each time.
- At each stop, compares a picture of the element with and without focus to check the focus outline is visible, and measures its contrast against what is around it.
- Opens every button and link that reveals something (menus, dialogs, accordions) with Enter or Space, then checks that focus moves into it, that Escape closes it, and that focus returns to where it was.
- Finds controls that only work with a mouse by comparing what looks clickable with what the keyboard can reach.
- Runs at desktop and mobile screen sizes at the same time.
Screen reader
Checks what blind and partially sighted users would hear.
- Reads the browser's accessibility information, the same information screen readers use, at every keyboard stop: each element's name, role and state.
- Checks that opening a menu, a dialog or a message is announced, and that dialogs have a name.
- Checks that a button's visible text is part of its spoken name, so voice control works.
- Checks images, link text, tables, headings, lists, page title and page language across the whole page.
Low vision
Checks what people with reduced vision would struggle with.
- Calculates contrast from the colours actually painted on screen, not just the colours in the code, including shadows, transparency and backgrounds behind text.
- Measures the contrast of field borders, icons and focus outlines (at least 3:1) and of text (at least 4.5:1).
- Switches the page to Windows high-contrast mode, 200% text size, a 320-pixel-wide screen and wider text spacing, and checks that nothing disappears or is cut off.
- Looks for colour used as the only way to tell things apart.
Reading
Checks how easy the content is to read and follow.
- Scores how hard the text is to read, with a formula suited to the language (one for English, another for Norwegian and Swedish), calibrated on 25 real pages.
- Flags long unbroken blocks of text and very long lines.
- Finds content that moves or updates by itself without a way to pause it.
4. How we validated the results
We used two kinds of validation. Each one answers a question the other cannot.
Re-testing findings on real websites
This answers the question that matters most: when the scanner reports a problem on a real website, is it true?
- Scan the way a client would
Each website was scanned with the same default settings a client uses, and only the findings a client would actually see were counted (normal findings and "Needs review").
- Re-test every finding independently
Each finding was re-tested in a separate Chrome browser, using measurement scripts written separately from the scanner, so a mistake in the scanner could not repeat itself in the check. The method depends on the type of finding:
- Screen reader findings: ask the browser for the name, role and state it really gives the element, as a screen reader would receive them.
- Contrast findings: read the colours from the browser and from the screenshot's pixels, and calculate the contrast ratio.
- Keyboard findings: press the same real keys from the same starting point and record where focus goes.
- Layout findings: reproduce the narrow screen, 200% text or high-contrast mode and measure what is lost.
- Decide true or false with a strict rule
A finding counts as true only if a real user would meet the problem as the finding describes it. A finding that points at a real problem but describes it wrongly counts as false. Findings that are best-practice advice rather than strict failures, or measurements very close to the limit, are marked as such.
- Record the evidence
Every verdict is stored with the measurement behind it (the colours and ratio, the announced name, where focus landed) and, if false, the reason the scanner got it wrong. Each verdict was reviewed by our team. The same problem appearing again in a later scan keeps its verdict, so results can be compared over time.
- Calculate how often findings are correct
Correct = true findings ÷ all findings checked. We report it separately for normal findings and "Needs review" findings, and for websites we had worked on and websites the scanner had never seen.
Which websites, and why
| Set | Websites | Why |
|---|---|---|
| Study websites | 22 websites in Norway, Sweden, Denmark, the UK and elsewhere: public services, health, transport, banking, retail, media, technology | Measured repeatedly to see the effect of each round of fixes |
| New websites | 12 websites the scanner had never been tested on, in two rounds of 6 | The honest test: fixes made for the study websites must also work elsewhere |
| Demonstration scans | 3 websites scanned on the system that will be demonstrated | To confirm that the results shown in the demonstration are correct |
Automated test suites
These answer a different question: did a change to the scanner make anything worse? They run in minutes, use no AI, and give the same answer every time, so they are run before every release.
- The accuracy benchmark. The UK Government Digital Service publishes 142 small web pages, each with one known accessibility failure. The benchmark counts how many the scanner finds. It also scans a demonstration website published by the W3C (the body that writes the accessibility guidelines) in a broken and a repaired version. Anything reported on the repaired version counts as a false alarm.
- The real-website check. 10 public websites (government, news, retail, classifieds, an encyclopedia). It fails if any scan ends up on a different page than requested, and flags cookie banners left open and unexplained drops in findings.
- More than 1,500 automated tests of individual parts of the scanner, including a test for each false alarm we have fixed.
5. How we made it find more real problems
September to mid-October
- Four testers instead of one sequence. Screen reader, low vision and reading each got their own tester with checks built for those users (section 3). Before, these areas relied on axe and the AI.
- Measurements instead of guesses. Most new checks do not use AI. They measure the page in the browser (colours on screen, announced names, where focus goes), which makes them repeatable and checkable.
- A deeper keyboard tester. It now opens and closes menus, dialogs and pop-ups and follows focus through them, finds mouse-only controls, checks Space as well as Enter, and handles content that opens a moment after a key press.
- An up-to-date axe. We upgraded from version 3.1 to 4.13, the version used by Google's Lighthouse: 105 rules instead of 73. It also removed many old false alarms, 54 on one Norwegian news website alone.
- Reliable on real websites. Cookie banners in Norwegian and Swedish and inside embedded frames are answered without leaving the page, and the mobile pass really runs at mobile size.
- Faster. Running the testers side by side halved the time for a two-page scan, from 377 to 179 seconds.
Result: 75 of the 89 known failures found (36 in August), including all 15 keyboard failures (5 in August). axe alone finds 32. But finding more also meant more chances to be wrong, which is what the next phase dealt with.
6. How we removed the false alarms
14 October to 29 October
Step 1: measure every finding
On 14 October we scanned the 22 study websites and re-tested all 194 findings, as described in section 4. 42% were true. The scanner now found about twice as many real problems as in August, but also about twice as many false ones. The study showed exactly how often each type of check was right.
Step 2: only show what has proven reliable
We sorted every type of check into three groups based on that measurement:
| Group | Rule | How it appears in a report | Correct on 14 Oct |
|---|---|---|---|
| Shown | Right at least 75% of the time | As a normal finding | 93% |
| Needs review | Right 50–74% of the time, or too few examples to judge | Shown, clearly labelled "Needs review", never presented as a failure | 41% |
| Hidden | Right less than half the time | Not shown; kept in the scan's records, so nothing disappears silently | 18% |
A check moves to a higher group only after a new measurement shows it belongs there. axe's rules are always shown, because every axe finding we measured was correct.
Step 3: find the cause of each false alarm and fix it
Hiding a check also hides its true findings, so for each false alarm we went further:
- Reproduce it. Open the page in a browser and repeat what the scanner did, until the wrong result appears.
- Find the reason. Compare what the scanner measured with what is really on the page, and pin down where they differ.
- Fix the reason, not the website. Change how the scanner measures, so the fix works on every website with the same pattern, not just the one where it was found.
- Lock it in. Add a small test page with the same pattern and an automated test, so the mistake cannot return.
- Check nothing else broke. Run the full test suites and re-scan the website to confirm the false alarm is gone and the true findings are still there.
Some examples:
| What the scanner reported | What was really happening | What we changed |
|---|---|---|
| "The focus outline has poor contrast" (a public broadcaster) | It compared the outline with the button's own red colour instead of the page around it. | Contrast is now measured against what is actually next to the outline. |
| "This icon has poor contrast" (same site) | It took the icon's colour from the code (black) instead of what is shown on screen (white). Some icons were faded because they were disabled. | It uses the colour actually drawn, and skips faded and disabled icons. |
| "This search box has no visible border" (a grocery retailer) | The border was drawn with a shadow effect, which the scanner ignored. | Shadows that draw a border now count as a border. |
| "There is no skip link" (a public health portal) | The skip link was inside a self-contained page component the scanner could not look into. | It now looks inside those components. |
| "Text on a focused button is unreadable" (a bank) | It used the section's background colour, not the colour painted behind the button's text. | It finds the real background behind the text. |
| "The visible label does not match the spoken name" (a news website) | It treated a photo caption inside a link as the link's label. | Captions no longer count as a link's visible label. |
| "The menu opens without being announced as a dialog" (a travel company) | It was a full-screen menu that correctly announces itself as opened. | Menus that announce their open state are no longer expected to be dialogs. |
| "Focus is lost after adding to cart" (an electronics retailer) | The cart pop-up opened slowly, so the scanner connected it to the wrong button. The website actually returned focus correctly. | Focus returning to the button that was really pressed now counts as correct. |
| "The Play button does not announce what it opened" (a music streaming service) | The scanner only looked at the first part of a very large page; the pop-up was at the end. | It now also checks the end of the page, where pop-ups usually sit, and reports that pop-up's real problems instead. |
| "The dialog is not announced" (a public services portal) | The dialog is announced; it just has no name. The scanner measured this correctly, but the report described a different problem. | A dialog with no name now gets its own accurate description and fix advice. |
| "This form field has no label" (a postal service) | The field was named by its placeholder text, with a heading and button beside it. This rule was wrong both times it appeared. | The rule is now hidden. |
The electronics retailer, music streaming and dialog fixes were completed and tested this week and are included in the release for the 3 November demonstration.
Step 4: make the report itself clearer
- One severity scale: critical, serious, moderate and minor, the same four levels axe uses. Previously, different parts of the report used different scales.
- "Needs review" is clearly separated from confirmed findings and never shown as a failure.
- The live view says "possible issues so far", because the final count can be lower once unreliable findings are removed.
- Each finding has one topic with a filter, and comes with its evidence and suggested fixes.
Step 5: confirm on websites the scanner had never seen
After each round of fixes we scanned new websites and re-tested every finding the same way:
- 27 October, first round (6 websites in Norway, Sweden and the UK: postal, health, tax, broadcasting, grocery, news): 74% correct. We found 5 causes and fixed them.
- 27 October, second round (6 more: a municipality, a bank, a TV channel, an employment service, a newspaper, a software foundation): 88% correct. We found 2 causes and fixed them.
- 29 October, demonstration scans (3 websites, on the system that will be demonstrated): 10 of 10 findings correct.
Each round turned up fewer new mistakes than the one before. That trend is the main sign the approach is working.
7. Results
How often a finding a client sees is correct
Every finding shown, including "Needs review". Each bar is a separate measurement; the websites and the number of findings differ between them.
Before and after
| Measure | August | Now |
|---|---|---|
| Findings a client sees that are correct, on websites never seen before | ≈ 40% | 88% |
| Same, on the websites used while improving the scanner | ≈ 40% | 94% |
| Known failures found (public test suite, WCAG levels A and AA) | 36 / 89 | 75 / 89 |
| — keyboard | 5 / 15 | 15 / 15 |
| — screen reader | 26 / 57 | 48 / 57 |
| — low vision | 3 / 11 | 9 / 11 |
| — reading | 2 / 6 | 3 / 6 |
| False alarms on the W3C's repaired demonstration website | 17 | 9 |
| Scans that checked a different page than requested | 2 of 11 websites | 0, checked on every change |
| Time for a two-page scan | 377 s | 179 s |
| Keyboard testing, desktop and mobile, on one page | 71 s | 34 s |
For comparison, axe on its own finds 32 of the 89 known failures in the same test suite.
8. How we keep it this way
Every change to the scanner must pass three checks before it is released:
- More than 1,500 automated tests, including one for each false alarm we have fixed, so none of them can come back unnoticed.
- The accuracy benchmark: no known failure may be lost, and no new false alarms may appear on the W3C's repaired website.
- The real-website check: all 10 websites must be scanned on the right page, with cookie banners closed and no unexplained drop in findings.
Each release has a version number, shown in the application, and every scan records which version produced it. All validation data is kept, including the evidence behind each verdict, so any figure in this report can be traced back to the findings it comes from.
9. What the figures do not show
- The samples are small. Each round of new websites produced 16 to 20 findings, so a single mistake moves the percentage by about 5 points. The trend across rounds matters more than any single figure.
- Being careful has a cost. Hidden checks also hide true findings. On one website, a software foundation's home page, the scanner showed no findings at all, because all four of its findings came from hidden checks.
- Hidden checks were not re-tested. Validation covered what a client sees. Hidden checks will be measured one at a time before any of them is shown again.
- Some correct findings are recommendations, not strict failures. A few findings counted as correct are best-practice advice or measurements close to the limit. Each finding's evidence says which.
- One known false alarm is still open. A chat window on one website opens 4–6 seconds after it is requested, and the scanner judges it before it appears. It is labelled "Needs review".
- No automated tool covers all of WCAG. Some questions, such as whether the description of an image is meaningful, still need a person. The scanner finds problems and shows the evidence; it does not replace an expert accessibility audit.
10. Terms used in this report
- False alarm
- A finding that says something is wrong when, on re-testing, it is not.
- Correct
- The share of findings confirmed true on re-testing: true findings divided by all findings checked.
- WCAG
- The Web Content Accessibility Guidelines, the international standard for accessible websites. Levels A and AA are what most regulations, including the EU's, require.
- axe
- A widely used free engine of automated accessibility rules. The scanner runs it alongside its own checks.
- Focus
- The element that keyboard input goes to, usually shown by an outline. Keyboard users need to see it and need it to move sensibly.
- Screen reader
- Software that reads a page aloud for blind and partially sighted people, relying on the names and roles the page gives each element.
- Accessibility information
- What the browser tells assistive technology about each element: its name, its role (button, link, dialog) and its state (open, selected).
- Needs review
- A finding from a check that is not yet reliable enough to be stated as a failure. Shown so it can be checked, never counted as a confirmed problem.