How Watchkeeper Was Tested Against a Page Built to Fool It
A tool that says “this page contacted five sites” is only worth something if the page really contacted five sites. Watchkeeper’s main promise is that nothing a page does gets missed, and that the page can’t hide from it or tamper with it. So the tests don’t ask Watchkeeper what it saw and trust the answer. They ask the servers.
The self-test runs the real app, on the system’s own web engine, against two small web servers started on the same machine. One plays the site at 127.0.0.1. The other plays a tracker at localhost, which counts as a different site. Both write down every request they receive, and those logs are the ground truth: whatever reached a server must be in Watchkeeper’s report.
The servers’ logs decide whether a check passes.
An Ordinary Page
The first page behaves like a typical site with a tracker on it. The tracker loads a script, a stylesheet, a pixel and a frame, then calls fetch, XMLHttpRequest, a beacon and two WebSockets. It sets cookies, writes storage from the page and from inside its frame, reads a canvas, WebGL and audio, starts a worker and makes one more call a second and a half later.
On Linux the tracker’s server logged 15 requests. Watchkeeper had 14 of the 14 addresses it could have seen. The fifteenth came from inside the worker, a gap Watchkeeper states rather than hides. The storage written by the tracker’s frame was credited to the tracker, and every fingerprinting read was credited to the tracker’s script rather than to the page.
A Hostile Page
The second page is built to break things. It goes looking for Watchkeeper’s collector and tries to empty it without the secret. It replaces the browser’s own JSON, URL, string and array functions, in case Watchkeeper relies on them. It learns the key a same-site frame uses to send events and forges 4,207 messages with it. It names 2,100 made-up sites, makes 61,000 calls that go nowhere and floods thousands of addresses and cookies. It writes a storage key with a broken character in it and another built to break open a saved report. Then it loads a real tracker.
The collector stayed shut. The real tracker’s script, pixel, fetch, beacon and WebSocket were all in the report. The flood came to 73,827 events Watchkeeper couldn’t count, and the panel said so instead of quietly dropping them. A report saved from that page parsed cleanly, and its HTML had no script, image or link in it.
Back Past a Trap
The third phase goes to a page on the other site that has replaced the browser’s Back function with one of its own. Watchkeeper’s Back button went back anyway, because it uses the engine’s own control, and the page’s version was never called. The page that came back from the engine’s cache was watched as its own site again, with the calls it made after returning in the report.
Reports and History
The last checks save reports from both test pages and compare them with the servers’ logs, and confirm that the page itself can’t save one. They also check the per-site history: it must hold site names only, none of the made-up ones, and only the panel may erase it.
The Numbers
There are 28 checks in all. They pass on Linux with WebKitGTK 2.52.6, where the run takes about half a minute, and the same test runs on Windows and macOS on every change. Each installer runs it again on its own system before a release goes out. A separate reviewer, not involved in writing the code, also attacked each version with probe pages of its own, and everything it found was fixed before release. The recorded results are published with the code.