For five months in 2020 I worked with Wikimedia Foundation on a question that sounds simple until you have to answer it in public: should the team keep its current browser automation framework, or move to something else? The project was part of Google Summer of Code, and the work was less about picking a winner than about making the decision defensible.
The brief behind the work
Wikimedia maintains a large collection of repositories and extensions. Many of them have browser tests, and those tests have to survive changes in browsers, libraries, and continuous integration environments. The team had a framework in place. The question was whether a different one would make the suite faster, less fragile, or easier to maintain.
We narrowed the field to three options: Cypress, Puppeteer paired with Jest, and WebdriverIO. Each one sits at a different point on the spectrum between a library and a complete test product, which is exactly why the comparison could not be reduced to a feature table.
How we judged a framework
A browser automation tool has to do more than open a page. We looked at the cost of the first setup, the reach across browsers, mobile emulation, handling tabs and frames, average runtime, flakiness, parallel execution, debugging support, documentation, and the health of the plugin ecosystem. Those concerns pull in different directions. A tool that is quick to start may be hard to scale; a tool with every feature may carry a heavy configuration burden.
We also looked at the shape of the existing suite. A replacement that ignores the team's current patterns is not really a replacement. It is a second migration waiting to happen.
What Cypress did well
Cypress arrived with a strong first impression. The runner, assertions, network stubbing, and time travel debugging live in one place. The documentation explains not only which command to use but why a test should be written a certain way. For a developer who is new to browser testing, that combination lowers the entry cost a great deal.
The friction appeared in continuous integration. We ran into failures that did not reproduce in the same way locally, and the security flags around the browser environment made the failures harder to isolate than we wanted. The issue was not that Cypress could not do the work. It was that the cost of making it reliable in our specific pipeline was higher than the schedule allowed us to absorb.
What Puppeteer with Jest offered
Puppeteer gives direct control over Chromium through a small, promise based API. Paired with Jest, it can run tests in parallel, emulate devices, and stay close to the browser without a Selenium layer. The result is fast and pleasant when the suite is small and the team is comfortable building its own harness.
The tradeoff is that Puppeteer is a browser control library, not a complete testing framework. Reporting, video capture, and some of the conveniences a team expects from a test runner have to be assembled or added. That is not a flaw. It is a different product boundary, and it matters when the suite is large.
Why WebdriverIO stayed
WebdriverIO is built as a test framework first. It supports the WebDriver protocol, ships a large plugin ecosystem, and had recently added network primitives that made stubbing and throttling much more practical. For a team that already had tests and a pipeline, that continuity counted for a lot.
In the end, the recommendation was to keep WebdriverIO. The comparison was useful because it gave the team evidence, not because it produced a dramatic switch. We documented the criteria, the experiments, and the reasons for the decision so that the next person would not have to start from a blank page.
What I took away
Tool selection is a product decision wearing technical clothes. The right answer depends on the suite you already have, the people who maintain it, and the failures you can afford to debug. A feature list can start the conversation, but it cannot finish it.
I also learned to write down the reasoning while the work is fresh. A decision without a record becomes a rumor. A decision with a record becomes a starting point for the next engineer.
Score tools against the work you actually have, not against the feature list you wish you had time to use.