As I wrote earlier in a blogpost now we have cypress.io test framework integrated into Collabora Online codebase. Since we are testing an online, Javascript-based application that means we are working in an asynchronous environment. So triggering a user event via this framework, does not mean that this event is actually finished at the same time when the related cypress method is called. On the other hand, we try to cover real user scenarios, which are usually a sequence of events, as the user doing things on the UI step-by-step. So the question is, how to write sequential tests in an asynchronous environment. Fortunately, cypress has the answer to that.

First, let's see an example of simulating user events in the cypress test framework. The following code triggers a selection of the text by simulating CTRL+A shortcut:

// Select all text
cy.get('textarea.clipboard')
 .type('{ctrl}a')

With calling this method we trigger a keyboard input, which will be handled by the client code (loleaflet). The client JS code sends a message to the server about the keyboard input. On the server-side, we will execute the shortcut, which results in a text selection. Then the server will send back the new text selection to the client-side, which will display it for the user. However the cy.get().type() call won't wait for that, it just triggers the event on the client-side and moves on.

Why is it a problem? Let's extend this code with another event:

// Select all text
cy.get('textarea.clipboard')
 .type('{ctrl}a')

// Apply bold font
cy.get('#tb_editbar_item_bold')
 .click();

Let's assume that this '#tb_editbar_item_bold' item exists. It's a toolbar button which applies bold font on the selected text. The problem here is that the selection is not finished yet when we already triggered the font change. Triggering these events does not mean that they will be executed in the specified order. Both events are handled in an asynchronous way, so if the first event (e.g. typing) takes more time to execute, then it will be finished later. Which means the application won't apply bold font on anything, because text selection is not there yet.

What is the solution here? What we need is the event-indicator method. It's very simple. After every event, we trigger in the test code, we should use an indicator that the event is actually processed. Let's see how we can fix the previous scenario:

// Select all text
cy.get('textarea.clipboard')
 .type('{ctrl}a')

// Wait for the text selection to appear
cy.get('.leaflet-marker-icon')
 .should('exist');

// Apply bold font
cy.get('#tb_editbar_item_bold')
 .click();

The '.leaflet-marker-icon' DOM element is part of the text selection markup in the application, so we can use that as an indicator. In cypress, every cy.get().should() call has an implicit retry feature. It tries to get the related DOM element, again and again, until it appears in the DOM …

At Collabora, we invest a lot of hard work to make LibreOffice's features available in an online environment. Recently we greatly improved the Collabora Online mobile UI, so it's more smooth to use it from a mobile device. While putting more and more work into the software, trying to support more and more different platforms, we need also to spend time improving the test frameworks we use for automatic testing. These test frameworks make sure that while we enrich the software with new features, the software remains stable during the continuous development process.

One step on this road was the integration of cypress.io test framework into Collabora Online code. cypress.io is an end-to-end test run in the browser, so any online application can be tested with it. It mainly allows us to simulate user interaction with the UI and check the event's results via the website's DOM elements. That allows us to simulate user stories and catch real-life issues in the software, so our quality measurement matches the actual user experience.

When I investigated the different alternatives for browser testing I also checked the Selenium test framework. I didn't spend more than two days on that, but I had an impression that Selenium is kind of "old" software, which tries to support many configurations, many language bindings which makes it hard to use and also makes it stuck in the current state, where it is. While cypress.io is a newer test framework, which seems more focused. It is easier to integrate into our build system and easier to use, which is a big plus because it's not enough to integrate a test framework, but developers need to learn how to use it too. I saw one advantage of Selenium: the better browser support. It supports all the main browsers (Chrome, Mozilla Firefox, Safari, Internet Explorer), while cypress.io mainly supports only Chrome, but it improves on this area. Now it has a beta Mozilla Firefox support. So finally I decided to use cypress.io and I'm happy I did that because it nicely works.

So cypress.io is now integrated into the Collabora Online code and we already have 150 tests mainly for mobile UI. As we submit most of our work to upstream, these tests are also available in the community version of the software. It's integrated into the software's GNU make based build system, so a make check will run these tests automatically. This is also part of the continuous integration system, so we can catch any regression instantly, before it actually hits the code. It's recommended to all developers of the online code to get familiar with the test framework, so it will be easier to understand if a test failure indicates an issue in their proposed patch. There are a set of useful notes in the source code, in the readme file …