Functional testing tools like Selenium are essential for verifying that the different components of an application work together as intended. Integrating Selenium into a CI/CD pipeline can demand significant rework and high resource utilization, and running Selenium tests in a Kubernetes environment can be complex and resource-intensive. Running the pipeline multiple times for functional testing can quickly become inefficient and costly.
To address these challenges, we explored Testkube’s innovative Test Workflows for executing Selenium tests. Test Workflows provides a streamlined approach to defining, running, and automating tests in Kubernetes, allowing you to define your tests as code, manage test executions, and generate artifacts without disrupting your existing setup.
In this blog, we will delve into how Testkube’s capabilities simplify the functional testing process with Selenium. We will explore how to set up and automate your Selenium tests within a Kubernetes environment using Testkube, overcoming the common challenges of integration and resource management.
Functional testing is the process of verifying whether the functionality of a product is built right. It involves systematically testing each feature to ensure they function correctly according to specifications. This type of testing validates input, output, and user interactions to confirm that the product behaves as expected.
With functional testing, we can ensure that tests execute without errors, including handling exceptions and HTTP 404 responses. It also involves verifying correct redirections, ensuring usability, and assessing accessibility according to specifications. For web applications, functional testing can be automated using Selenium, which tests for expected outcomes in your web applications.
Selenium is an open source project that comprises many tools and libraries for browser automation. In Selenium, we can write test scripts in different languages like Python, Ruby, Java, Perl, etc. It supports different browsers such as Firefox, Chrome, Edge, Safari, etc. Selenium allows various types of testing such as acceptance, functional, regression, performance, load, etc.
There are the main Selenium components that allow testing:
Selenium IDE: For developing Selenium test cases, Selenium provides a Chrome and Firefox extension. It records user actions in the browser using Selenium commands, making it efficient and educational for learning Selenium script syntax.
Selenium WebDriver: WebDriver provides an abstraction layer over browser engines, exposing APIs that command the browser to run tests, simulating real user interactions. WebDriver is non-intrusive, allowing you to test the same application you deploy live.
Selenium Grid: Once WebDriver tests are ready, we may require testing on multiple browsers, different operating systems, and their combinations. Selenium Grid allows parallel execution of test cases across multiple machines and also enables running tests remotely.
These Selenium components help create automated test scripts, cross-browser/platform testing, data-driven testing, and parallel testing with different language support across multiple browsers or operating systems. Let us understand in the coming section the working of Selenium and where these components fit.
Web browsers, despite appearing similar, operate differently beneath the surface. Selenium abstracts these differences, enabling code to execute seamlessly across browsers like Firefox, Microsoft Edge, and Chrome. This simplifies coding, allowing complex workflows to be written in just a few lines while ensuring consistent functionality across supported browsers.
To build a test suite in Selenium using WebDriver, we need to understand some of the commonly used terms:
Driver: A driver controls the actual browser. They are also known as proxies. These are created and provided by the browser vendors in most browsers.
WebDriver: The task of WebDriver is to command the browser but to do so, it needs the driver. WebDriver talks to the browser through the driver and receives information back via the same route.
Framework: The task of the framework is to run and execute the WebDriver. These could test frameworks like JUnit for Java, RSpec for Ruby, NUnit for .Net, etc.
How does a WebDriver communicate with a browser?
In the execution of the Selenium test, the first step is to understand how the communication is happening between the WebDriver and the browser. Here are the two methods:
- Direct: In the direct communication setup, the WebDriver, Driver, and the browser are on the same system.
- Remote: In the case of remote communication setup, the communication between the WebDriver and the browser happens through a Selenium Server/Grid or RemoteWebDriver. The Selenium Server or Grid runs on a different system while the RemoteWebDriver can run on the same system as the driver and browser.
Why do we need a Framework?
The WebDriver communicates with the browser but is unaware of the testing. It has no information on asserting a test fail or pass, or reporting an issue. This is where the frameworks help. A test framework allows the execution of WebDriver that runs the steps mentioned in the test.
The above image shows the working of a Selenium test with remote communication. Let us understand with the help of an example. Let us suppose, you have to create a Selenium test in Java. To execute this test, you will need a test framework like JUnit that will call the WebDriver which runs the test. Once the WebDriver is invoked, it communicates(remote) with the browser via the driver. Thus a Selenium test is run to verify the functionality of a browser.
Now we understand here that multiple components need to be set up in a Kubernetes native environment to perform tests. This comes with some challenges. One major challenge is scaling Selenium tests to run in parallel. Kubernetes can handle scaling, but configuring Selenium to run multiple browser instances effectively within Kubernetes can be complex. You need to figure out browser-specific dependencies, ensure that Kubernetes nodes have optimum resources, and that the test environment closely mimics the production environment.
Addressing these challenges requires a well-planned approach, leveraging Kubernetes features effectively, and utilizing tools like Testkube to simplify and streamline the deployment and management of Selenium tests.
Testkube offers a comprehensive solution to address the challenges of deploying Selenium tests on Kubernetes. With Testkube, teams can efficiently manage resource consumption by leveraging Kubernetes’ scalable infrastructure, ensuring optimal use of CPU and memory resources. Its simplified setup process eliminates the complexities of configuring Selenium within Kubernetes, facilitating seamless communication between components while managing browser dependencies effortlessly.
As a Kubernetes-native test orchestration framework, Testkube simplifies the creation and execution of complex test workflows across various environments without the need for extensive scripting. Test Workflows provide granular control over test execution parameters, resource usage, and setup/teardown procedures. Also, Test Workflows are stored as Kubernetes Custom Resources (CRs), ensuring compatibility, scalability, and portability.
Let’s explore how to use Selenium with Testkube for functional testing in the coming section.
Pre-requisites
To follow along, you will need to set up the following:
Once the prerequisites are in place, you should have a target Kubernetes cluster ready with a Testkube agent configured.
Use Case: Single Browser
We have written a test in Java that verifies a specific button on the website exists and is clickable. This test has been designed to check button functionality on Chrome, Firefox, and Edge. To build the dependencies and run the test, we are using Maven. You can find the Project Object Model (POM) configuration for Maven (pom.xml) and the Java Test (SeleniumTest.java) in this repository.
We will now create a Test Workflow in Testkube using Maven and integrate Selenium tests into it. In this first use case, we will run the test only for the Chrome browser and store corresponding artifacts to get an understanding of the process.
Creating a Test Workflow
Here are the steps to create a test workflow from the Testkube Dashboard:
- Navigate to the Test Workflows tab on Testkube and click on “Add a new test workflow”.
This will provide you with three options:
Create from scratch – use the wizard to create a Test Workflow.
Start from an example – use existing k6, cypress, and playwright examples.
Import from yaml – import your own Test Workflow.
- Select the “Import from yaml” option to create this workflow and provide the configuration as shown below.
- Click on Create and a Test Workflow will be created. Here you can trigger the Test Workflow and execute the tests.
Before we execute the Test Workflow, let us glance at the Test Workflow and the Java test.
kind: TestWorkflow
apiVersion: testworkflows.testkube.io/v1
metadata: name: selenium-single-browser-test #Name of the workflow namespace: testkube #Namespace where TestWorkflow run
spec: content: git: uri: https://github.com/cerebro1/selenium-testkube.git paths: - selenium-java #git repo and path that contains Selenium Java Test container: workingDir: /data/repo/selenium-java #Mount point for the repo image: maven:3.9.6-eclipse-temurin-22-alpine #Maven image for container services: chrome: image: selenium/standalone-chrome:4.21.0-20240517 readinessProbe: httpGet: path: /wd/hub/status port: 4444 periodSeconds: 1 steps: - name: Run Chrome Test run: env: - name: SELENIUM_HOST value: '{{ services.chrome.0.ip }}:4444' # get hostip from service shell: | mvn test #execute the test mvn surefire-report:report #generate reports artifacts: paths: paths: - target/site/**/* - target/surefire-reports/**/*
In this Test Workflow, we have used the Selenium Standalone Chrome image that allows WebDriver tests to be run remotely. The WebDriver will use the chromedriver to communicate with the Google Chrome browser. We have updated the path of the directory in the Workflow that contains the functionality test case in Java for Selenium. In the code, we have added comments to help you understand each step.
Here is the test that we are going to run. In this test, we sent a GET request to a website, searched for the button using the text on the button, and made a click. This test will check if the click was successful and return the outcome based on that.
Maven refers to the pom.xml file and installs the dependencies such as selenium-java and junit-jupiter. We are using JUnit as the test framework and enabled the maven-surefire-report-plugin for gathering the artifacts.
Now that we have our Selenium test, POM, and the Test Workflow ready, let us go ahead and execute it.
Executing the Test Workflow
Here are the steps to execute the workflow and view the artifacts:
- Click on “Run this Test Workflow now” to trigger the first run. As the execution starts, you will see the workflow running as shown below.
- Click on the running workflow to view the log output.
The above image shows that the Chrome Test is running. The next step here is uploading the artifacts once the shell commands are executed successfully.
- Once the test execution is complete, click on Artifacts.
The above image shows the list of artifacts generated and stored at the path target/chrome-artifacts.
- In the site directory, click on
surefire-report.html. Testkube loads the Surefire report in the browser as shown below.
Thus, with the help of Testkube, we were able to automate the execution of a Selenium Java test on a Kubernetes environment using Maven.
In this use case, we ran the test only for one browser—Chrome. In the real world, for testing an application development, you would want to perform the functional test across multiple browsers to ensure your application works for the majority of users. Here you have two options: you can either run a test on each browser serially or invoke parallel testing. Let us see with the help of the following use case how parallelization is better than serial test execution and can help improve the overall testing process.
Use Case: Sequential Test Execution with Multiple Browser Engines
We utilized the previous use case to create a serial Test Workflow for the test execution on multiple browsers. You can create and run this workflow as described for the basic use-case above—the execution will look as follows:
(clicking on the individual items will reveal their logs as applicable)
The workflow is configured to collect artifacts from each execution into a folder named after the browser:
As you can see, we now have three browser instances (Chrome, Firefox, Edge) and three corresponding test executions, one after the other. Although this works as expected, there are some points we noticed:
Tests run one after another will make the entire testing process time-consuming, particularly for large test suites.
If a test fails, it can halt the entire testing process until the issue is resolved, causing delays.
Serial testing is not an optimal use of available computing resources.
In the coming section, we will see with the help of a use case how we can run the same Selenium test across different browsers using parallelization in Testkube.
Use Case: Parallel Execution with Multiple Browser Engines
With parallelization, not only could we improve the use of infrastructure resources for test execution, but also execute more tests in the same amount of time. This would lead to earlier detection of issues and an increase in the overall efficiency of the testing process.
Let’s modify our Test Workflow to make use of two powerful Testkube features: matrix parameters and parallel execution. To achieve this, in the Test Workflow we have defined the IP and name of each browser service in parallel.matrix.browser. The execution of shell commands and storing of artifacts will happen in parallel for all browsers using the parallel execution object (the full workflow is here):
... steps: - name: Run Cross-browser tests parallel: matrix: # define matrix of browsers to test against browser: - ip: '{{ services.chrome.0.ip }}' name: chrome - ip: '{{ services.firefox.0.ip }}' name: firefox - ip: '{{ services.edge.0.ip }}' name: edge transfer: # make sure each parallel node has the test - from: /data/repo/selenium-java fetch: # retrieve generated artifacts from node - from: /data/repo/selenium-java/target to: /data/artifacts/{{ matrix.browser.name }} container: workingDir: /data/repo/selenium-java image: maven:3.9.6-eclipse-temurin-22-alpine steps: - run: env: - name: SELENIUM_BROWSER value: '{{ matrix.browser.name }}' - name: SELENIUM_HOST value: '{{ matrix.browser.ip }}:4444' shell: | mvn test mvn surefire-report:report - condition: always workingDir: /data/artifacts artifacts: paths: - '**/*'
Create and Execute the Workflow as we have done previously:
In the above image, we can see that three workers are running at the same time. Testkube not only has improved the process of test execution, but also made optimal use of available computing resources. Since these workers are independent of each other, the failure of any one of them will not impact the execution of the others, providing test isolation.
Performance Comparison: Parallel vs. Sequential Test Execution
On the completion of the above tests, we compared the execution times of five serial and parallel test executions. (Execution times are all in seconds)
While the execution of the entire serial test workflow took roughly 73 seconds on average, the parallel execution workflow completed in 48 seconds on average, an improvement of roughly 34%. Drilling down into the actual times spent on running tests within each workflow (excluding setup/teardown/etc.), the difference was 57 seconds on average in serial flows vs. 30 seconds on average for parallel executions—an improvement of almost 50%.
Conclusion
Selenium is a widely used suite of tools for browser automation. Leveraging its power in Testkube without the complexities of resource management, environment configuration, and setting up dependencies streamlines the testing process for Kubernetes applications.
In this blog post, we have seen how functional testing with Selenium could be done using Testkube. We started with setting up a Testkube workflow for a single browser and explored parallelization to optimize multiple browser testing. We were able to run a Selenium test in Testkube effortlessly for single and multiple browsers and explored the benefits of parallel execution in the case of multiple browsers. Testkube abstracts the complexities and allows optimized test execution in a containerized environment.
Using Testkube, you can ensure compatibility across Selenium, browser drivers, and Kubernetes versions, reducing maintenance overhead. Overall, it offers a strategic and efficient approach to deploying Selenium tests on Kubernetes, enabling teams to focus on delivering high-quality software with confidence.
We invite you to try Test Workflows. Visit the Testkube website to get started. If you find yourself struggling with anything, feel free to drop a note in our active Slack community, and someone will help you out.