Automated continuous decision making in testing of robust and industrial-grade network equipments

Photo: https://unsplash.com/

Westermo offers solutions for industry networks in harsh environments, and a typical product is a robust switch or router. A typical customer is one that builds trains, factories, or water treatment plants where many controllers and subsystems need to communicate reliably in a network. Another type of customer is one that desires to upgrade or automate an existing power substation or the like.

WeOS (the Westermo Operating System) is the core software in many of the developed products. The development process of WeOS involves many actors, subprocesses and steps for quality assurance. The process has elements of agile kanban, continuous integration, continuous deployment and many, but not all steps in this process are automated. WeOS code is under revision control, and most software development is made in code branches that are isolated from each other. Each evening the nightly testing and continuous deployment processes are triggered. The devices under test in the test systems are upgraded with the latest WeOS builds. The results from nightly testing are stored in a test results database. The next morning,
developers that made the code change can analyze test results in a web portal.

By integrating smart solutions (e.g., based on AI/ML) as part of the build environment and engineering process of WeOS the following challenges can be solved in the use-case and improvements can be achieved:

i) improved risk mitigation: given an improved understanding of the development process, we can expect the risk analyses to have a lower tolerance for residual risk. This could be measured by identifying the amount of identified risks, their risk levels, and the level of residual risk. Residual risk levels are expected to decrease
by adopting a smart solution;

2) improved test coverage: with a smart system to guide or automate the test resource allocation we can expect test coverage to improve. This could mean that feature code branches have a lower initial test coverage, but it could be expected to increase when approaching merge. This could be measured by analyzing different aspects of test coverage over time, over test systems and over code branches. Untested or overlooked combinations of software features and hardware platforms in release branches are expected to decrease;

3) reduced amount of field bugs and other waste: given an improved process we can expect fewer bugs to propagate. This could be measured by analyzing historical data, and comparing with future projects. Similarly, other types of waste (over-testing, commits related to fixes that could have been avoided, etc.) could be identified, defined and analyzed; 4) increased pace: given increased confidence in the development process and increased awareness of the level of quality, we can expect the release process to be faster and require less manual work. This could be measured by analyzing various aspects of timing in the process -- duration between commits, duration of feedback loops, duration between final code change and release, etc.