In the presentation An approach testing larger industrial systems by Johan Blom, he shares his experiences testing complex industrial systems, specifically at Mobile Arts, a telecom company. The team faced challenges testing their protocol-based software in data centers, despite using a static type checker and manual tests. They identified core modules for testing and used a static type checker, but were unsatisfied with their testing approach due to potential severe consequences of system failures. Johan then discusses his use of model-based testing, creating an executable model of a telecom system using state machines and Alloy specification language. He developed a tool for better Erlang implementation and improved upon existing tools. Johan also discusses the Erlang Finite State Machine (FSM) specification for testing, using input and output events, changing variables, and reaching end states. He introduces observers and symbolic execution, allowing handling of multiple execution paths and collecting guards. Johan then discusses testing larger industrial systems using observers and concurrent execution, generating abstract test cases, and mapping them to concrete test cases. He also shares his experience using a model-based testing tool to identify issues in a location server at Mobile Arts, which led to the discovery of over 100 issues.
An approach testing larger industrial systems: A comprehensive overview
The complexity of testing industrial systems
Challenges at Mobile Arts
At Mobile Arts, Johan Blom and his team faced significant hurdles in testing their sophisticated, protocol-based software. Despite having a static type checker and conducting manual tests, they struggled to cover all use cases and their combinations. This gap left potential risks for system failures, which could have severe consequences, such as crashing the entire telecom operator network. Identifying core modules within their location server, they utilized a static type checker, dialyzer, to some success, but were ultimately dissatisfied with their approach due to the untested scenarios and potential system vulnerabilities.
Embracing Model-Based Testing (MBT)
Capturing functional requirements
Johan turned to model-based testing to capture functional requirements and create an executable model of their telecom system. This approach not only simplified the creation of the model but also allowed for the generation of test cases directly from the formal model. Dissatisfied with existing tools, Johan developed his own, tailored to work seamlessly with their Erlang implementation.
State machines and alloy specification language
The model-based testing approach involved using state machines represented in the Alloy specification language. For black-box testing, the team used Erlang FSM (Finite State Machine) as a specific specification language. This approach enabled them to simulate input and output events, change variables and state variables, and reach end states to finish test cases.
Generating and selecting test cases
Automating test case generation
Johan explained that using the Erlang FSM specification, they could generate a large number of test cases. The challenge then became selecting the appropriate set of test cases. To address this, they introduced the observer, a tool that helps select test cases based on requirements, even in complex environments. Using Erlang OPS and observer predicates, smaller state machines could express coverage criteria, ensuring comprehensive testing.
Symbolic execution
Symbolic execution played a crucial role in handling sets of executions with symbolic parameters, allowing the exploration of multiple execution paths and the collection of guards along the way. This approach resulted in test cases with symbolic parameters, enhancing the testing process’s efficiency and effectiveness.
Observers and concurrent execution
Defining observers
Observers are components that check variables and predicates during execution. They can be defined in any language and linked to the testing tool for maximum flexibility. The primary components are the model and the observer, with test suites created by checking each edge in the state machine. By ensuring no new definitions of variables are introduced, observers continue to monitor and validate test cases.
Concurrent execution
Johan demonstrated generating test cases for a small observer and finding five test cases to cover it. He also mentioned the potential of executing multiple observers in parallel, generating a large number of symbolic test cases. These cases could then be instantiated with various parameters, creating a vast number of possible abstract test cases, which could be mapped to concrete test cases in the next step.
Mapping and identifying issues
From abstract to concrete
The model-based testing tool allowed the team to map abstract values to concrete ones, enabling numerous possible executions and even random generation. During a larger case study, they identified over 100 issues in the server, which was previously believed to be flawless. This demonstrated the tool’s effectiveness in uncovering hidden problems.
Open-sourcing the tool
Johan expressed his intention to make the tool open source after refactoring it to separate different parts. He shared his background, including his early experience with Erlang during a student project in 1991, underscoring his long-standing expertise in the field.
Conclusion
Johan Blom’s presentation offers invaluable insights into the complexities and innovative solutions for testing larger industrial systems. By embracing model-based testing, developing tailored tools, and leveraging observers and symbolic execution, Johan and his team have significantly improved their testing processes. This approach not only ensures comprehensive coverage and robustness but also uncovers hidden issues that could have severe consequences in an industrial setting. For those interested in learning more, Johan’s tool will soon be available as an open-source resource, promising to aid developers facing similar challenges in testing large-scale systems.