Theses

The members of the NTCForum - WG4 also supervise academic theses. If you are a student of University of Oulu and interested in particular aspect of or in general software testing, reviews or inspections, do not hesitate to contact one of the teachers in the group.

Doctoral Theses

Improving the software inspection process with patterns

Harjumaa, Lasse
Doctoral thesis, Department of Information Processing Science, University of Oulu, 2005.
Acta Universitatis Ouluensis, Scientiae Rerum Naturalium, A 447, ISBN 951-42-7893-3, 2005.
Abstract
The quality of a software product depends largely on the quality of the process that is used to develop it. In small software companies, the development process may be informal or even ad hoc, which causes uncertainty and variation in the product quality. However, quality issues are as important in small companies as in their larger counterparts. To sustain their dynamics and competitiveness, small organizations need to concentrate on the most effective quality assurance methods.
Software inspection is a proven method for improving product quality and it provides a very cost-effective way for small companies to improve their development processes. This study introduces a framework for adjusting the inspection process for the organizations specific needs and evaluating its capabilities. The main focus of this work, however, is on refining and improving the inspection process. The improvement is guided by concrete instructions that are documented as process patterns. The pattern approach has already been used successfully in several other areas of software engineering. Patterns aim at capturing the best practices of software development and transferring this knowledge between people or organizations.
The framework for inspection process capability originates from the literature relating to different types of peer review methods and experiments with flexible and tool-supported inspections in small companies. Furthermore, generic process improvement models are studied to find a feasible structure for the framework. As a result of the analysis the i3 capability model is introduced. The feasibility of the model has been investigated in real-life software organizations carrying out inspections.
After the capability evaluation, the inspection process can be upgraded with the aid of improvement patterns, which provide structured and easy-to-follow guidelines for implementing improvements. An initial list of patterns, describing solutions to the most common problems confronted in the establishment of inspections, is extracted from related inspection research and an industrial experiment.
The contributions of this study are, first, the new view of the inspection process, based on the fundamental activities that are performed during an inspection instead of a series of stages, as it is usually presented. An activity-based process description enables tailoring of the process for organization-specific needs and its targeted improvement. Secondly, the study introduces a practical, lightweight method for implementing the improvement. Patterns are especially suitable in companies where resources are limited and full-scale improvement programmes cannot be initiated. Furthermore, the generic process improvement models do not provide detailed information on how improvements should be carried out, and the pattern approach represents a promising method for that. Third, the inspection process currently does not have a very significant role in generic software process improvement models; this study helps in outlining the importance of inspections. A similar approach could be applied to other software subprocesses to enable their evaluation and improvement.

Licentiate Theses

..........................

Master Theses

On software testing focused Master theses (supervised by NTCForum WG4 and i3GO group members) completed in Department of Information Processing Science (TOL) since summer 2004

Correlation between test coverage and the structure and quality of the code being tested

Kanstren Teemu (26.5.2004)
Comprehensive programmer unit testing is sometimes considered to be linked to the structure and quality of the code being tested. In this thesis a simple framework for measuring this relationship is presented. This framework is based on the existing QMOOD model. The framework is applied to three open source projects to test its applicability and see the results for these projects. The programmer unit testing is studied in the form of test coverage. As a result some of the measured metrics seem to get higher, some lower and for some no connection is observed as the test coverage rises




Software defect management in product projects with long supplier chains

Kurikka Arto (17.6.2004)
The author studies in thesis software defect management and defect prevention in the context of the product project. The impact of long supplier chains on the defect management is observed, in particular. The results of thesis showed that long supplier chains were related to long integration chains, to the diverse use of different defect management procedures and defect databases, and to the difficulty of finding the person responsible for the defect. It was concluded based on these findings that improvement could be achieved by harmonizing the defect management procedures within both the product project and the suppliers. By using a uniform defect database, significant improvements in defect management can be attained.

The automation of integration testing in the case of network elements through an automation platform

Kirves Jyri (3.11.2004)
The author compares in thesis two conventional test system-specific automation solutions to the new approach based on an automation platform. The comparison studies the benefits of an automation platform and the major problems in automation of integration testing in the case of network elements. The comparison is based on four characteristics, maintainability, number of functionalities, controllability and usability. None of the evaluated solutions was best in all characteristics, but each has some strong point. The automation platform is a good choice, when the number of functionalities is increasing. The platform also standardizes the structure of automation implementation and thus improves maintainability.




Test Driven Development in Mobile Java Environment

Hanhineva Antti (3.11.2004)
This study uses conceptual-theoretical approach to present an approach for TDD in mobile Java environment. The approach is composed of tools, their integration and changes in program code to enable TDD in Java based mobile Java environment. Theory testing approach is used to evaluate the proposed approach. The evaluation is done in a project where a mobile application is developed for real market in mobile Java environment. The results of the evaluation show that the TDD practice is difficult to adopt and the project did not use TDD as much as other reported studies implicate. Therefore the applicability of the proposed approach cannot be unambiguously stated. However the qualitative results gathered from the project implicate that the development team did not consider the proposed approach difficult to use. The problems were caused because of teams' inexperience of TDD practice and its applicability for the application area in the evaluation project.




A Reading Technique for Inspecting Correct Usage of Design Patterns

Kotaniemi Juha (24.11.2004)
Software inspection is a powerful method for detecting defects in software documentation. Inspections can be used to evaluate all types of artefacts including code, requirements and designs. If the design is not straightforward, which is often the case when utilizing object oriented design patterns, inspecting the design documentation may become difficult. Inspectors should be provided adequate guidance and instruments to efficiently read through the material and defects in it. This paper introduces a reading technique that helps the inspector to recognize typical problems in design pattern use. The reading technique was experimented by 70 computer science students, and it helped the inspectors to reveal defects in usage of design patterns and would have been difficult to find without assistance.

A Support Method for Automating the Functional Testing of the User Interface of an Extensive Embedded System

Turtinen Markus (2.12.2004)
The method presented in thesis is based on the existing practical and theoretical principles of model-based testing combined with the recent research on model-based development. The externally observable behaviour of a mobile phone software is modeled as hierarchical state machine components divided on feature level. The feature dependencies of mobile phone software are presented in models by using global state variables. The essential solutions concerning the testing architecture are given, which enables a possibility of creating sufficient controllability in testing the user interface software of mobile phones.




A defect classification method for defect management in the component factory

Räisänen Ilpo (20.12.2004)
The author studies different methods for defect classification and chooses the most appropriate one for the target company. The chosen method is the ODC (Orthogonal Defect Classification) method. The thesis introduces its adoption to the company. The adoption proceeds through a pilot project and the results will be used when planning further adoption and implementation.

Motivators and demotivators of peer reviews in software companies in the Oulu region

Huttunen Anna (20.12.2004)
Peer reviews are an efficient quality assurance method in software development. Several reviewing methods exist to match the needs of different organizations and situations. Still, peer reviews are not practiced as commonly as one would suppose. This study aims at finding out what types of reviewing methods are in use in software companies, surveying the most important benefits of peer reviews and investigating reasons for not utilizing reviews. The study is carried out in companies locating in the Oulu region, but the results can be generalized to all small software companies. The results show that companies that use reviews have adjusted the process for their own needs. The main motivator for arranging reviews is the decreased amount of defects in products while the other aspects of reviews, such as process improvement or knowledge sharing are not considered as important. The main demotivator for reviews is lack of time and people resources

Ensuring test coverage in the automated functional testing environment

Petäjäjärvi Pasi (4.5.2005)
Frequently launched new software products cause problems for adequate testing. Their development cycles are very short and due to this the manual testing with good coverage is impossible. The automation of software testing is then required to ensure test coverage. The author studies in thesis analysis methods for test coverage, tools for measuring test coverage and tailoring of the chosen tool to support coverage measurement in the target company. The tailoring process produced an automated functional testing environment, which was further evaluated in the thesis. The results of evaluation were promising and in the chosen test cases the testing cycle has been decreased from days to some minutes.




Component Framework for a TTCN-3 based Test System

Karinsalo Mikko (4.5.2005)
This work focuses on TTCN-3 test systems. The current test systems provide no support for easy adaptation of test system components, and they typically couple all required adapters and codecs in a single entity. To overcome the limitations of current TTCN-3 test systems, the system structure was further refined and a component framework was developed for test system components. Theoretical basis for the framework was taken from the systematic software reuse theory. This includes the software reuse factor in product-line architectures, component-based and object-oriented reuse techniques, and framework technology. Constructive research method was used in this work, and a component framework was produced in order to increase flexibility and improve reusability in a TTCN-3 test system. Usage of the framework reduces implementation work by enabling reusability within the test system.

Test automation and organisation in test-driven software development

Pehkonen Aki (11.10.2005)
Test-driven development (TDD) completely turns traditional development around. Instead of writing functional code first and then your testing code as an afterthought, if you write it at all, you instead write your test code before your functional code. Furthermore, you do so in very small steps - one test and a small bit of corresponding functional code at a time. Successful test automation in TDD method requires continuous integration and supporting tools, such as integration server, test coverage analyser and unit test tool. Tools may be commercial or open source ones. The author reports in thesis an experiment built on top of .NET platform. The environment was a collection of CruiseControl.NET, NCover, NDepend, NUnit and NAnt tools. The experiment was successful and indicates that chosen tools provided an appropriate toolbox for test automation.

Improving system testing with metrics

Isohätälä Milla (2.11.2005)
The author studied improvement of system testing in a target company. The appropriate metrics for the evaluation of the system testing process were chosen by means of the Goal Question Metric (GQM) paradigm. The eight metrics chosen for measuring the system testing were related to test efficiency, number of defects, time/effort and test coverage. The study indicates that the use of metrics improves visibility of the project progress and milestones. An important finding of the study was the inefficiency of the test cases in system testing. At the moment of study the major number of metrics were collected manually. Automation of the measurement will be the next development step in the target company.

The impacts of different test tool interfaces on module testing

Sipola Outi (7.12.2005)
The author studied the impacts of different test tool interfaces on test coverage and speed of the test execution in module testing. The research method was the controlled experiment of two testing tools chosen for study. The system under testing was Symbian OS Telephony Subsystem (TSY). The results of study were somehow surprising, the testing tool focused on module testing was faster in test execution, whereas the tool focused on integration testing was better in test coverage.

Towards automating testing in B3G communicational applications

Puhakka Tapani (22.2.2006)
B3G (Beyond 3G) services consist of services like context awareness and group communications. The field of services offered by B3G is huge, but a common characteristic among these services is that they all use a network to interact with servers and with other terminal. The term B3G itself is not well-establish yet, it refers to future mobile networks with multiple different network connections, multiple protocols, faster network and new technologies that are not commonly deployed in the current 3G networks of today. The results of the study prove that context aware and group communicational applications require new methods for automated testing as the testing process will require interaction between multiple terminals. The test results depend not only on the device that starts the sequence, but also on the other terminals that it is interacting with. To apply automation to this kind of environment the test sequence must be expanded to contain multiple interacting devices simultaneously. This may cause new problems, e.g. services that rely heavily on the context information and heterogeneous networks require new ways of testing.

The potential of test automation

Seppälä Juha (31.5.2006)
The author studied the potential of test automation in GUI-applications in the target company. Typically the GUI-level application testing tools include functionality to record script and playback of that script to repeat the same sequence. Recording of the script causes high level language to be generated that describes the sequence to be performed, like buttons to press and intervals between them. This recording ability usually is only a small portion of a test tool's capabilities. To really take advantage of an automated testing tool, test engineers needs to modify the script by hand to create a reusable and maintainable test procedure. The analysis of the applications indicate that automation of smoke tests would be the first step to start automation in the target company. The study includes also an evaluation of the commercial testing tools based on the IEEE model. Based on the evaluation the author suggests the IBM Rational Functional Tester tool as the most appropriate tool for test automation in the target company.

Software testing pressures for change in software companies in the Oulu region

Ylihannu Riitta (31.5.2006)
The author studied the status of software testing and testing pressures for change in software companies. The study was carried out in 11 companies locating in the Oulu region. The research was qualitative research and based on interviews of 14 representatives for the companies. This study aims at finding out what types of testing are in use in software companies and how they plan to conduct testing in near future. The companies involved in study were grouped into three groups (testing process varied from immature to mature), the companies in the first group were just started the standardization of the testing, the companies in the second group owned ISO 9001 certificate and the companies in the third group continually improved their testing process and the automation of the testing was conducted in larger scale compared to the group two. The study indicates that software testing will focus in future more on test automation, agility (TDD approach), model based testing (MBT) and process improvement. These results are alike with the earlier study conducted in Finland two years ago.




Test process improvement with the organization specific defect profile

Öhman Kaisa-Maria (6.2.2008)
The author analyses defect data in the target organization and constructs the organization specific defect profile. The defect data was collected from 11 projects and the size of projects varied between 500 - 14864 person hours. The tentative defect profile is promising but not yet accurate enough, because defect classification was not used equally in different projects. When standardizing the practices in defect collection, the accuracy of the profile will increase. The profile will be utilized in further projects when estimating testing resources and coverage of testing in early phases of software development. The tentative defect profile was experimented in an ongoing project and the results were promising. The profile brought out that the project has not used enough resources for testing. Due to this the project fell behind schedule.




Test generation by using variable finite state machine for modeling application's behavior

Toivola Jussi (20.2.2008)
The thesis presents a method using a variable finite state machine for modeling application's behavior for automatic testing purposes. To model application using the presented method, one needs to find out the situations to test and express the situations using variables and actions. The author introduces a system, which creates a finite state machine of the model configuration by linking the actions together based on the input and output values. The system was found to be effective especially in situations when an action is to be tested in multiple different situations. Writing each situation, by using conventional manual scripting, is a tedious and error prone task. On the other hand, the system still needs better tools to make it more usable. Creating or modifying the model can be hard when the size of the model grows since there is much data to analyze in order to understand the changes done to the model. This phase is also crucial for the testing system to be efficient, since the main reason to develop the testing method is to reduce the amount of required work to perform testing.




Inspection Process Analysis: Case Base Station SW Development

Salmela Sirpa (2.4.2008)
The thesis focuses on the research question: "How can the software inspection process be improved with existing inspection data?" The author studies the question in the BTS (Base Station) SW case. The inspection data was collected over a period of two years, and it contains altogether over one thousand inspections. The inspection data is measured with eight inspection metrics, with few variations of some of the metrics, which are used in the case organization. The metrics results are analysed against company level metrics criteria to find out the problem areas in inspection for the future development of the inspection process.
The inspection metrics analysis points out that the inspection process of the case organization is in quite good shape. Improvement is needed for getting the data more accurate. Defect finding efficiency is slightly under the target, and average inspection effort shows that not enough time is used for checking, and a bit too much time for the inspection meeting. The improvement suggestions consist mainly of adding guidelines and instructions for how to plan inspections, how to check a work product, how to lead meetings, and for reading techniques. A reminder of regular trainings and of the community for moderators is also provided in this thesis.




Deriving test cases from misuse cases

Åström Mikko (2.4.2008)
The thesis presents a technique, how to derive test cases from misuse cases. The work is based on research conducted earlier with use cases. The author adapts the techniques from use cases and found out that the typical path testing and scenario testing techniques were not feasible with misuse cases, because the refinement of misuse cases requires a lot of security expertise, which the software designers typically don't have. It is more relevant to focus on requirements level and and generate test cases for acceptance testing. The author tested the technique in an experiment in the target company and the results were promising. The technique has been further refined and results will be reported in forthcoming research papers.




Towards measurable test purpose in robustness testing of protocol implementations

Lämsä Jarkko (23.4.2008)
Testing can be defined as the process of executing a program with the intent of finding errors. Negative testing focuses on finding undesired behaviour, side effects in software. It aims to show a program does something it is not supposed to do. Since side effects represent the single biggest source of vulnerabilities in contemporary software, negative testing can help to improve software security in a proactive manner. In PROTOS (Security Testing of Protocol Implementations) project, a functional robustness testing method for assessing communication protocol implementation security was developed. It has since been applied to numerous protocols and products and found a feasible mean to discover information security and robustness related faults. Defined coverage criteria, however, are missing. The purpose of this thesis is to present the different aspects affecting the PROTOS input coverage and focus on a specific sub-problem in it, namely how to measure the PROTOS test purpose. Test purpose is generally defined as the aim of a set of test cases. Issues related to the PROTOS input coverage are analysed, including implementation vulnerabilities, communication protocols, negative testing and grammar-based modeling. The primitive elements of typical protocol message formats are identified. The PROTOS fault model - how to malform these elements and formats - is also summarised. Through conceptual analysis, the input coverage is decomposed and a quantitative test purpose measure proposed. The result is considered feasible from the viewpoints of the test engineers and test suite users.




The role of continuous integration in the software code quality improvement

Mylläri Mika (23.4.2008)
The thesis focuses on continuous integration and how to improve software code quality with it. The author studies, what quality metrics are relevant indicators of code quality and how to make quality visible for developers and project managers. Continuous integration enables real-time and automatic measurement and control of code quality. The software size, complexity and coverage metrics were the most important ones and these metrics constitute the basis for improvement activities. The reports and history data are valuable tools for finding the software components and functions with poor quality and for further improvement decisions of the software code.




Offshore outsourcing of NBG's test automation

Mustonen Timo (14.5.2008)
The aim for this case study is to point out that offshore outsourcing of test automation can be challenging even for companies which have been doing it before and already have the needed offshore connections. These challenges were studied with three different test automation outsourcing cases that were carried out by Nokia Oyj. All cases were related to NBG (Nokia Browsing Gateway) development. The study also focuses on possibilities and constraints of both areas, test automation and offshore outsourcing.
After the outsourcing decision, there are still a lot of things for customer companies to take care of. The customer can affect the outcome significantly by preparing and planning own activities, schedules and responsibilities carefully, because subcontractors usually carry out their duties properly and do what is agreed on. Even though most of the work would be done offshore, related onshore support activities still have a very important role in the machinery.




Coordination of Experience Knowledge in Interface Inspection Process

Raulamo Päivi(14.5.2008)
This research addressed coordination of experience knowledge in the interface inspection process of the case organization. The interface change process had been used for a few years and a pattern-approach had been utilized to capture critical recurring issues. The problem with the existing process was the lack of official model or process to describe how the documentation under inspection should be reviewed. As an outcome for the research a SW change context model was introduced to help conceptualize the role of the inspection process and related dependencies within the software change context. The identified and needed knowledge based artifacts were generated following the preliminary model and stored in the intranet for the stakeholders to utilize. Coordination and validation of the experience knowledge based artifacts were considered to provide for control and consistency of the organizational culture. The statistical abstract of processing changes indicated improvements in the process. Coordinated utilization of experience knowledge and awareness of that matter within stakeholders were concluded to affect the atmosphere of commitment to the process. Coordination provided the ability for the organization to utilize its power, knowledge, in a convenient and effective way.




Absorbing software testing into the Scrum method

Tuomikoski Janne (14.5.2008)
This thesis handles the topic, how to absorb software testing into the Scrum method. The focus is this thesis is in target organization, which recently started the transition from waterfall software development model to the Scrum method. With the Scrum method the organization wanted to get more accurate software delivery and easier controlled development process with independent Scrum teams.
The result showed that testing can and even should be absorbed to the Scrum method. Most important thing and power in the Scrum are self-organizing teams, which can deliver a piece of working software after each sprint. To achieve this situation, test engineers and developers should together form a Scrum team, and each individual should be able to focus and commit totally to teams common goal.
Every work that the Scrum teams are doing should come through one channel, which is the product backlog. This one channel helps to coordinate, plan and estimate the work. The Product backlog can be used to control both coding and testing work. In target organization the Product Owner team was established to maintain the Product Backlog. The test manager, who used to lead the former testing team, joined the Product Owner team and can affect to the testing work through Product Backlog. Executed actions to absorb testing into the Scrum method were successful. Some of the actions were strongly supporting the basic theories of the Scrum method, and some actions were a bit modified to meet the target organization needs.




Risk based testing using test patterns

Kauppinen Timo
The research topic of this thesis is study how a test strategy could be defined and how software test design methods could be selected to reduce software product risks caused by software faults. Another aim of the study is to find out whether such an approach for test design would be useful in a real software project.
The study was motivated by the need to understand how test design methods and test patterns should be selected and justified so that they would be focused on finding especially the kind of faults which might have high impact.
The study was conducted as a case study and it was done in Nokia Devices organization and the qualitative evaluation of the approach was done using a questionnaire. The main finding of the study was that it was possible to derive a test strategy by using identified product risks as an input and it was possible to justify the selection of used test design method by the fault model the test design method in question addresses. The result of the study was that the proposed approach was deemed useful in a real software organization.




Generating test cases from behavioral models

Könönen Hannu
The thesis presents a technique, how to generate test cases from behavioral models. The technique is based on the model-based testing and as usually the user interface is described by means of an abstract model based on the state transition model. The model-based approach makes easier the updates to behavioral model and through that the generating of test cases, compared to manual updating of test scripts. Also the change management will be easier, which for one's part supports agile methods, in which the user interface is under continuous development.
The author introduces also software, which supports the construction of the user interface model and test cases. The use of the software is illustrated by a tiny example. The software is based on the use of keywords and thus its use does not require programming skills and the used notation is easy to learn.




Possibilities for test automation: Case Cross-Customer Cross-Distribution

Virta Anne
The author researches in thesis the possibilities of test automation in the Cross-Customer Cross-Distribution (CCD) system. The testing characteristics and test automation benefits and weaknesses in the case project were studied. The test automation aspects are considered in graphical user interface, in particular. This gives a possibility to use the capture/replay strategy, in which test cases can be recorded and run step by step from the user interface. Also the data-driven strategy can be used. When combining these two strategies and their testing capabilities, the CCD can be tested with a wider coverage.
Regression and smoke testing suit well to the CCD project, because a lot of same kinds of tests are done in the iterations from the beginning to the end. Most of the basic functionalities of the CCD do not change and they can be tested with the same test cases during each of the iterations. A customer release of CCD is made about twice a year where new features are added. The old functionalities need to be tested in each release, which supports regression testing.
In the tool evaluation a managed standard process was used. The evaluation of the tools based on the user needs (gathered by means of a questionnaire) and the guidelines found from the literature.
The result of the research shows that the project should not proceed to test automation before the testing is better organized and the testing process is well defined. The results of the tool evaluation can be used in future, when the testing process is improved.