Applying Test Suite Reduction Techniques on GUI Tests
Diego Hilario Barinas
Rochester Institute of Technology
Rochester, NY, USA
dh7445@rit.edu
ABSTRACT
Regression testing is the type of testing that occurs every time a
change to the application is made. It implies executing all the test
suites of the project to ensure that the new changes do not break
other parts of the code. The problem with regression testing is that
it requires lots of effort and time depending on the size of the suites
(which increases as the software evolves). To tackle this problem,
several techniques have been created to reduce the suites while
maintaining its fault detection capability rate. Most of the state-of-
the-art techniques focus on unit test suite and while not completely
ignoring other types of suites, they are often not applied on them.
It is well known that web-based GUI tests are larger and take
longer to execute than unit tests, thus the existing problem increases
for this type of tests. While different approaches have been done to
reduce web-based GUI suites, there hasn’t been any research that
applies the existing state-of-the-art approaches on this type of suite.
We aim at identifying the most relevant techniques for Test
Suite Reduction and applying them on Web-based GUI Tests suite,
specifically suites created using the Selenium framework to see
how well and how efficient these approaches are on this type of
tests.
Our study will be performed in two settings, a controlled
experiment and a real-world scenario experiment. By doing the
experiment in a controlled setting first, we will assure that the
results of our implementation of the approaches is valid and that it
can be used on real scenarios. We will also use this first experiment
as a way of improving our methodology, if needed. Our research
will show whether the creation of completely new approaches are
needed or if the ones that have already been proposed and used
throughout the years are enough, with or without modifications to
handle the different type of suites.
CCS CONCEPTS
• Software and its engineering → Software testing and
debugging;
KEYWORDS
test minimization, test reduction, test redundancy, regression
testing
1 INTRODUCTION
Quality assurance has many different focuses throughout the
Software Development Life Cycle, one of which is making sure
that the product, on each of its iterations is of compliance with all
of the requirements that it was set to implement (e.g., on a Scrum
based team, on a given iteration, the set of requirements would be
those defined at the beginning of the iteration as the sprint backlog),
this process is commonly referred to as testing.
Every software (if something of value is expected out of it)
needs to be tested to assure that it is working as intended, and to
find and remove bugs. Quality Assurance has, is and will continue
to be one of the most important areas on Software Engineering.
This claim will continue to be true and valid even if the industry of
software engineering manages to somehow, develop to the point
were creating bug-free software becomes an easy to achieve task
without the need of verification. This is because Software Quality
Assurance not only makes sure that the software under test does not
contain failures, but it also ensures that the software behaves as per
the set of requirements it was set to implement. For example, an
application could accomplish something magnificent like resolving
historically unsolvable mathematical problems, but if it was not
required to do so, it could be of no value to the product owner and
stakeholders.
Since source code can get extensive for a single application, it
is very likely that the amount of testing code that is going to be
needed to properly test the application will be extensive as well.
Whenever part of the source code is modified, or new enchantments
are added, test cases need to be created or modified to properly test
those changes. The action of re-testing existing parts of the
software after modifications is referred to as regression testing.
Due to the addition of new functionality to the application, new
test cases need to be added to the suites. For this reason, old test
cases can become unnecessary or the new ones might be redundant
or partly redundant. A test case is said to be redundant if the same
test objective is already satisfied by another test case [3] or partly
redundant if the test case shares steps with another test case. In
theory, when adding new test cases to a suite, the tester should
ensure that it is not redundant by checking the suite for redundancy
before the addition. In practice, since this task requires a lot of
effort depending on the size of the test suite it is often ignored, and
the suites start accumulating redundant test cases. To ensure that
the suites do not contain redundancy or unnecessary test cases,
Diego Hilario Barinas
improving the testing efforts, a proper test suite management is
desired.
The problem with having redundant test cases is not only that
they might be unnecessary since they test parts of the applications
that have already been tested, but that the amount of test cases in a
suite directly impacts its execution time. The more test cases that
are in a suite the longer it takes to be executed and therefore, the
longer it takes to complete the analysis of the results [12]. During
the analysis, bugs and issues might be found requiring fixes from
the developers (which can also take a considerable amount of time),
after the fixes, because of the new modifications of code, the
regression suite would need to be executed again thus, repeating the
process. This process adds to the need of a proper test suite
management during the life-cycle of a software.
There are many types of test suite management techniques and
many approaches to accomplish it within each technique. Yoo et al
[18], identify 3 types of techniques: Test Suite Minimization (also
called Test Suite Reduction), Test Case Selection and Test Case
Prioritization. On this paper, we will focus on Test Suite Reduction
(TSR) techniques. The goal of TSR technique is to identify the
redundant test cases on a suite [14]. After the identification of the
redundant test cases, approaches take different course of actions to
reduce them from the suite. Adequate approaches aim producing a
reduced test suite that completely covers all the test requirements
of the original suite, while Inadequate approaches produce a
reduced test suite that does not necessarily preserves the original
test requirements [6].
The process of executing a regression test suite can vary in time
depending on the types of test cases that are in the suite. Unit test
suites are often the focus of the majority of research done to achieve
Test Suite Reduction. While one of the most important and
common types of testing, unit tests are often small and fast to be
executed (still the accumulation of them creates an execution time
problem), other types of test cases are often not as small and take
more time to be executed. Tests for UI, mobile or web for example,
often require long sets of interaction with the GUI (Graphic User
Interface) of the software under test. Regularly, this type of tests
verifies or validates a complete functionality of the system, while
unit test cases only aim at testing individual units of code (methods
of the source code for example).
Different approaches [7, 1, 4] have been proposed throughout
the years to reduce these types of tests (that interact with the GUI).
Others, like the work of Vahabzadeh et al [17] base their research
on reducing desktop unit test cases but indicate that their proposed
approach can be used to reduce other types of test cases such as the
ones mentioned before. Since most of the research regarding TSR
focuses on unit test cases, those proposed approaches are often
more developed and studied. However, to the best of our
knowledge there has not been any attempt at verifying whether the
state-of-the-art approaches for TSR can successfully be applied to
suites with high GUI interaction. For the rest of the proposal, we
will refer to the web-based GUI tests, specifically those created
using a testing framework such as Selenium [13], as GUI Tests. We
will be focusing on tests suites written using Selenium because
while being one of the most used framework for web test creation,
it is also an open source tool. With this we are expecting to reduce
dh7445
efforts in finding test subjects that are publicly available and use
GUI Tests.
2 RESEARCH OBJECTIVE
Most of the research done regarding Test Suite Reduction focuses
on reducing the redundancies found in unit test cases. Unit test
cases are often small and, individually, do not take a long time to
be executed in comparison with other types of test cases such as
GUI test cases. When joined together with other unit test cases,
due to the large amount of unit test cases needed to correctly test a
software, the execution time increases considerably. This increase
reflects the same way on other types of test cases (the more test
cases, the longer it takes for the suite to be executed and analyzed).
Since most of the created techniques to achieve TSR have only
been applied on unit test cases, our purpose is to analyze how well
those techniques perform when applied on different types of
testing, specifically web-based GUI Test Suites created using the
Selenium Framework. The need for this kind of research was
identified by Orso et al [21], they mention how new techniques are
constantly being created but very few of them are often applied in
different software testing domains.
3 RESEARCH QUESTIONS
Our research aims at answering the following research questions:
RQ1 How prevalent are redundant test cases in GUI Test
Suites?
RQ2 Is it possible to apply TSR techniques that were
developed for unit test suites on GUI Tests?
RQ3 What is the redundancy reduction for each technique
when applied on GUI Tests?
RQ4 What is the achieved reduction in execution time of the
test suites?
RQ5 What percentage of the test requirements are preserved
on the reduced suites?
Since most of the literature regarding TSR focuses on unit test
suites, research question 1 focuses on identifying whether GUI
Tests accumulate redundant test cases the same way unit test cases
do. If results show that they do not accumulate any type of
redundancy between their tests, then the rest of the research might
not be worth finishing. The second question, aims at identifying if
it is possible or not to apply some of most used TSR techniques
GUI Tests suites. Research question 3 and 4 take each of the
techniques and looks into identifying how efficient they are on
terms of reduced percentage and execution time. The last research
question aims at checking if the reduced suite preserves the same
fault detection capability rate of the original suite.
4 RESEARCH PLAN
4.1 Techniques
To conduct the research, a set of TSR techniques will need to be
studied and analyzed. The selection of these techniques will be
based on different criteria. First the approach will need to be
2
Diego Hilario Barinas
considered a state-of-the-art approach in TSR. Extensive literature
that refers to these techniques is available [21] and based on them,
the techniques will be identified. In addition, some techniques that
might not be considered state-of-the-art techniques, due to their
recent publication or lack of attention they might have received
since its publication, will be taken into consideration based on their
ability/easiness to be applied to different types of testing, in
particular, GUI Tests.
For the study, the range of selected techniques could vary
depending on their adequacy criteria (adequate or inadequate) to
provide a broader generalization of the results. For adequacy type
of TSR techniques, the possible candidates to be selected are
Greedy [9, 11], GE [2], GRE [3], and HGS [8]. These 4 techniques,
while being proposed several years ago, are still considered the
state-of-the-art techniques for TSR by many of the research that has
been done in recent years [15, 6]. These four algorithms are widely
used in prior work regarding TSR [14, 16, 18, 19, 20].
One of the recent approaches that is to be included on this
research, is the one proposed by Vahabzadeh et al [17] on their
paper “Fine-Grained Test Suite Minimization”. This approach falls
under the category of adequate techniques since it aims at reducing
a test suite while preserving its behavior. This approach, using a
test model created by analyzing the test cases at the statement level,
reorganizes redundant test cases into one. By doing this, while
executing the test cases during regression testing, the redundant
part of the test cases will only need to be executed once and then
the different assertions for each test are executed. The particularity
of this approach is that it does not delete the redundant test cases
but instead generates a new reduced suite and keeps a link to the
original suite (each set of redundant test cases in the original suite
will be mapped to one or many test cases of the reduced suite).
To comply with the previously mentioned criteria, inadequate
state-of-the-art techniques are to be identified. As of now, to the
best of our knowledge there is no research that identifies state-of-
the-art techniques for inadequate TSR. Shi et al [14] however,
indicate that each adequate approach can be modified and
converted into an inadequate approach. This would be done by
simply creating a reduced suite that does not necessarily satisfies
all the requirements of the original one. Thus, the same previously
mentioned set of adequate techniques could be modified and used
as inadequate techniques for this study.
4.2 Tools
Most or all of the previously mentioned techniques have long been
proposed and researched widely in the past years, providing
improvements, tweaks, new approaches based on them and tools on
which the approaches are implemented. To perform our
experiment, we will need to identify or create ourselves a set of
tools on which each of the TSR approaches are implemented.
If we decide to create the tools ourselves, we will need to
thoroughly study each of the approaches (which in order to
complete the research we will have to do regardless) and come up
with the best way to implement them. The upper hand in creating
the tools ourselves is that, based on what each approach allows, we
will be able to alleviate some constraints on the test subject
dh7445
selections, for example, some already implemented tools, like the
one proposed by Vahabzadeh et al [17] can only be executed on
software that have been built using Java, use the Maven build
system [22] and their test suites are create using a version of JUnit
[10]. By creating the tools, we could also add extra features that
could come in handy to analyze the results of the experiments.
Other research like the one by Zhong et al [20], to complete their
objectives, implement 4 different approaches in a single suite. On
this case however, they do not make their implementation publicly
available so, we will need to find some research that do make their
implementation of the approaches available to everyone if we
decide to follow this path.
Rehman et al [23] created a survey on TSR frameworks and
tools. On their research, they look at different state-of-the-art TSR
frameworks and tools that have been proposed to address the test
suite size problem. For our research, the identification or
implementation of tools will greatly benefit based on their work
since they identify the strengths and weaknesses of the tools, which
we could use to assess the correct way of using the tools in our
research.
4.3 Test Subjects
To validate that the different approaches can be executed on GUI
Tests, we will first need to create or select a public repository on
which, the amount of test cases, the amount of redundant test cases,
number of bugs, lines of code and some more possible attributes of
the suite are identified before running the experiment. By
identifying this data before running the experiment, we will
guarantee that the obtained results will be correct, and it will allow
us to provide answers to our research questions in a controlled
experiment setting.
After executing the experiment in a controlled environment, we
will proceed on selecting (no more than 20) real-world software and
their test suites. We will only select 20 test subjects because with
that amount we can provide a wide range of software systems
pertaining to different domains. By not selecting an extremely large
set of test subjects, we ensure that the obtained results will be easy
to handle and assess. For this experiment, we need to keep in mind
that each of the selected approaches will be executed against each
of the selected test subjects. We currently have selected at least 5
different approaches (only the adequate ones), meaning that we will
have a minimum of 20*5 = 100 reduced test suites. For each of the
reduced test suites we will assess and identify the reduction
percentage, the resulting reduced amount of test cases, test
requirement compliance percentage, among other characteristics.
We will also need to execute each of the resulting reduced test
suites against its pertaining software, in order to obtain, among
others, the execution time and fault detection capability rate.
To account for possible variability and randomness when
generating the reduced test suites and when calculating the
execution time and fault detection capability rate, we will perform
the process at least 5 times for each approach on each test subject.
For the selection of the test subjects, we will also need to pay
attention to the technologies used in their implementation. We will
3
Diego Hilario Barinas
either need to adapt the selection of the test subjects to what the
tools support or adapt the tools to the selected test subjects.
5 EVALUATION
The research will consist on one experiment done in two different
settings, a controlled one and one that simulates a real-world
scenario. The evaluation process for each of the settings will be
pretty much the same with the difference that the results obtained
on the controlled experiment will be used to learn, identify
improvements and do modifications to the approaches (in case they
are needed so that they can actually be executed on GUI tests).
The evaluation will start by identifying whether each approach
can be applied on GUI Tests and if not, what necessary
modifications need to be done to them in order to do so. After that
we will be running each approach against the test subject so to
obtain a reduced test suite corresponding to each technique. The
reduced suites will then be analyzed to obtain the resulting amount
of test cases and total lines of code. For each reduced suite, we will
also analyze the possibility of obtaining different requirement
coverage according to some criteria, the criteria to be used is still
pending for definition and some more research needs to be done
regarding this issue. Criteria such as code coverage (statement,
branch, etc.) are often used for Unit tests were the source code is
available to measure the coverage (white-box type of test), on GUI
tests (black-box) the measurement of this type of requirement
coverage arises a challenge. Research like the one done by Devaki
et al [7] mention a different type of requirement coverage for GUI
tests based on the test states and GUI elements. After identifying
the properties of the resulting reduced suites, each of them will be
executed against the test subjects, with these executions we look
into identifying the execution time and the number of bugs that they
are able to detect.
As previously mentioned, to account for possible variability in
the results, the process will be executed 5 times for each approach
on each system. Therefore, the results obtained for each approach
will be presented as an average of all these executions. For
example, the execution time for approach #1 on test subject #1 will
be the sum of all the execution times obtained divided by the
amount of times it was executed (5).
After all the results (averages) are obtained, a comparison with
the original test suites will be performed to identify how the
reduced suite stands against them in terms of: amount of test cases,
lines of code, execution time, fault detection capability, etc. With
these processes we aim at identifying how effective each approach
is on reducing GUI Tests. We will also be comparing the results of
the approaches against each other to identify which one performs
the best on each characteristic.
The complete workflow of our approach can be seen on Figure
1.
6 LIMITATIONS AND THREATS
Software exists on a wide range of different domains, so our results,
while definitely trying to be global, will most likely not be. In order
to cover some of the different domains, we chose to perform the
dh7445
Figure 1. Proposed Methodology
study on 20 different systems belonging to different domains. The
range of different domains could be countless and therefore it
would be practically impossible to perform the experiment on each
domain, and even so to provide a full globalization of results, we
would need to perform the experiment on multiple systems on each
of the domains.
If we decide to implement the approaches in automated tools
ourselves or if we make use of public tools that already implement
the approaches, our results could be affected by issues or bugs
within the tools. Although we will have the controlled experiment
to test the tools and perform fixes (in case they are needed), we
could never be 100% sure that it is bug-free. To reduce this threat,
we will perform all the steps of the experiment 5 times for each
approach and the results will be in form of averages of all the
executions. This way we account for possible variability on the
results when reducing the suites and when executing the suites
against the system.
To minimize threats to reliability validity, we will make all our
implementations publicly available or all the used tools will also
have to be publicly available. We will also provide all the details
regarding the approaches used, tools, test requirements, results and
process so that anyone that wishes to replicate our experiment will
able to do it without major setbacks.
7 CHALLENGES
There are a number of significant challenges that we will need to
overcome in order to complete the research and so that the obtained
results are of value to the Software Engineering community. First
of all, given the recent identification and proposal of inadequate
Test Suite Reduction techniques, it will be challenging to identify
the approaches that fall into this category that will be used in the
experiment. While some authors like Coviello et al [6] mention that
because of the lack of proposed inadequate approaches, an
approach that they have proposed [5] could be considered as one of
the inadequate state-of-the-art techniques for TSR, this opinion
could clearly be biased. Regardless, it could be worth examining
their approach and comparing it against others.
The issue with the identification of the approaches continues
with their implementation. We will need to implement each of the
approaches in an automated tool since previous research has proved
that manually identifying redundant test cases and analyzing the
4
Diego Hilario Barinas dh7445

reduced suites is unfeasible. While some tools have already been to refer to the original suite when new test cases are to be added or
created to implement one or many of the TSR approaches, they if modifications to the already existing test cases are due.
could or could not be of use to our experiment based on what the
implementation allows. 9 CONCLUSIONS
Another challenge to overcome will be the identification of
Test Suite Reduction remains, alongside or above other test suite
applicable test requirements. Most of the approaches use code
management techniques, a predominant research area in software
coverage as their main test requirement objective. Some aim at
engineering. The need for reducing execution times of regression
covering all branches, statements, classes, etc. The commonality
testing is an everyday issue. Regression suites are constantly
that these approaches have is that they are all used for unit test cases
increasing in size as the software they belong to evolves. In the fast
were the source code of the software under test is available and thus
pacing world, we live on, were the success of an application could
facilitating the code coverage analysis. On GUI web tests were the
be determined by how soon it was released compared to others, all
test cases do not directly interact with the source code, but with the
the stages of the software development life-cycle need to be
software, it is a challenge to assess the coverage of the source code.
completed as soon as possible. Spending too much time doing
regression is not an option.
8 LITERATURE REVIEW The goal of our proposal is to identify whether it is possible to
There are multiple ways of to achieve a proper test suite apply current Test Suite Reduction Techniques in GUI Tests. GUI
management. Yoo et al [18] identify and differentiate 3 classes of Tests are often used in both the industry and on open source
techniques, (i) Test Suite Minimization (sometimes called Test projects as one of the main ways of identifying bugs and problems
Suite Reduction). (ii) Test Case Selection and (iii) Test Case in software systems. The need for this kind of research arises based
Prioritization. On their paper, each technique is defined. Test Suite on the continuous creation of new approaches but the lack of
Minimization means to find a representative set of the original test applying the already existing ones in different domains of software
suite, that satisfies all the requirements of the original suite. Test testing.
Case selection means to find a subset of the test suite that can be As a result of this research, we will provide a set of identified
used to test a modified version of the program. Test Case techniques and their implementations in tools that can be used to
Prioritization means to order the set of test cases to maximize a reduced GUI Tests. This way developers will be able to efficiently
desirable property of the suite (e.g. fault detection rate). Zhong et reduce their GUI test suites by having a set of identified techniques
al [20] agrees on the concept of test suite reduction by Yoo and and how well they perform the reductions. Reducing their suites
expands on that “obviously” reducing a test suite will result in a size directly reflects on reducing regression execution times which
decrease of execution time and testing costs. means they will most likely be able to release their software sooner.
Throughout the years, TSR techniques have been categorized as
“Adequate” and “Inadequate” techniques [6]. Inadequate
techniques aim at reducing the suite without necessarily keeping
the same coverage criteria of the original suite. Adequate
approaches (Greedy [9], GE [2], GRE [3], and HGS [8]) focus on
reducing the suite and maintaining the coverage criteria of the
original suite. As Shi et al [15] indicate, all these techniques start
with an empty reduced test case list and progressively add test cases
to it based on different criteria. For our experiment, we will be
using the previously mentioned adequate approaches (considered
state-of-the-art TSR techniques) to see how well they perform
when applied on GUI web test suites. For the selection of the
Inadequate approaches, we will take into consideration the work
done by Coviello et al [6] where they indicate that since the
inadequate approaches are relatively new, the state-of-the-art
techniques on this category would be modified versions of the
adequate approaches.
In general, TSR techniques look at completely removing the set
of redundant test cases from the test suite, thus producing a reduced
test suite [18]. There have been other approaches such as the one
by Vahabzadeh et al [17] that aim at minimizing the test suite but
keeping both the original and the reduced suite. Our possible
implementation of the selected approaches will look into keeping
both suites for comparison purposes. This approach makes it
possible to execute the reduced test suite for regression testing and

5
Diego Hilario Barinas dh7445

Conference on Software Engineering (ICSE '18). ACM, New

REFERENCES York, NY, USA, 210-221. DOI:
https://doi.org/10.1145/3180155.3180203
[18] S. Yoo and M. Harman. 2012. Regression testing minimization,
[1] Stephan Arlt, Andreas Podelski, and Martin Wehrle. 2014. selection and prioritization: a survey. Softw. Test. Verif. Reliab.
Reducing GUI test suites via program slicing. In Proceedings of 22, 2 (March 2012), 67-120.
the 2014 International Symposium on Software Testing and DOI=http://dx.doi.org/10.1002/stv.430
Analysis (ISSTA 2014). ACM, New York, NY, USA, 270-281. [19] Lingming Zhang, Darko Marinov, Lu Zhang, and Sarfraz
DOI: https://doi.org/10.1145/2610384.2610391 Khurshid. 2011. An Empirical Study of JUnit Test-Suite
[2] T. Y. Chen and M. F. Lau. 1995. Heuristics Towards the Reduction. In Proceedings of the 2011 IEEE 22nd International
Optimization of the Size of a Test Suite. In SQM. 415–424. Symposium on Software Reliability Engineering (ISSRE '11).
[3] T.Y. Chen, M.F. Lau, A new heuristic for test suite reduction, IEEE Computer Society, Washington, DC, USA, 170-179.
Information and Software Technology, Volume 40, Issues 5–6, DOI=http://dx.doi.org/10.1109/ISSRE.2011.26
1998, Pages 347-354, ISSN 0950-5849, [20] Hao Zhong, Lu Zhang, and Hong Mei. 2008. An experimental
https://doi.org/10.1016/S0950-5849(98)00050-0. study of four typical test suite reduction techniques. Inf. Softw.
[4] Lin Cheng, Zijiang Yang, and Chao Wang. 2017. Systematic Technol. 50, 6 (May 2008), 534-546.
reduction of GUI test sequences. In Proceedings of the 32nd DOI=http://dx.doi.org/10.1016/j.infsof.2007.06.003
IEEE/ACM International Conference on Automated Software [21] Alessandro Orso and Gregg Rothermel. 2014. Software testing: a
Engineering (ASE 2017). IEEE Press, Piscataway, NJ, USA, 849- research travelogue (2000–2014). In Proceedings of the on Future
860. of Software Engineering (FOSE 2014). ACM, New York, NY,
[5] C. Coviello, S. Romano, G. Scanniello, A. Marchetto, G. Antoniol USA, 117-132. DOI: http://dx.doi.org/10.1145/2593882.2593885
and A. Corazza, "Clustering support for inadequate test suite [22] Apache Maven Project. https://maven.apache.org
reduction," 2018 IEEE 25th International Conference on Software [23] Saif Ur Rehman Khan, Sai Peck Lee, Raja Wasim Ahmad, Adnan
Analysis, Evolution and Reengineering (SANER), Campobasso, Akhunzada, Victor Chang, A survey on Test Suite Reduction
2018, pp. 95-105. frameworks and tools, International Journal of Information
doi: 10.1109/SANER.2018.8330200 Management, Volume 36, Issue 6, Part A, 2016, Pages 963-975,
[6] Carmen Coviello, Simone Romano, and Giuseppe Scanniello. ISSN 0268-4012,
2018. An empirical study of inadequate and adequate test suite https://doi.org/10.1016/j.ijinfomgt.2016.05.025.
reduction approaches. In Proceedings of the 12th ACM/IEEE
International Symposium on Empirical Software Engineering and
Measurement (ESEM '18). ACM, New York, NY, USA, Article
12, 10 pages. DOI: https://doi.org/10.1145/3239235.3240497
[7] Pranavadatta Devaki, Suresh Thummalapenta, Nimit Singhania,
and Saurabh Sinha. 2013. Efficient and flexible GUI test execution
via test merging. In Proceedings of the 2013 International
Symposium on Software Testing and Analysis (ISSTA 2013).
ACM, New York, NY, USA, 34-44. DOI:
https://doi.org/10.1145/2483760.2483781
[8] M. Jean Harrold, Rajiv Gupta, and Mary Lou Soffa. 1993. A
methodology for controlling the size of a test suite. ACM Trans.
Softw. Eng. Methodol. 2, 3 (July 1993), 270-285.
DOI=http://dx.doi.org/10.1145/152388.152391
[9] David S. Johnson. 1973. Approximation algorithms for
combinatorial problems. In Proceedings of the fifth annual ACM
symposium on Theory of computing (STOC '73). ACM, New
York, NY, USA, 38-49.
DOI=http://dx.doi.org/10.1145/800125.804034
[10] JUnit https://junit.org/junit5/
[11] Zheng Li, Mark Harman, and Robert M. Hierons. 2007. Search
Algorithms for Regression Test Case Prioritization. IEEE Trans.
Softw. Eng. 33, 4 (April 2007), 225-237. DOI:
https://doi.org/10.1109/TSE.2007.38
[12] G. Rothermel, R. H. Untch, Chengyun Chu and M. J. Harrold,
"Test case prioritization: an empirical study," Proceedings IEEE
International Conference on Software Maintenance - 1999
(ICSM'99). 'Software Maintenance for Business Change' (Cat.
No.99CB36360), Oxford, England, UK, 1999, pp. 179-188. doi:
10.1109/ICSM.1999.792604
[13] Selenium https://www.seleniumhq.org
[14] August Shi, Alex Gyori, Milos Gligoric, Andrey Zaytsev, and
Darko Marinov. 2014. Balancing trade-offs in test-suite reduction.
In Proceedings of the 22nd ACM SIGSOFT International
Symposium on Foundations of Software Engineering (FSE 2014).
ACM, New York, NY, USA, 246-256. DOI:
https://doi.org/10.1145/2635868.2635921
[15] August Shi, Alex Gyori, Suleman Mahmood, Peiyuan Zhao, and
Darko Marinov. 2018. Evaluating test-suite reduction in real
software evolution. In Proceedings of the 27th ACM SIGSOFT
International Symposium on Software Testing and Analysis
(ISSTA 2018). ACM, New York, NY, USA, 84-94. DOI:
https://doi.org/10.1145/3213846.3213875
[16] August Shi, Tifany Yung, Alex Gyori, and Darko Marinov. 2015.
Comparing and combining test-suite reduction and regression test
selection. In Proceedings of the 2015 10th Joint Meeting on
Foundations of Software Engineering (ESEC/FSE 2015). ACM,
New York, NY, USA, 237-247. DOI:
https://doi.org/10.1145/2786805.2786878
[17] Arash Vahabzadeh, Andrea Stocco, and Ali Mesbah. 2018. Fine-
grained test minimization. In Proceedings of the 40th International