UNIVERSIDADE FEDERAL DE PELOTAS

Centro de Desenvolvimento Tecnológico

Curso de Bacharelado em Ciência da Computação

Trabalho de Conclusão de Curso

Optimizing Patrolling Strategies for Multiple UAVs Through Genetic Algorithms

Patrick Guimarães de Vargas

Pelotas, 2019
 Patrick Guimarães de Vargas

Optimizing Patrolling Strategies for Multiple UAVs Through Genetic Algorithms

Trabalho de Conclusão de Curso apresentado

ao Centro de Desenvolvimento Tecnológico da
Universidade Federal de Pelotas, como requisito
parcial à obtenção do título de Bacharel em Ci-
ência da Computação.

Orientador: Prof. Dr. Paulo Roberto Ferreira Junior

Pelotas, 2019
 Insira AQUI a ficha catalográfica
(solicite em http://sisbi.ufpel.edu.br/?p=reqFicha)
 Patrick Guimarães de Vargas

Optimizing Patrolling Strategies for Multiple UAVs Through Genetic Algorithms

Trabalho de Conclusão de Curso aprovado, como requisito parcial, para obtenção

do grau de Bacharel em Ciência da Computação, Centro de Desenvolvimento
Tecnológico, Universidade Federal de Pelotas.

Data da Defesa: 3 de dezembro de 2019

Banca Examinadora:
Prof. Dr. Paulo Roberto Ferreira Jr (orientador)
Doutor em Computação pela Universidade Federal do Rio Grande do Sul.

Prof. Dr. Taua Milech Cabreira

Doutor em Computação pela Universidade Federal de Pelotas.

Profa . Dra . Lisane Brisolara de Brisolara

Doutora em Computação pela Universidade Federal do Rio Grande do Sul.
Dedico. . .
 AGRADECIMENTOS

Agradeço. . .
Só sei que nada sei.
— S ÓCRATES
 RESUMO

VARGAS, Patrick Guimarães de. Otimizando estratégias de patrulhamento para

múltiplos VANTs através de algoritmos genéticos. Orientador: Paulo Roberto
Ferreira Junior. 2019. 46 f. Trabalho de Conclusão de Curso (Ciência da Computação)
– Centro de Desenvolvimento Tecnológico, Universidade Federal de Pelotas, Pelotas,
2019.

O problema de patrulhamento consiste em inspecionar uma área de interesse

regularmente e coletar informações usando os sensores a bordo dos Veículos Aéreos
Não Tripulados (VANTs). Para esta tarefa, o algoritmo NC-Drone e um conjunto de
estratégias de patrulhamento, como Evaporation (ES), Time-based (TS) e Quad-
rant (QUAD), foram propostos. Essas estratégias são compostas de parâmetros
cujos valores foram selecionados empiricamente. Neste trabalho, propomos uma
otimização de parâmetros para essas estratégias usando um algoritmo genético
(AG). Analisamos estratégias simples e várias combinações em cenários livres de
obstáculos e contendo zona de exclusão aérea (ZEA). Resultados experimentais
mostram que as abordagens com parâmetros otimizados superam as estratégias com
valores empíricos. O QUAD-ES GA apresenta os melhores resultados entre todos os
métodos, proporcionando uma distribuição mais uniforme das visitas. Essa estratégia
obtém uma melhoria de até 18,3 % em relação ao algoritmo NC-Drone, dependendo
do cenário.

Palavras-chave: Patrulhamento. VANTs. Algoritmo-Genético. Drones.

 ABSTRACT

VARGAS, Patrick Guimarães de. Optimizing Patrolling Strategies for Multiple

UAVs Through Genetic Algorithms. Orientador: Paulo Roberto Ferreira Junior.
2019. 46 f. Undergraduate Thesis (Computer Science) – Technology Development
Center, Federal University of Pelotas, Pelotas, 2019.

The patrolling problem consists of inspecting an area of interest regularly while

gathering information using the onboard sensors of the Unmanned Aerial Vehicles
(UAVs). For this task, the NC-Drone algorithm and a set of patrolling strategies, such
as Evaporation (ES), Time-based (TS), and Quadrant (QUAD) were proposed. These
strategies are composed of parameters whose values were empirically selected. In
this work, we propose a parameter optimization for these strategies using a Genetic
Algorithm (GA). We analyze single strategies and several combinations over scenarios
free of obstacles and containing no-fly zones (NFZ). Experimental results show that
the approaches with the optimized parameters overcome the strategies with empirical
values. The QUAD-ES GA presents the best results among all methods, providing
a more uniform distribution of visits. This strategy obtains an improvement of up to
18.3% over the NC-Drone algorithm, depending on the scenario.

Keywords: Patrolling. UAVs. Genetic-Algorithm. Drones.

 LIST OF FIGURES

1 Geometric flights patterns . . . . . . . . . . . . . . . . . . . . . . . . 19

2 The scenario is discretized into a regular grid. . . . . . . . . . . . . 20

3 Random decision of RSTM. . . . . . . . . . . . . . . . . . . . . . . . 22

4 UAV performing the NC-Drone algorithm. . . . . . . . . . . . . . . . 24

5 Evaporation Strategy in three distinct cycles of simulation. . . . . . . 25
6 Quadrant strategy with four UAVs. . . . . . . . . . . . . . . . . . . . 26

7 Population, Chromosomes and Genes. . . . . . . . . . . . . . . . . 28

8 Crossover Process. . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
9 Mutation Process. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
10 Genetic algorithm process. . . . . . . . . . . . . . . . . . . . . . . . 30

11 Test platform built with PyGame. . . . . . . . . . . . . . . . . . . . . 33

12 Trajectory generated by the QUAD-ES GA strategy. . . . . . . . . . 38

13 Three different scenarios consisting of a 50 x 50 grid containing 10%
of no-fly zones marked in red. . . . . . . . . . . . . . . . . . . . . . . 39
 LIST OF TABLES

1 Update rules of the main real-time search methods . . . . . . . . . . 22

2 Summary of the Genetic Algorithm parameters. . . . . . . . . . . . . 32

3 Summary of the experiments parameters. . . . . . . . . . . . . . . . 35

4 Empirical and optimized parameters for single strategies. . . . . . . 36
5 Patrolling strategies with parameter optimization. . . . . . . . . . . . 37
6 Empirical and optimized parameters for combination strategies. . . . 37
7 Combination of patrolling strategies using GA. . . . . . . . . . . . . 39
8 Optimized parameters in scenarios with NFZ. . . . . . . . . . . . . . 40
9 QMI results for the combination of patrolling strategies using GA in
scenarios with NFZ. . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
10 SDF results for the combination of patrolling strategies using GA in
scenarios with NFZ. . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
11 NTM results for the combination of patrolling strategies using GA in
scenarios with NFZ. . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
 LIST OF ABBREVIATIONS AND ACRONYMS

VANTs Veículos Aéreos Não Tripulados

ES Evaporation Strategy
TS Time-Based Strategy
QUAD Quadrant Strategy
AG Algoritmo Genético
ZEA Zona de Exclusão Aérea
UAVs Unmanned Aerial Vehicles
GA Genetic Algorithm
NFZ No-Fly Zones
GCS Ground Control Station
RTSM Real-Time Search Methods
NC Node Counting
LRTA* Learning Real-Time A*
TVUR Thrun’s Value Update Rule
WVUR Wagner’s Value Update-Rule
ACO Ant Colony Optimization
CACOC Chaotic Ant Colony Optimization to Coverage
BF Back-and-Forth
SP Spiral Flight Pattern
QMI Quadratic Mean of Intervals
NTM Number of Turning Maneuvers
SDF Standard Deviation of the Frequencies
ET Evaporation Time
EF Evaporation Factor
TT Time Threshold
M Mean
SD Standard Deviation
 CONTENTS

1 INTRODUCTION . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
2 RELATED WORK . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
3 GEOMETRIC FLIGHT PATTERNS . . . . . . . . . . . . . . . . . . . . . . . 19
4 REAL-TIME SEARCH METHODS . . . . . . . . . . . . . . . . . . . . . . . . 20
5 PATROLLING STRATEGIES FOR NC-DRONE . . . . . . . . . . . . . . . . 23
5.1 Evaporation Strategy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
5.2 Time-based Strategy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
5.3 Quadrant Strategy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
6 GENETIC ALGORITHMS . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
6.1 Initialize Population . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
6.2 Fitness Function . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
6.3 Selection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
6.4 Crossover . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
6.5 Mutation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
7 OUR GENETIC OPTIMIZATION . . . . . . . . . . . . . . . . . . . . . . . . . 31
8 SIMULATION ENVIRONMENT . . . . . . . . . . . . . . . . . . . . . . . . . . 33
9 EXPERIMENTAL RESULTS . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
9.1 Single Strategies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
9.2 Combination of Strategies . . . . . . . . . . . . . . . . . . . . . . . . . . 37
9.3 Scenarios with No-Fly Zones . . . . . . . . . . . . . . . . . . . . . . . . 38
10 CONCLUSION . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
REFERENCES . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
1 INTRODUCTION

Unmanned Aerial Vehicles (UAVs) are autonomous aerial platforms with no pilot
aboard. These platforms are usually controlled by a human navigator on the ground,
a ground control station (GCS), or even an intelligent system. A large number of tasks
are solved with the help of these vehicles, such as search and rescue, inspection,
surveillance, collecting data, and payload delivery. Many of these tasks are related
to the Patrolling Problem, which consists of inspecting an area of interest regularly
while gathering information using the onboard sensors of the UAV (HEFFERAN; CLIFF;
FITCH, 2016).
In such a dynamic environment, the UAV must visit and revisit every point of the
area to keep the collected data updated. Thus, patrolling performance is measured
concerning the number of visits in each region, the uniform distribution of visits, and the
interval evenness between those visits (SAMPAIO; SOUSA; ROCHA, 2016). The UAVs
have different constraints compared to terrestrial vehicles, such as payload, endurance,
and maneuverability. Simple geometric flight patterns, such as back-and-forth and
spiral (ANDERSEN, 2014), are often employed in patrolling missions considering these
constraints. However, these movements have pre-defined and repetitive behavior that
can be easily interpreted by an external observer and avoided by an intelligent target,
for instance.
Real-Time Search Methods (RTSM) may be explored to suppress this flaw, per-
forming the patrolling over an area and searching for a target, while keeping motion
unpredictable. These methods were introduced to solve land vehicle problems (KOE-
NIG; LIU, 2001) and expanded to aerial missions (CABREIRA et al., 2018).
In the work of CABREIRA et al. (2018), it is proposed the NC-Drone algorithm, an
RTSM, which is an extension of the well-known Node Counting (NC)(PIRZADEH; SNY-
DER, 1990). This algorithm allows UAVs to explore a grid-discretized scenario through
a pheromone-scheme. Additional strategies were also implemented to improve the
NC-Drone performance, such as Time-based (TS), Evaporation (ES), and Quadrant
(QUAD). The TS considers not only the amount of pheromone to make a decision but
also the time when the last visit occurred. The ES can reduce pheromone in places not
 15

visited for a while to attract vehicles. Despite promising results regarding the perfor-
mance improvement of the algorithm, the parameters of the proposed strategies were
obtained empirically (KAPPEL et al., 2019).
This work proposes a parameter optimization for patrolling strategies based on the
NC-Drone algorithm using the Genetic Algorithm (GA). The optimization model em-
ploys a fitness function that can tune the parameters according to the selected perfor-
mance metric. We aim to obtain the set of parameters for the strategies by which its
best performance comes out considering a uniform distribution of visits. We compare
single and combined approaches with empirical values against methods with optimized
parameters by the GA. Experimental results show that the QUAD-ES GA strategy over-
comes all strategies, improving the NC-Drone up to 18.3% depending on the scenario.
This work is summarized as follows: The related work is discussed in Chapter
2. Geometric flight patterns and Real-time search methods are discussed in Chapter
3 and Chapter 4, respectively. Chapter 5 revises the NC-Drone algorithm and the
patrolling strategies. Chapter 6 revises the Genetic Algorithm. Chapter 7 presents
the genetic approach and describes the parameter optimization. The test platform is
presented in Chapter 8. Chapter 9 presents and discusses achieved results. Finally,
Chapter 10 comes up with the conclusion and mentions future works.
2 RELATED WORK

A survey on patrolling algorithms using multiple robots is presented in (PORTU-

GAL; ROCHA, 2011). The authors introduce the primary aspects and limitations of the
problem, along with studies and methods that aim to solve these issues. They also
classified a variety of works regarding perception, communication, coordination, and
decision-making strategy.
One of the most important papers on the RTSM study field is (KOENIG; LIU, 2001).
This work compares four classic heuristics for patrolling, Node Counting (NC), Lear-
ning Real-Time A* (LRTA*), Thrun’s Value-Update Rule (TVUR), and Wagner’s Value
Update-Rule (WVUR). Despite the tests being performed with land robots, these heu-
ristics can be extended and adapted to a UAV scenario (CABREIRA et al., 2018). In
the experiments presented in (KOENIG; LIU, 2001), the agents do not have any me-
mory. They only sense the ground when covering, communicating indirectly with other
robots by leaving marks on the floor. The simulation is robust and deals with com-
plex issues, such as marks getting destroyed, robots failing, and agents being moved
without realizing.
Four different heuristics are analyzed in a simulated experiment in (NATTERO et al.,
2014). These real-time heuristics are Edge Counting, PatrolGRAPH*, NC, and LRTA*.
As the NC heuristic overcomes the other methods in simulations, the authors employ
it in real flights with two UAVs, proving the real possibility of using RTSM in aerial
platforms.
In another work (KUIPER; NADJM-TEHRANI, 2006), an Ant Colony Optimization
(ACO) algorithm is adapted to coverage path planning using UAVs. This adaptation
relies on the idea of having a team of agents dropping repulsive pheromone in the
visited places. These agents share a map of the scenario, which contains the amount
of pheromone in each position. This method reduces the chance of vehicles to go to
high visited areas. Chaotic Ant Colony Optimization to Coverage (CACOC) algorithm
combines ACO with a chaotic dynamical model for surveillance missions (ROSALIE
et al., 2016). The mission operator has access to the pheromone map of the agents
and does not need any communication to track the vehicles. The main advantage of
 17

this deterministic model is that the next steps are traceable for whoever is executing
the mission while they are unpredictable for external observers.
An algorithm exploring the process of evaporation called EVAP is proposed to solve
the patrolling problem in multi-agent systems of an unknown environment (CHU et al.,
2007). In this approach, the pheromones lose intensity as time goes. This process
is employed to represent the arising uncertainty about an unvisited place. Later, Glad
et al. (GLAD et al., 2008) compared the EVAP algorithm against the VAW (WAGNER;
LINDENBAUM; BRUCKSTEIN, 1999), which is a similar strategy that also explores the
evaporation. In the EVAP method, agents move and then drop pheromone in the last
visited cell while the VAW algorithm does the opposite, dropping pheromone and then
moving. However, this particular behavior of the VAW leads to the encounter of agents
in the same cell more often. VAW has a more straightforward evaporation function than
EVAP. The authors also created a combination of both approaches called EVAW, which
has the evaporation function of the VAW and follows the ordered steps of moving and
dropping pheromones of the EVAP.
The NC-Drone (CABREIRA et al., 2018), which aims to minimize the UAV turning
maneuvers, already marks the visited cells with pheromone and move to places with
the least amount. Furthermore, another approach also explores a variation of VAW as
the evaporation function (KAPPEL et al., 2019), keeping the simplicity and dropping
pheromone values linearly as the time goes. The optimization using GA intends to im-
prove this function finding the parameter setup by which obtains the best performance.
As for genetic algorithms (GA), a survey is presented by THENGADE; DONDAL (2012)
showing the basic functionalities of a GA, such as selection, crossover, and muta-
tion and their advantages and disadvantages. Also, the authors offer a comparison
between GAs and other algorithms and heuristics for problem-solving
The work of PARADZIK et al. (2016) uses evolutionary strategies to terrain cove-
rage using multi-UAVs. A Genetic Algorithm (GA) is applied to decompose the area in
rectangles to be covered by the UAVs. This method explores two types of pheromo-
nes on these areas for the agent covering the map. Search pheromones describe the
uncertainty of a location. The higher the concentration of this type of pheromone, the
higher the need to visit that area. Path pheromones mark the specific regions visited
by the UAVs. Therefore, other agents should search for places with the least amount
of path pheromone to prevent covering the same locations.
A Genetic Algorithm was applied to optimize the coverage of vacuum cleaner robots
in (YAKOUBI; LASKRI, 2016), where a scenario is split into a grid containing obstacles
marked as unreachable cells. The agent starts with an unknown map and discovers
the area along the way. This approach ends up with a static coverage path planning
strategy to further optimize the path. However, due to the problem of having an in-
telligent observer who can memorize the route and evade the agent, the randomness
 18

of the RTSM is a desirable feature. A terrain field coverage on 3D scenarios for agri-
cultural robots is presented in (HAMEED, 2014). The authors were able to reduce by
10% to 15% the operational time using a Genetic Algorithm approach. Furthermore,
they cut the energy consumption in 6.5% by optimizing the driving angle in the 3D
configurations.
GA optimization is usually combined with other methods to generate coverage tra-
jectories (LI; WANG; SUN, 2016; TRUJILLO et al., 2016; HAYAT et al., 2017; BOUZID;
BESTAOUI; SIGUERDIDJANE, 2017). Despite the optimization, all the mentioned
works create predefined coverage paths that can be easily recognized by intelligent
targets during the mission execution. Our goal is to find the ideal configuration of pa-
rameters for different patrolling strategies using the GA. These strategies influence the
behavior of the UAV during the coverage while intending to improve the performance
metrics related to uniform distribution and interval of visits. Our unforeseeable trajecto-
ries are indirectly affected by the parameters’ optimization provoked by the GA but still
are generated online using the NC-Drone algorithm.
3 GEOMETRIC FLIGHT PATTERNS

Nowadays many of the tasks related to UAV’s , such as photogrammetry

(DI FRANCO; BUTTAZZO, 2016), search and rescue missions (NATTERO et al.,
2014),and crop field monitoring (BARRIENTOS et al., 2011), use simple geometric
flight patterns to determined the robots trajectory. These flight patterns guarantee
a path in which the agent will pass on every point of an area of interest (CHOSET,
2001). The most used pattern is the Back-and-forth (BF), Figure 1a, where the vehicle
follows a long straight line and does a 180o turn maneuver once it reaches the border
of the area. Motion Planner, the most popular flight control software (OBORNE, 2015),
adopts the BF as its standard coverage area pattern. The spiral flight pattern (SP),
Figure 1b, uses circular movements, where the agent flies towards the center of the
area of interest. Both of these patterns requires low computation power and can deal
with the problem of no communication (ÖST, 2012).

(a) Back-and-forth (b) Spiral

Figure 1 – Geometric flights patterns

Despite being the most common flight patterns, for more specific problems, like
military surveillance, these types of geometric patterns can not be used, due to its
trajectory being easily predictable. Once the pattern is known, an intelligent target can
avoid the UAV by moving in the opposite direction or hiding in a place for a while.
4 REAL-TIME SEARCH METHODS

Real-time search methods (RTSM) comes as a promising solution to resolve the

problem presented by the predictable behavior of the geometric flight patterns. In these
methods, the robot behavior its unknown to an external observer. Computational power
represents a hard constraint to any autonomous vehicle, as RTSM presents a low com-
putational cost, these methods have been proposed to solve patrolling problems with
land vehicles (KOENIG; LIU, 2001) and later expanded to aerial robots (CABREIRA
et al., 2018).
The robots’ motion is planned online ,i.e., the robots interleave between path plan-
ning and execution. RTSM usually employ local search to the nearest neighborhood
area. The scenario is discretized into a regular grid, and the field of view is restricted
to the von Neumann neighborhood, illustrated in Figure 2a. At each step of the simu-
lation, the neighborhood is adopted for search and displacement, as shown in Figure
2b, through a connected graph.

2 4 3 3 4 3

3 2 2 2
4 1 3
1 3 1

(a) Field of view (b) Connected graph

Figure 2 – The scenario is discretized into a regular grid.

The RTSM uses a digital pheromone scheme. It guides biologically-inspired robots

over the scenario by reading and writing pheromones for motion planning. To represent
 21

the pheromone marks left by the robots, a value, named u-value, is associated with
each cell of the environment. The u-value indicates the number of visits performed
by all robots on each location. A robot senses the pheromones in the neighbor cells.
Since the goal is to explore the scenario, the vehicles use the u-value to select its next
location, always choosing the cell with the lowest amount. When visiting a cell, the
robot also marks the spot by writing pheromone. Updating the pheromone value of the
current cell before moving to the next location. When two or more cells have the same
minimum-value, it randomly decides the cell be visited.
There are four main RTSM: Node Counting (NC)(PIRZADEH; SNYDER,
1990), Learning Real-Time A* (LRTA*)(KORF, 1990), Thrun’s Value Update Rule
(TVUR)(THRUN, 1992) e Wagner’s Value Update Rule (WVUR) (WAGNER; LINDEN-
BAUM; BRUCKSTEIN, 1997).
The NC algorithm (Algorithm 1) checks the neighborhood and selects the cell with
the lowest u-value. In the case of two or more cells with the same amount of u-value,
the next position is chosen randomly. Then the robot updates the u-value of the current
cell and moves to the next selected location. This process is repeated until a stop
criterion is reached, such as the number of simulation cycles.

Algorithm 1: Node Counting

Input: A set of cells (c1, ...cp)
Output: A set of metric values (m1, ...mp)
1 repeat
2 Read the u-value of the neighbor cells;
3 Select the neighbor cell with lowest u-value;
4 if there is a tie then
5 Randomly choose one of the cells;
6 end if
7 Update with +1 the u-value of the current cell;
8 Move to the selected neighbor cell;
9 until Number of cycles is reached;

The other methods, LRTA*, TVUR, WVUR, uses the same algorithm as the NC.
What differentiates each RTSM is the adopted update rule. Node Counting approach
adds one unit to the current value. LRTA* changes the value of the current cell u(s) by
the value of the next cell to be visited by the vehicle u(next(s)) added by one unit. TVUR
compares the value of the current cell u(s) with the value of the next cell to be visited
u(next(s)). The highest value is increased by one unit and replaces the original value
of the current cell u(s). WVUR presents a conditional update rule where the value of
the current cell u(s) is increased by one unit if it is not larger than the value of the next
cell u(next(s)). If the condition is not met, the value of the current cell is not updated.
Table 1 presents the update rules for the four main RTSM.
 22

Table 1 – Update rules of the main real-time search methods

Method Update rule

Node Counting u(s) := u(s) + 1
Learning Real-time A* u(s) := u(next(s)) + 1
Thrun’s Value-Update Rule u(s) := max(u(s), u(next(s)) + 1
Wagner’s Value-Update Rule if u(s) ≤ u(next(s)) then u(s) := u(s) + 1

The unpredictability of the Real-Time Search Methods comes from the fact that
when two or more cells have the same pheromone value, the robot randomly chooses
its next location, as shown in Figure 3 . This behavior leads to a massive number

3 0 3
?
? ?
0 0

1 1 1

Figure 3 – Random decision of RSTM.

of turning maneuvers (NTM). When an aerial robot performs a turning maneuver, it

must to slow down, turn, and then accelerate again. The more the turning maneuvers,
the higher the time spent during missions, and consequently, the higher the energy
consumed (MARINS et al., 2018).
5 PATROLLING STRATEGIES FOR NC-DRONE

NC-Drone (Algorithm 2) was previously presented by CABREIRA et al. (2018) and

consists of an extension of the classical Node Counting (NC). The primary goal of the
NC-Drone is to reduce excessive NTM while keeping unpredictable behavior by solving
the tie issue. NC-Drone checks the position of the cells with the same minimum-value.
In the case of one of these cells is following the sweeping direction, this cell is picked
as the next spot to be explored. In this way, the aerial vehicle maintains a straight
trajectory. If there is no such cell, a random decision is made, and the UAV performs a
turn to reach the selected side cell, an example of this process can be seen in Figure
4.

Algorithm 2: NC-Drone
Input: A set of cells (c1, ...cp)
Output: A set of metric values (m1, ...mp)
10 repeat
11 Read the u-value of the neighbor cells;
12 Select the neighbor cell with lowest u-value;
13 if there is a tie and one of the cells is aligned with the sweep direction then
14 Choose the aligned cell as the next place to be visited;
15 else
16 Randomly choose one of the cells;
17 end if
18 Update with +1 the u-value of the current cell;
19 Move to the selected neighbor cell;
20 until Number of cycles is reached;

The reading and writing pheromone scheme depends on a ground control station
(GCS) to operate in UAVs. The GCS monitors the scenario, and the robot swarm.
Besides, it tracks the least visited places and sends the corresponding motion coordi-
nates to the UAVs. However, this system always requires communication between the
base and the UAVs to perform every action. Thus, a novel matrix-based communica-
tion protocol is also proposed in (CABREIRA et al., 2018). In this multi-robot system,
each aerial robot keeps an internal matrix. This matrix is a representation of the sce-
 24

nario, where the vehicle computes the visits performed by itself. Instead of reading and
writing in the scenario, the robot follows its internal matrix. The swarm can exchange
information about the visited places and synchronize their matrices. The system is de-
signed to work in case of a lack of communication, a critical issue that may compromise
the mission. However, in our work, we focus on the centralized NC-Drone as we are
aiming to obtain the best performance from the algorithm.

3 3 3 4 3 3 4 3

2 3 2 2 2 2 4 2

3 2 3 3 2 3 3 3

Figure 4 – UAV performing the NC-Drone algorithm.

One of the main concerns of the patrolling problem is to maintain a uniform distribu-
tion of visits, both spatial and temporal. In this way, additional patrolling strategies were
also proposed to improve the performance of the NC-Drone (KAPPEL et al., 2019). All
the parameters for the patrolling strategies were obtained empirically. In this regard,
our work aims to tune these parameters through GA. The strategies consists of Evapo-
ration (Chapter 5.1) and Time-Base (Chapter 5.2). We also proposed a new strategy
called Quad (Chapter 5.3).

5.1 Evaporation Strategy

The Evaporation Strategy (ES) consists of reducing the amount of pheromone on
places not visited for a while. This strategy intends to raise the uncertainty in these
places to attract the aerial robots and balance the coverage. The ES works with two
parameters, Evaporation Time (ET) and Evaporation Factor (EF). The former defines
the interval between each evaporation, while the latter represents how much phero-
mone should be decreased.
The ES reduces the values stored in the matrix, depending on the time of the last
visit. Figure 5 illustrates the ES in three distinct cycles of simulation in a scenario 3 x 3.
To exemplify, we are considering only the first line in the matrix. The last visit in each
one of these cells occurred at cycles 50, 40, and 25, respectively.This example consi-
ders EF = 0.1 and a ET = 100, which means that the pheromone in each unvisited
location drops 0.1 at every 100 cycles.
 25

5 4 4 5 4 3.9 4.9 3.9 3.8

5 3 3 5 3 3 5 3 3
4 2 2 4 2 2 4 2 2
(a) Cycle of simulation (b) Cycle of simulation (c) Cycle of simulation
= 100 = 125 = 225

Figure 5 – Evaporation Strategy in three distinct cycles of simulation.

5.2 Time-based Strategy

Another element incorporated into the NC-Drone is the time of the last visit in a spe-
cific place. This information is useful to decide between cells with the same minimum-
value. Instead of randomly choosing one of these cells, the Time-based Strategy (TS)
selects the less recently visited location. This strategy works with a parameter called
Time Threshold (TT). This parameter is the difference between the last visit of both
side cells and is responsible for activating the TS. Considering that TT is equal to x,
LV l is the time of the last visit on the left cell, and LV r is the time of the last visit on
the right cell, TS will only be triggered if |LV l − LV r| >= x.

5.3 Quadrant Strategy

Finally, a strategy for splitting the scenario into four quadrants with the same size is
proposed. The Quadrant Strategy (QUAD) divides an area of interest and assigns the
resulting subareas to the UAVs. Each aerial robot performs the patrolling only inside
the correspondent subarea, respecting the boundaries delimited by the strategy. This
method eliminates the needing to exchange information about the map and reduces
the risk of collisions among the UAVs. Figure 6 shows the Quadrant Strategy, where
each one of the UAVs patrols its designated area represented by the different colors.
 26

Figure 6 – Quadrant strategy with four UAVs.

6 GENETIC ALGORITHMS

Genetic algorithms are a class of optimization procedures that aims to find the
optimum solution to a problem. John Holland first introduced them in 1975 (HOLLAND,
1975). They follow the Darwinian principle of reproduction and survival of the fittest,
where the most adapted individuals spread their offspring to the next generations. GA
is excellent at exploring a large and complex space to find the value close to the global
optimum. The only thing to know is the search space, and the GA will generate a
high-quality solution. Unlike traditional search heuristics, the GA does not evaluate
and improve a single solution but a set of possible solutions, a so-called population.
GAs have been proposed to solve a variety of problems, as they can work in any
search space. They can be used to design neural networks (MILLER; TODD; HEGDE,
1989), solve the travelling salesman problem (RAZALI; GERAGHTY et al., 2011), path
planning (YAKOUBI; LASKRI, 2016), and many more.
The GA process can be divided into five phases: Initialize population, Fitness func-
tion, Selection, Crossover, and Mutation.

6.1 Initialize Population

The first step of the process of the GA is to initialize the population. The population
is composed of a set of individuals, called chromosomes. Each one of the chromo-
somes is a possible solution to the problem to be solved. The individuals of the first
population can be solutions already know to the problem, with the GA working to im-
prove them. In most cases, they are randomly generated. Each individual has a set of
genes, encoded in some fashion. Usually, binary values are used. Figure 7, shows the
population, individuals, and the genes encoded as binary arrays.

6.2 Fitness Function

The fitness function is used to determine how adapted an individual is, also meaning
how well they respond to the problem. All chromosomes are evaluated and given a
fitness score. The probability of an individual being selected as a parent to reproduce
 28

A1 0 0 0 0 0 0 Gene

A2 1 1 1 1 1 1 Chromosome

A3 0 1 1 0 0 1

A4 1 0 0 0 1 1 Population

Figure 7 – Population, Chromosomes and Genes.

and generate the offspring is determined by the fitness score. The higher the score,
the higher is the probability of being selected.

6.3 Selection
Selection aims to determine which individuals will be selected and combined to
produce the next generation. Based on their fitness score, a pair of individuals is
selected to generate two new individuals. The population size is fix, the least adapted
chromosomes are eliminated, providing space for the new offspring.

6.4 Crossover
Once the individual has been selected, they need to be combined to generate a
new offspring. The solution for this is called crossover. A crossover point, Figure 8a,
is randomly chosen within the parents. The children are created by exchanging the
genes of parents among themselves until the crossover point is reached, as shown in
Figure 8b. The new offspring, Figure 8c, is then added to the population. Crossover
does not always occur. Sometimes, based on a set probability, the children can be
direct copies of the parent.

6.5 Mutation
After selection and crossover, a new population is produced. As some of the in-
dividuals can be direct copies of their parents, we need to allow a small chance of
mutation. In this process, a gene of the individual is chosen at random, and its value is
 29

A1 0 0 0 0 0 0 A1 0 0 0 0 0 0 A5 1 1 1 0 0 0

A2 1 1 1 1 1 1 A2 1 1 1 1 1 1 A6 0 0 0 1 1 1

(a) Crossover Point. (b) Exchanging genes. (c) new offspring.

Figure 8 – Crossover Process.

change. Figure 9, shows the mutation selecting the third gene and changing its value
to zero. The mutation is relatively simple; however, it is vital to ensure genetic diversity
within the population.

Before Mutation
A5 1 1 1 0 0 0

After Mutation
A5 1 1 0 0 0 0

Figure 9 – Mutation Process.

The genetic algorithm process, Figure 10, repeats until the population has conver-
ged, which means that the new offspring is not significantly different from the previ-
ous generation, or the process repeats for a number of generations. The number of
generations varies from problem to problem. More complex problems require more
generations until the GA can adapt and provide a suitable response.
 30

Initialize Population

Fitness Calculation

Terminate? Results

Selection

Crossover

Mutation

Figure 10 – Genetic algorithm process.

7 OUR GENETIC OPTIMIZATION

The patrolling strategies improved the performance of the NC-Drone algorithm ac-
cording to the experimental results obtained in the work of KAPPEL et al. (2019).
However, the values of the parameters of such strategies were empirically set up. Thus,
a Genetic Algorithm (GA) optimization is presented to define the values of the set of
parameters for the patrolling strategies. Initially, the GA randomly generates a set of
candidate solutions called population. Each solution represents an individual of the
population, and it is encoded as a chromosome. In our GA optimization approach, th-
ree parameters encoded in a binary array compose the chromosomes, the Evaporation
Time (ET), Evaporation Factor (EF), and Time Threshold (TT). The values for the ET
and TT vary from 1 to 1000, while the values for the EF vary from 0 to 1.
A fitness function evaluates all the individuals, measuring the quality of the candi-
date solutions. Our fitness function is based on the Quadratic Mean of Intervals (QMI)
(SAMPAIO; SOUSA; ROCHA, 2016). The QMI corresponds to frequency-regularity
equilibrium, a patrolling requirement that combines the frequency of visits and even-
ness of intervals between these visits. This performance metric balances values of
average, maximum, and standard deviation of intervals. Equation (1) presents the GA
fitness function based on the QMI performance metric:
  1/2
visits(x)+1
1 X X
QM I =   (ixj )2  (1)
Nintervals x∈cells j=1

where ixj represents the intervals between the visits j to the cells x and Nintervals cor-
responds to the total number of intervals. The lower the QMI, the more uniform is the
patrolling path.
The next step is to select the solutions to reproduce and spread their offspring for
the next generation. The proposed GA adopts the roulette-wheel selection, where the
fittest candidates have a higher chance of being picked for reproduction. This probabi-
lity is proportional to the amount by which its fitness is higher than its competitor’s fit-
ness. Then, the selected individuals, also known as parents, are submitted to a uniform
crossover to reproduce and generate a new set of possible solutions. In this process,
 32

the parents can contribute evenly with their chromosomes for the children, or children
could be direct copies of their parents, depending on the probability of crossover. Our
GA applies a uniform crossover with a chance of 80%.

Table 2 – Summary of the Genetic Algorithm parameters.

PARAMETER INFORMATION
Population 80 individuals
Selection Roulette-wheel
Crossover Uniform with 80%
Mutation Rate of 10%
Generation 200

Finally, the new generation goes through a mutation process.The individuals have
a 10% chance to suffer mutation. A process called “big mutation” is also employed
to overcome premature or local-best solution. In this process, the GA multiplies the
fitness of the best individual by α - in our case, α = 0.6. If the result is lower than the
mean of the population fitness, the probability of mutation is five times greater.
All new individuals replace the original population, and the GA restarts a new cycle
of execution. The algorithm runs for 200 generations and selects the best individual as
the ideal configuration of parameters for the patrolling strategies. Table 2 presents the
summary of GA. This particular configuration was obtained empirically. Different rates
of crossover and mutation were tested, as well as a variable number of individuals and
generations.
8 SIMULATION ENVIRONMENT

The test platform, presented in Figure 11, was built using the PyGame library
(SHINNERS, 2019). We based our simulation environment on the work of CABREIRA
et al. (2018). Our platform has two types of agents: Patches and Drones. We have a
grid-discretized scenario, where each one of the grid cells is modeled as patches. The

Figure 11 – Test platform built with PyGame.

patches have a fixed position on the scenario. They keep information related to the cell,
like the number of visits, the interval between visits, the time of the last visit, and the
amount of pheromones. The Drones, representing the UAVs, move over the patches
on the scenario using the NC-Drone algorithm. They can read and write information
on the patches.
 34

The platform allows us to combine different strategies for NC-Drone easily. We can
also experiment with different parameter values for said strategies. The number of
UAVs can be modified, as well as the number of simulation cycles. The simulation can
be performed step by step on-screen to observe and analyze the agents’ behavior or
off-screen to improve simulation performance.
All the patches start with the number of visits equals to zero. The UAVs are released
on the scenario with an interval of one cycle between each agent. We run m rounds of
n cycles for each experiment. We then compute the Mean and Standard deviation of
the evaluation metrics.
For the GA, we used a python framework for genetic algorithm called Gaft 1 . This
framework provides built-in genetic operators for target optimization and plugin inter-
faces for users to define their own genetic operators and on-the-fly analysis for al-
gorithm testing. First, we need to define the population. Gaft uses tuples to repre-
sent the chromosomes. We select the range and the population size. Then based
on these parameters, Gaft will generate the population. For selection, we have four
built-in operators: roulette wheel selection, exponential ranking selection, linear ran-
king selection, and tournament selection. After the selection operator is defined, we
need to select the crossover and mutation operators. Gaft supports uniform crosso-
ver and flip bit mutation. As we select these operators, we also need to define their
probabilities. We then create an engine that will run the GA. The most important part
is to defined our fitness function. The GA selects the best individual based on the
fitness score. If we want to minimize the fitness function, we can add a minimiza-
tion. In this way, the best candidates are the ones with the lowest score. At last,
we defined the number of generations. After the GA is completed, Gaft will gene-
rate a text file containing the best individual and its fitness value for each generation.
More details about our platform and the GA framework can be found on GitHub in
https://https://github.com/patrickguima/NC-Drone-Simulator.

1
Gaft Documentation, https://gaft.readthedocs.io/en/latest/
9 EXPERIMENTAL RESULTS

The scenario is a 50 × 50 grid containing four UAVs modeled as robot-agents. We

present four types of scenarios, one free of obstacles, and the other three containing
different compositions of no-fly zones (NFZ). The aerial robots can navigate through
the scenario, interact with each other, and avoid the NFZ. The starting position of the
UAVs is at the lower-left corner of the scenario.
We run one simulation with 10.000 cycles for each GA individual using the parame-
ters encoded in its chromosome to calculate the fitness. In this way, we run a total of
80 simulations × 200 generations. After the GA concluded, we run 30 simulations with
the best solution, i.e., the ideal configuration of parameters. Table 3 shows the parame-
ters of the experiment. Results presented in the tables are the mean (M) and standard
deviation (SD) of these 30 simulations. We consider three performance metrics from
the work of CABREIRA et al. (2018), Quadratic Mean of the Intervals (QMI), Standard
Deviation of the Frequencies (SDF), and Number of Turning Maneuvers (NTM).

Table 3 – Summary of the experiments parameters.

PARAMETER INFORMATION
Grid Size 50 x 50
UAVs 4
Cycles 10.000
Simulations 30

The QMI metric was described in Chapter 7. The SDF computes the distribution
equality of visits over the cells. The lower the value, the more uniform the frequency is.
Equation (2) illustrates the SDF performance metric:
" #1/2
1 X
SDF = ( f req(x) − Favg )2 (2)
|cells| x∈cells

where f req(x) corresponds to the frequency of visits on each cell x, Favg is the average
frequency of all cells and cells is the total number of cells present in the scenario.
 36

NTM is usually correlated with the energy spent to perform a coverage path. Every
time a UAV needs to execute a turning maneuver, it must decelerate, rotate, and ac-
celerate again. These maneuvers increase covering time and, consequently, the ne-
cessary energy to carry out a mission. Thus, the lower the NTM, the lower the energy
consumption is. Equation (3) shows the NTM performance metric:
 
turns(u)
X X
NT M =  tuk  (3)
u∈uavs k=1

where tuk denotes the number of turning maneuvers performed by each UAV u.

9.1 Single Strategies

The first experiment consists of single strategies in scenario free of obstacles. The
approaches include the original NC-Drone, the NC-Drone using Evaporation Strategy
(ES), Time-based Strategy (TS), and Quadrant Strategy (QUAD), and the optimized
versions using the parameter configuration obtained through the GA. The empirical
and optimized parameters are presented in Table 4. The empirical parameters of the
ES are ET = 100 and EF = 0.1, while the GA optimization indicates ET = 1 and
EF = 0.32.
Table 4 – Empirical and optimized parameters for single strategies.

PARAMETERS
APPROACHES
ET EF TT
ES 100 0.1 x
ES GA 1 0.32 x
TS x x x
TS GA x x 4
QUAD x x x

The results for the Time-based Strategy (TS) are also presented in Table 5. The
original TS has not a Time Threshold (TT) parameter. The TS-GA introduces this
parameter with T T = 4 and presents a QMI improvement of around 3% compared to
the original NC-Drone. In the TS-GA, the UAV selects the less recently visited cell
if there is a difference of at least four cycles of simulation between the last visit in
the two cells with the same value. Otherwise, the UAV randomly chooses one of the
cells. Despite the parameter optimization, the Student’s T-Test states that the difference
between the TS and TS-GA is considered to be not statistically significant.
Finally, the Quadrant Strategy (QUAD) presents a QMI improvement of around 2%
and reduces in 59% the standard deviation compared to the original NC-Drone. Consi-
 37

dering all strategies, the ES-GA obtains the best results in the QMI performance metric,
followed by the TS-GA. Despite the smaller improvement in the QMI, the TS-GA stra-
tegy maintains the SDF and NTM with similar results compared to the NC-Drone, while
ES-GA increases the SDF by 62% and NTM by 121%. The experimental results of the
patrolling strategies in a scenario free of obstacles are presented in Table 5.

Table 5 – Patrolling strategies with parameter optimization.

QMI SDF NTM

APPROACHES
M SD M SD M SD
NC-Drone 750.13 11.58 0.75 0.09 5.6k 366.51
ES 707.09 6.64 0.80 0.10 10.7k 468.34
ES-GA 678.82 1.96 1.22 0.13 12.4k 133.04
TS 730.59 8.26 0.77 0.08 5.2k 391.41
TS-GA 727.46 7.70 0.74 0.07 5.6k 354.36
QUAD 736.49 4.77 0.91 0.04 6.2k 212.86

9.2 Combination of Strategies

We experimented with a series of combination of patrolling strategies. The empiri-
cal parameters and the parameters obtain trough the GA for the combination strategies
are presented in Table 6. The ES-TS combines the Evaporation and Time-based stra-
tegies with the following empirical parameters, ET = 100 and EF = 0.1, while the
ES-TS GA sets ET = 1, EF = 0.32, and T T = 4. The ES-TS GA improves the
QMI around 3.5% and 9.5% compared to the ES-TS and the NC-Drone, respectively.
QUAD-TS combines Quadrant and Time-based strategies. The optimized GA version

Table 6 – Empirical and optimized parameters for combination strategies.

PARAMETERS
APPROACHES
ET EF TT
ES-TS 100 0.1 x
ES-TS GA 1 0.32 4
QUAD-TS x x x
QUAD-TS GA x x 4
QUAD-ES 100 0.1 x
QUAD-ES GA 1 0.83 x
QUAD-ES-TS 100 0.1 x
QUAD-ES-TS GA 5 0.32 3

sets T T = 4 and presents a QMI improvement of around 4.3% compared to the NC-
 38

Drone. According to the Student’s T-test, the difference between the QUAD-TS and
QUAD-TS GA is considered to be not statistically significant.
QUAD-ES mixes Quadrant and Evaporation strategies with the following empirical
parameters ET = 100 and EF = 0.1. The GA parameter optimization sets ET = 1
and EF = 0.83 for the QUAD-ES GA. This strategy presents the best results among all
combinations — the percentage of QMI improvement is around 10.7% over the QUAD-
ES and 18.3% over the NC-Drone.
The final combination includes all previously presented strategies, Quadrant, Time-
based, and Evaporation. The empirical QUAD-ES-TS adopts the same configuration
with ET = 100 and EF = 0.1, while the GA version sets ET = 5, EF = 0.32, and
T T = 3. The QUAD-ES-TS GA shows an improvement of 7.7% and 15.8% compared
to the QUAD-ES-TS and NC-Drone without the parameter optimization.
Three strategies are marked with * in Table 7, QUAD-TS, QUAD-ES GA, and QUAD-
ES-TS. These strategies have zero standard deviation in all performance metrics,
which means that the UAV moves through the scenario using the same trajectory in
all simulations. Figure 12 illustrates the patrolling pattern generated by the QUAD-ES
GA strategy, the best solution among the combination of strategies.

20

18

16

14

12

10

0
0 2 4 6 8 10 12 14 16 18 22

Figure 12 – Trajectory generated by the QUAD-ES GA strategy.

The complete results for the series of combination of patrolling strategies are pre-
sented in Table 7.

9.3 Scenarios with No-Fly Zones

Additional experiments are performed in three different scenarios. These scenarios
consist of a 50×50 grid with 10% of no-fly zones (NFZ), as illustrated in Figure 13. Table
 39

Table 7 – Combination of patrolling strategies using GA.

QMI
SDF NTM
APPROACHES
M SD M SD M SD
NC-Drone 750.13 11.58 0.75 0.09 5.6k 366.51
ES-TS 703.89 0.40 0.70 0.05 10.7k 3.55
ES-TS GA 678.74 1.72 1.23 0.13 12.4k 117.66
QUAD-TS* 718.23 0.00 0.89 0.00 6.8k 0.00
QUAD-TS GA 717.60 5.71 0.89 0.04 6.2k 202.04
QUAD-ES 685.97 3.23 0.93 0.04 11.6k 256.49
QUAD-ES GA* 612.64 0.00 0.81 0.00 4.5K 0.00
QUAD-ES-TS* 684.00 0.00 0.92 0.00 11.7k 0.00
QUAD-ES-TS GA 631.32 8.09 0.90 0.05 7.0k 1.0k

8 presents the resulting parameter optimization in the three scenarios containing NFZ.
These values are the ideal configuration that optimizes the QMI results of each patrol-
ling strategy presented in Table 9. The ET parameter remains almost constant, and
the interval between each evaporation varies only between 1 and 2 cycles of simulation
depending on the scenario. The ES parameter varies between 0.37 and 0.90 depending
on the scenario and the adopted strategy. Finally, the TT parameter changes from 3 in
the TS to 898 in the ES-TS.

(a) (b) (c)

Figure 13 – Three different scenarios consisting of a 50 x 50 grid containing 10% of

no-fly zones marked in red.

The QMI results for all strategies, and the respective combinations are presented
in Table 9. The SDF and NTM results are presented in Table 10 and Table 11, Res-
pectively. As one can see, they get worse as the QMI improves. Once again, the
approaches whose parameters were optimized through the GA present better results
than the strategies with the empirical values, with the exception of the TS GA and
QUAD-TS GA. The ES GA shows a statistical difference compared to the empirical ES
 40

Table 8 – Optimized parameters in scenarios with NFZ.

SCENARIO 1 SCENARIO 2 SCENARIO 3

APPROACHES
ET EF TT ET EF TT ET EF TT
ES GA 1 0.37 x 1 0.62 x 1 0.64 x
TS GA x x 3 x x 3 x x 3
ES-TS GA 47 0.69 190 15 0.60 705 2 0.80 898
QUAD-ES GA 1 0.83 x 2 0.84 x 1 0.70 x
QUAD-TS GA x x 1 x x 96 x x 5
QUAD-ES-TS GA 1 0.83 881 2 0.47 315 2 0.90 446

and improves the QMI of the NC-Drone in around 8% in the three scenarios. Consi-
dering the TS and the QUAD-TS, there is no statistically significant difference between
the empirical approaches and the optimized ones by the GA. The ES-TS GA presents
a slight improvement compared to the original ES-TS.
The QUAD-ES GA and QUAD-ES-TS GA strategies obtain the best results among
all combinations. Both approaches present no statistical difference in the first two
scenarios. However, the QUAD-ES GA overcomes QUAD-ES-TS GA in scenario 3.
The percentage of improvement of the QUAD-ES GA compared to the NC-Drone is
around 10% in scenarios 1 and 2 and almost 12% in scenario 3.
Table 9 – QMI results for the combination of patrolling strategies using GA in
scenarios with NFZ.

SCENARIO 1 SCENARIO 2 SCENARIO 3

APPROACHES
M SD M SD M SD
NC-Drone 706.81 7.78 701.66 9.19 705.70 9.68
ES 658.42 7.28 657.21 7.12 658.09 6.42
ES GA 648.34 6.48 644.54 4.48 643.50 6.50
TS 681.08 6.40 681.44 7.45 680.50 6.46
TS GA 681.97 8.21 683.08 5.48 683.76 8.96
ES-TS 654.73 6.67 654.26 5.45 655.91 8.24
ES-TS GA 650.35 6.19 651.52 5.64 647.72 6.45
QUAD 696.72 6.40 690.32 5.10 685.77 5.18
QUAD-ES 650.51 4.13 645.62 3.72 636.98 3.71
QUAD-ES GA 638.30 3.68 630.50 3.93 623.26 3.54
QUAD-TS 677.39 4.76 670.64 5.08 663.78 4.74
QUAD-TS GA 679.61 5.11 673.68 5.40 665.10 5.05
QUAD-ES-TS 647.65 3.25 641.32 2.98 635.56 3.12
QUAD-ES-TS GA 638.50 4.24 631.82 4.46 627.89 3.00
 41

Table 10 – SDF results for the combination of patrolling strategies using GA in

scenarios with NFZ.

SCENARIO 1 SCENARIO 2 SCENARIO 3

APPROACHES
M SD M SD M SD
NC-Drone 0.80 0.08 0.82 0.13 0.82 0.10
ES 0.90 0.09 0.86 0.09 0.86 0.09
ES GA 1.28 0.12 1.76 0.28 1.80 0.24
TS 0.85 0.08 0.80 0.09 0.82 0.09
TS GA 0.79 0.08 0.80 0.11 0.82 0.13
ES-TS 0.88 0.10 0.88 0.13 0.87 0.10
ES-TS GA 1.01 0.13 1.06 0.12 1.31 0.20
QUAD 1.54 0.06 1.52 0.04 1.23 0.03
QUAD-ES 1.56 0.03 1.56 0.06 1.26 0.04
QUAD-ES GA 2.07 0.13 1.82 0.08 1.81 0.07
QUAD-TS 1.52 0.03 1.52 0.03 1.23 0.03
QUAD-TS GA 1.55 0.04 1.52 0.04 1.24 0.04
QUAD-ES-TS 1.55 0.03 1.51 0.03 1.24 0.03
QUAD-ES-TS GA 2.06 0.08 1.72 0.07 1.53 0.06

Table 11 – NTM results for the combination of patrolling strategies using GA in

scenarios with NFZ.

SCENARIO 1 SCENARIO 2 SCENARIO 3

APPROACHES
M SD M SD M SD
NC-Drone 7.8k 182.40 7.3k 268.61 7.6k 263.39
ES 13.3k 207.44 12k 377.93 12.8k 256.92
ES GA 17.7k 278.35 15.8k 348.82 16.9k 381.94
TS 7.7k 174.95 7.1k 210.51 7.4k 245.62
TS GA 7.5k 231.20 7.2k 238.29 7.5k 247.93
ES-TS 13.3k 212.49 12.1k 318.97 12.7k 291.56
ES-TS GA 14.2 240.58 14.7k 300.73 17.4k 316.10
QUAD 8.4k 161.06 7.7k 160.91 8.1k 138.16
QUAD-ES 13.3k 200.81 12.4k 205.05 13k 220.39
QUAD-ES GA 17k 168.84 16.1k 425.41 16.5k 304.44
QUAD-TS 8.3k 117.59 7.7k 109.64 7.9k 158.89
QUAD-TS GA 8.3k 148.30 7.7k 132.99 8k 123.74
QUAD-ES-TS 13.3k 193.04 12.2k 278.1 13.1k 191.85
QUAD-ES-TS GA 17k 188.30 16.4k 363.54 17.2k 250.86
10 CONCLUSION

This work proposes a parameter optimization for patrolling strategies based on the
NC-Drone algorithm using GA. These strategies consist of Evaporation, Time-based,
and Quadrant, and the parameters are the Evaporation Time (ET), Evaporation Factor
(EF), and Time Threshold (TT). The optimization model explores the QMI metric as
a fitness function to tune the parameters. We aim to minimize this metric to obtain a
uniform distribution of visits.
We present experimental results of single and combined strategies over scenarios
free of obstacles and containing no-fly zones. We compare approaches with empiri-
cal parameters against solutions with values optimized by the GA. The QUAD-ES GA
emerges as the most promising solution for the patrolling problem. This strategy over-
comes all combinations in all different scenarios. As future work, we intend to further
explore the patrolling strategies in specific applications, such as marine traffic, and
implement the solutions in UAVs to perform real flights.
 43

REFERENCES

ANDERSEN, H. L. Path Planning for Search and Rescue Mission using Multicop-
ters. 2014. Dissertação (Mestrado em Ciência da Computação) — Institutt for teknisk
kybernetikk, Norway.

BARRIENTOS, A. et al. Aerial remote sensing in agriculture: A practical approach

to area coverage and path planning for fleets of mini aerial robots. Journal of Field
Robotics, [S.l.], v.28, n.5, p.667–689, 2011.

BOUZID, Y.; BESTAOUI, Y.; SIGUERDIDJANE, H. Quadrotor-UAV optimal coverage

path planning in cluttered environment with a limited onboard energy. In: INTELLIGENT
ROBOTS AND SYSTEMS (IROS), 2017 IEEE/RSJ INTERNATIONAL CONFERENCE
ON, 2017. Anais. . . [S.l.: s.n.], 2017. p.979–984.

CABREIRA, T. M.; KAPPEL, K. S.; FERREIRA, P. R.; BRISOLARA, L. B. de. An

Energy-Aware Real-Time Search Approach for Cooperative Patrolling Missions with
Multi-UAVs. In: LATIN AMERICAN ROBOTIC SYMPOSIUM, 2018 BRAZILIAN SYM-
POSIUM ON ROBOTICS (SBR) AND 2018 WORKSHOP ON ROBOTICS IN EDUCA-
TION (WRE), 2018., 2018. Anais. . . [S.l.: s.n.], 2018. p.254–259.

CHOSET, H. Coverage for robotics–A survey of recent results. Annals of mathema-

tics and artificial intelligence, [S.l.], v.31, n.1-4, p.113–126, 2001.

CHU, H. N. et al. Swarm approaches for the patrolling problem, information propa-
gation vs. pheromone evaporation. In: TOOLS WITH ARTIFICIAL INTELLIGENCE,
2007. ICTAI 2007. 19TH IEEE INTERNATIONAL CONFERENCE ON, 2007. Anais. . .
[S.l.: s.n.], 2007. v.1, p.442–449.

DI FRANCO, C.; BUTTAZZO, G. Coverage path planning for UAVs photogrammetry

with energy and resolution constraints. Journal of Intelligent & Robotic Systems,
[S.l.], v.83, n.3-4, p.445–462, 2016.

GLAD, A.; SIMONIN, O.; BUFFET, O.; CHARPILLET, F. Theoretical study of ant-
based algorithms for multi-agent patrolling. In: EUROPEAN CONFERENCE ON AR-
 44

TIFICIAL INTELLIGENCE INCLUDING PRESTIGIOUS APPLICATIONS OF INTEL-

LIGENT SYSTEMS (PAIS 2008)-ECAI 2008, 18., 2008. Anais. . . [S.l.: s.n.], 2008.
p.626–630.

HAMEED, I. A. Intelligent coverage path planning for agricultural robots and autono-
mous machines on three-dimensional terrain. Journal of Intelligent & Robotic Sys-
tems, [S.l.], v.74, n.3-4, p.965–983, 2014.

HAYAT, S.; YANMAZ, E.; BROWN, T. X.; BETTSTETTER, C. Multi-objective UAV path
planning for search and rescue. In: ROBOTICS AND AUTOMATION (ICRA), 2017
IEEE INTERNATIONAL CONFERENCE ON, 2017. Anais. . . [S.l.: s.n.], 2017. p.5569–
5574.

HEFFERAN, B.; CLIFF, O. M.; FITCH, R. Adversarial Patrolling with Reactive Point
Processes. In: ARAA AUSTRALASIAN CONFERENCE ON ROBOTICS AND AUTO-
MATION (ARAA, 2016), 2016. Proceedings. . . [S.l.: s.n.], 2016. p.39–46.

HOLLAND, J. H. Adaptation in natural and artificial systems: an introductory analy-

sis with applications to biology, control, and artificial intelligence. [S.l.]: U Michigan
Press, 1975.

KAPPEL, K. S. et al. Strategies for Patrolling Missions with Multiple UAVs. Journal of
Intelligent & Robotic Systems, [S.l.], p.1–17, 2019.

KOENIG, S.; LIU, Y. Terrain Coverage with Ant Robots: A Simulation Study. In:
FIFTH INTERNATIONAL CONFERENCE ON AUTONOMOUS AGENTS, 2001. Pro-
ceedings. . . [S.l.: s.n.], 2001. p.600–607.

KORF, R. E. Real-time heuristic search. Artificial intelligence, [S.l.], v.42, n.2-3,

p.189–211, 1990.

KUIPER, E.; NADJM-TEHRANI, S. Mobility models for UAV group reconnaissance ap-
plications. In: INTERNATIONAL CONFERENCE ON WIRELESS AND MOBILE COM-
MUNICATIONS (ICWMC’06), 2006., 2006. Anais. . . [S.l.: s.n.], 2006. p.33–33.

LI, D.; WANG, X.; SUN, T. Energy-optimal coverage path planning on topographic map
for environment survey with unmanned aerial vehicles. Electronics Letters, [S.l.], v.52,
n.9, p.699–701, 2016.

MARINS, J. L.; CABREIRA, T. M.; KAPPEL, K. S.; JÚNIOR, P. R. F. A Closed-Form

Energy Model for Multi-rotors Based on the Dynamic of the Movement. In: VIII BRA-
ZILIAN SYMPOSIUM ON COMPUTING SYSTEMS ENGINEERING (SBESC), 2018.,
2018. Anais. . . [S.l.: s.n.], 2018. p.256–261.
 45

MILLER, G. F.; TODD, P. M.; HEGDE, S. U. Designing Neural Networks using Genetic
Algorithms. In: ICGA, 1989. Anais. . . [S.l.: s.n.], 1989. v.89, p.379–384.

NATTERO, C.; RECCHIUTO, C. T.; SGORBISSA, A.; WANDERLINGH, F. Coverage

Algorithms for Search And Rescue with UAV Drones. In: ARTIFICIAL INTELLIGENCE,
WORKSHOP OF THE XIII AI*IA SYMPOSIUM ON, 2014. Anais. . . [S.l.: s.n.], 2014.

OBORNE, M. Mission planner-ground station. 3D Robotics, [S.l.], 2015.

ÖST, G. Search path generation with UAV applications using approximate convex
decomposition.

PARADZIK, M. et al. Multi-agent search strategy based on digital pheromones for

UAVs. In: SIGNAL PROCESSING AND COMMUNICATION APPLICATION CONFE-
RENCE (SIU), 2016 24TH, 2016. Anais. . . [S.l.: s.n.], 2016. p.233–236.

PIRZADEH, A.; SNYDER, W. A unified solution to coverage and search in explored and
unexplored terrains using indirect control. In: IEEE INTERNATIONAL CONFERENCE
ON ROBOTICS AND AUTOMATION, 1990. Proceedings. . . [S.l.: s.n.], 1990. p.2113–
2119.

PORTUGAL, D.; ROCHA, R. A survey on multi-robot patrolling algorithms. In: DOCTO-

RAL CONFERENCE ON COMPUTING, ELECTRICAL AND INDUSTRIAL SYSTEMS,
2011. Anais. . . [S.l.: s.n.], 2011. p.139–146.

RAZALI, N. M.; GERAGHTY, J. et al. Genetic algorithm performance with different

selection strategies in solving TSP. In: OF THE WORLD CONGRESS ON ENGINEE-
RING, 2011. Proceedings. . . [S.l.: s.n.], 2011. v.2, n.1, p.1–6.

ROSALIE, M.; DANOY, G.; CHAUMETTE, S.; BOUVRY, P. From random process to
chaotic behavior in swarms of UAVs. In: ACM SYMPOSIUM ON DEVELOPMENT
AND ANALYSIS OF INTELLIGENT VEHICULAR NETWORKS AND APPLICATIONS,
6., 2016. Proceedings. . . [S.l.: s.n.], 2016. p.9–15.

SAMPAIO, P.; SOUSA, R.; ROCHA, A. New Patrolling Strategies with Short-Range
Perception. In: XIII LATIN AMERICAN ROBOTICS SYMPOSIUM AND IV BRAZILIAN
ROBOTICS SYMPOSIUM, 2016., 2016. Proceedings. . . [S.l.: s.n.], 2016.

SHINNERS, P. PyGame. (accessed on 22 April 2019), Available online: http://

pygame.org/.

THENGADE, A.; DONDAL, R. Genetic algorithm-survey paper. In: MPGI NATIONAL

MULTI CONFERENCE, 2012. Anais. . . [S.l.: s.n.], 2012. p.7–8.

THRUN, S. B. Efficient exploration in reinforcement learning. , [S.l.], 1992.

 46

TRUJILLO, M. M. et al. Optimized flight path for 3D mapping of an area with structures
using a multirotor. In: UNMANNED AIRCRAFT SYSTEMS (ICUAS), 2016 INTERNA-
TIONAL CONFERENCE ON, 2016. Anais. . . [S.l.: s.n.], 2016. p.905–910.

WAGNER, I. A.; LINDENBAUM, M.; BRUCKSTEIN, A. M. Distributed covering by ant-

robots using evaporating traces. IEEE Transactions on Robotics and Automation,
[S.l.], v.15, n.5, p.918–933, 1999.

WAGNER, I.; LINDENBAUM, M.; BRUCKSTEIN, A. On-line graph searching by a

smell-oriented vertex process. In: AAAI WORKSHOP ON ON-LINE SEARCH, 1997.
Proceedings. . . [S.l.: s.n.], 1997. p.122–125.

YAKOUBI, M. A.; LASKRI, M. T. The path planning of cleaner robot for coverage region
using genetic algorithms. Journal of innovation in digital ecosystems, [S.l.], v.3, n.1,
p.37–43, 2016.