Infrastructure Computer Vision

Infrastructure Computer Vision

Edited by

Ioannis Brilakis

Department of Engineering, University of Cambridge, Cambridge, United Kingdom

Carl Haas

Department of Civil and Environmental Engineering, University of Waterloo, Ontario, Canada

Table of Contents

Cover image

Title page

Copyright

Dedication

Contributors

Foreword

Acknowledgments

Chapter 1. Introduction

1.1. Executive summary

1.2. The current business environment for ICV

1.3. Documented inefficiencies of traditional approaches

1.4. ICV: new tools and new processes

1.5. Synergies of ICV and other technologies

1.6. What is not ICV technology?

1.7. What challenges can be expected?

Chapter 2. Surveying, Geomatics, and 3D Reconstruction

2.1. Executive summary

2.2. Introduction

2.3. The evolution of surveying infrastructure: from theodolites to mobile phone cameras

2.4. Surveying user requirements

2.5. Overview of the major ICV-based surveying tools

2.6. Terrestrial laser scanning and point cloud registration

2.7. Photogrammetric and videogrammetric 3D reconstruction

2.8. Structured lighting scanners

2.9. Integration with geospatial surveying equipment and other sensors

2.10. Conclusion

2.11. Discussion questions

Chapter 3. Scene understanding and model generation

3.1. Executive summary

3.2. Introduction

3.3. Overview of the major as-is modeling platforms

3.4. Data preparation

3.5. Surface, region, and object detection and segmentation

3.6. 3D object fitting

3.7. Semantic relationship learning

3.8. Using as-designed BIM as prior knowledge

3.9. Exploring the potential of deep learning

3.10. ICV data management

3.11. Conclusion

3.12. Discussion questions

Chapter 4. Use Cases for Owners and Maintainers

4.1. Executive summary

4.2. The case for ICV from the owners' perspective

4.3. Progress so far and challenges related to modeling existing infrastructure: a single asset scenario

4.4. From an asset to a portfolio of assets: scaling-up as-is modeling

4.5. ICV for retrofitting

4.6. ICV for intelligent transport systems

4.7. Damage detection for maintenance inspection

4.8. Generation of 3D building thermal profiles for energy modeling

4.9. Guidelines and issues for owners to consider when adopting ICV

4.10. Discussion questions

Chapter 5. Use Cases for Architects and Engineers

5.1. Executive summary

5.2. The case for ICV from the designers' perspective

5.3. Learning design, construction, and operation knowledge from models of existing structures

5.4. Establishing existing site conditions: rapid land surveying

5.5. Establishing, modeling, and using tie-in points for upgrades and add-ons

5.6. Postdisaster condition assessment

5.7. BIM and mixed reality for design visualization (cont: Y. Turkan)

5.8. BIM and mixed reality for construction inspection

5.9. As-designed and as-built BIM comparison (cont: Y. Turkan)

5.10. Chapter 5 discussion questions

Chapter 6. Use Cases for Contractors

6.1. Executive summary

6.2. The case for ICV from contractors' perspectives

6.3. Simulation verification for construction operations planning

6.4. Materials management

6.5. Safety and health monitoring

6.6. Defect detection

6.7. Progress monitoring

6.8. Implications for contractual and organizational changes

6.9. Chapter 6 discussion questions

Chapter 7. Use Cases for Subcontractors and Fabricators

7.1. Executive summary

7.2. The case for ICV from the subcontractors' perspective

7.3. Production monitoring

7.4. Fabrication control

7.5. As-built surveying, modeling, alignment, and fitting for off-site fabrication

7.6. Replacing the tape measure

7.7. Discussions and future directions

Chapter 8. The Future

8.1. Executive summary

8.2. Introduction

8.3. The development of ICV up to 2019

8.4. Current trends

8.5. Vision 2030

8.6. Drivers of change and ICV impacts up to 2030

Author Index

Subject Index

Copyright

Butterworth-Heinemann is an imprint of Elsevier

The Boulevard, Langford Lane, Kidlington, Oxford OX5 1GB, United Kingdom

50 Hampshire Street, 5th Floor, Cambridge, MA 02139, United States

Copyright © 2020 Elsevier Inc. All rights reserved.

No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopying, recording, or any information storage and retrieval system, without permission in writing from the publisher. Details on how to seek permission, further information about the Publisher’s permissions policies and our arrangements with organizations such as the Copyright Clearance Center and the Copyright Licensing Agency, can be found at our website: www.elsevier.com/permissions.

This book and the individual contributions contained in it are protected under copyright by the Publisher (other than as may be noted herein).

Notices

Knowledge and best practice in this field are constantly changing. As new research and experience broaden our understanding, changes in research methods, professional practices, or medical treatment may become necessary.

Practitioners and researchers must always rely on their own experience and knowledge in evaluating and using any information, methods, compounds, or experiments described herein. In using such information or methods they should be mindful of their own safety and the safety of others, including parties for whom they have a professional responsibility.

To the fullest extent of the law, neither the Publisher nor the authors, contributors, or editors, assume any liability for any injury and/or damage to persons or property as a matter of products liability, negligence or otherwise, or from any use or operation of any methods, products, instructions, or ideas contained in the material herein.

Library of Congress Cataloging-in-Publication Data

A catalog record for this book is available from the Library of Congress

British Library Cataloguing-in-Publication Data

A catalogue record for this book is available from the British Library

ISBN: 978-0-12-815503-5

For information on all Butterworth-Heinemann publications visit our website at https://www.elsevier.com/books-and-journals

Publisher: Matthew Deans

Acquisition Editor: Matthew Deans

Editorial Project Manager: Peter Llewellyn

Production Project Manager: Anitha Sivaraj

Cover Designer: Matthew Limbert

Typeset by TNQ Technologies

Dedication

This book is dedicated to our students, our families, and our sabbatical hosts (Professors Borrmann, Adey, and Hartmann) who made it possible for us to find sufficient time to produce this book.

Contributors

Eva Agapaki,     Department of Engineering, University of Cambridge, United Kingdom

Burcu Akinci,     Civil and Environmental Engineering, Carnegie Mellon University, Pittsburgh, PA, United States

Ioannis Brilakis,     Department of Engineering, University of Cambridge, Cambridge, United Kingdom

Jiawei Chen,     School of Sustainable Engineering and The Built Environment, Arizona State University, Tempe, AZ, United States

Carl Haas,     Department of Civil and Environmental Engineering, University of Waterloo, Ontario, Canada

Varun Kasireddy,     Civil and Environmental Engineering, Carnegie Mellon University, Pittsburgh, PA, United States

Hyoungkwan Kim,     Civil and Environmental Engineering, Yonsei University, Seoul, Korea

Minkoo Kim,     Building and Real Estate, The Hong Kong Polytechnic University, Kowloon, Hong Kong

Marianna Kopsida,     Department of Engineering, University of Cambridge, Cambridge, United Kingdom

Mohammad Nahangi,     Advanced Building Systems, KATERRA, Toronto, Canada

Stephanie German Paal,     Zachry Department of Civil and Environmental Engineering, Texas A&M University, United States

Pingbo Tang,     School of Sustainable Engineering and The Built Environment, Arizona State University, Tempe, AZ, United States

Steven Vick,     Department of Engineering, University of Cambridge, Cambridge, United Kingdom

Yujie Wei,     Civil and Environmental Engineering, Carnegie Mellon University, Pittsburgh, PA, United States

Foreword

Construction and management of infrastructure systems regularly uses artificial vision. It promises to become even more important in the next few decades as a critical technology to improve the efficiency and effectiveness of our infrastructure services. This "Infrastructure Computer Vision" book is a welcome synthesis of the techniques and applications of such artificial perception for anyone interested in or working with infrastructure.

Artificial vision has a long and beneficial history. Eyeglass lenses first appeared around 1300 to provide corrected vision for a myriad of grateful people. Galileo used crystalline lenses to create a telescope and began viewing celestial objects in 1609 (unfortunately to his eventual peril). Use of this same lens technology for microscopes emerged soon after. Photography provided a different approach for artificial vision, using chemical processes for recording images. Photographic pictures of infrastructure began appearing in the 19th century. Television and digital cameras became widespread in the 20th century as photography moved away from chemical processes for recording images. With these technologies, artificial vision could extend to cameras within small pipes or monitoring the earth from satellites among many other possible applications.

The marriage of computing and artificial vision has been magnificent. Computing can transform different wavelengths of electromagnetic radiation into visible pictures, extending human vision into new domains. For example, infrared pictures can reveal heat loss spots in buildings. Computing can identify as built, three-dimensional structures as well as changes over time and movements from camera images. Computing on images can identify problems such as construction safety risks or component defects. With the improvement of computing power and the reduction in computing cost, artificial vision with computing can be used anywhere on our infrastructure systems. Modern computing can also handle very detailed pixel resolutions from sensors.

With all the development of computing and sensors, it is difficult for many professionals to understand the potential applications and uses of this technology as well as develop their own applications or new techniques. This book admirably fills the role of providing comprehensive yet approachable descriptions of the existing techniques and use of the technologies.

Ioannis Brilakis and Carl Haas are notable experts and leaders in the development and use of infrastructure computer vision. I have known both since their days as doctoral students. I well remember Carl’s struggles with limited computing to map out pavement cracks during his doctoral research. Much has changed since then!

Ioannis and Carl have developed a large variety of applications and techniques, from pothole detection, through construction worker movements, material tracking on construction sites, postdisaster structural assessments, and three-dimensional reconstructions, to name only a partial set. They have also been leaders in professional meetings and publications on the topic. Readers are fortunate to have these two knowledgeable guides for the complex and evolving domain of infrastructure computer vision.

I trust you will find new enthusiasm for the use of computer vision to improve our infrastructure systems from this book. Enjoy!

Chris Hendrickson,     Hamerschlag University Professor Emeritus, Carnegie Mellon University Member, US National Academy of Engineering

Co-author: ‘Project Management for Construction’ and ‘Fundamentals ofInfrastructure Management’

Acknowledgments

The authors gratefully acknowledge the support of key organizations and colleagues. The Leverhulme Trust’s International Academic Fellowship award along with a Visiting Professorship at the Technical University of Munich made it possible for Ioannis Brilakis to take a sabbatical and concentrate on this book. The University of Waterloo’s IRPG Program award along with a Visiting Professorship at the ETH (Eidgenössische Technische Hochschule) in Zurich as well as a German DAAD award for visiting the Technical University of Berlin made it possible for Carl Haas to also take a sabbatical and concentrate on this book. Particularly, we thank the chapter authors and their student assistants who tirelessly worked through draft revisions to bring this book to its standard of quality. Further acknowledgments must go to the Elsevier staff for their strong technical and editorial support. Special acknowledgments to Professor Chris Hendrickson from Carnegie Mellon University for the Preface for this book.

Chapter 1

Introduction

Ioannis Brilakis ¹ , and Carl Haas ²       ¹ Department of Engineering, University of Cambridge, Cambridge, United Kingdom      ² Department of Civil and Environmental Engineering, University of Waterloo, Ontario, Canada

Keywords

infrastructure; computer vision; infrastructure computer vision; construction automation; image analysis

1.1 Executive summary

1.2 The current business environment for ICV

1.3 Documented inefficiencies of traditional approaches

1.4 ICV: new tools and new processes

1.4.1 Level of automation

1.4.2 Application scope or use case

1.4.3 Capabilities

1.4.4 Delivery model

1.4.5 Dominant data source

1.4.6 Learning curve

1.4.7 Intended users

1.5 Synergies of ICV and other technologies

1.6 What is not ICV technology?

1.7 What challenges can be expected?

1.1. Executive summary

Infrastructure computer vision (ICV) has become a thriving commercial ecosystem within the engineering, construction and infrastructure business, and government sectors. Within this ecosystem, technical advances in sensors, deployment platforms, software as a service (SaaS) models, algorithms, development environments, and cloud-based delivery systems are all integrated into multidisciplinary services that involve highly trained computer vision specialists, field engineers, software developers, and business entrepreneurs.

Separating this ecosystem from those such as the 3D design environments (e.g., Building Information Modeling [BIM]), emerging 3D printing technologies, automated prefabrication systems, asset management systems, and the emerging digital twin environments is impossible. They are symbiotic and synergistic ecosystems. Understanding ICV therefore is more than just a technical challenge. It is a challenge in understanding complex systems spanning a number of domains. If this is true, where should we place the boundaries of the ICV body of knowledge, within which this book will delve?

Fig. 1.1 is a roadmap to help answer that question, describe the ICV body of knowledge, and explain the structure of this book. It describes ICV as consisting of layers of technologies, services, processes, and participants. At the bottom layer are the devices and sensors that are combined and used in multiple ways to acquire the raw sensor data used to generate images and point clouds; altitude, location, and spectral data; and raw models (as in simultaneous localization and modeling—SLAM—systems). Those data are used in the data acquisition layer by integrated hardware and software systems to generate image data in structured forms that are nominally standardized so that they are useful at the data management layer. At this layer, data are structured, further defined by meta-data, organized into databases, fused with 3D models' data, resampled, and converted into representations more useful to the next layer. At the next layer, experts use mathematical, statistical, or artificial intelligence (AI) tools to convert structured image data to useful models such as BIMs and digital twins, derive information (such as the existence, location and orientation of objects of interest), and add semantic content such as material properties and defects of objects and surfaces scanned. This information begins to generate business value at the services layer. This layer serves to detect incidents, such as heavy equipment dangerously close to workers; measure flow of people, materials, and equipment, including queuing assessment and activity analysis (e.g., direct work rate); assess conditions such as the pavement serviceability index from multiple sources of information; apply dimensional quality control such as floor flatness, steel alignment, and pipe fitting; conduct quantity and earned value tracking for construction projects; detect and track materials for management applications; and control fabrication processes through fit-up analysis, relative position, and orientation assessment (e.g., robotic brick laying) and volumetric analysis (e.g., automated welding and earth moving). All this information is used at the next layer in business and government applications that add value, such as asset management, project control, and design. At the top layer is the pull from owners, operators, fabricators, constructors, and architects who match market needs to ICV business and service offerings. In practice, researchers and businesses typically span some of these layers and their elements.

Figure 1.1 ICV body of knowledge covered in this book.

For example, consulting companies combine AI, computer vision, and manual activities to process images from unmanned aerial vehicle (UAV)-mounted and fixed-location cameras to (1) monitor safety conditions such as hardhat-wearing compliance, (2) conduct activity analysis that classifies workforce direct work, travel, planning, idle, and other states over time, (3) maintain security over materials stores and laydown yards, (4) conduct automated earned value tracking for volumetric activities such as concrete, steel, and services placement, and (5) support remote presence for senior management and engineers. Other companies collect road surface condition data and process it using manual and computer vision–assisted approaches to deliver pavement condition assessments to transportation asset managers and owners. Other companies make the laser scanners and the software that makes using the laser scanners easy and streamline the process of acquiring merged point clouds and feeding them into the information generation layer (Fig. 1.1). Research groups at universities typically focus on one or two of the layers in Fig. 1.1.

In this book, the following chapters generally progress through the layers in Fig. 1.1 from the bottom up. However, examples that help describe the application of a basic concept described in chapter 2 on data acquisition might focus on an asset management application implemented by a transportation system owner-operator. Chapter 3 necessarily spans the data acquisition, management, and information generation layers, while reaching up and down the layers to develop and explain specific examples. The following chapters from the owner's, engineer's, architect's, contractor’s, subcontractor’s, and vendor's perspectives typically span the services and business layers.

In the following sections, the scope, nature, and context of ICV are further explored in order to help the reader better understand and apply the knowledge transferred in this book.

1.2. The current business environment for ICV

It is not necessarily obvious to early career readers with little work experience what is the industry model within which ICV exists. It varies in its nature across different regions of the world, and acronyms abound that have different meanings depending on context and world region. At its simplest, the ICV business environment boundary for this book can be described as infrastructure and construction engineering and management. Typically, a business environment can be further defined by its stakeholders, sectors, drivers, trends, sources of change (external and internal), and key uncertainties.

Infrastructure and construction engineering and management stakeholders and their roles with respect to ICV include the following:

• Owners—Infrastructure and business capital asset owners are responsible for maintaining and operating existing assets as well as planning, financing, and managing the delivery (building) of new assets. Their interest in ICV is typically related to its potential for efficiently delivering objective, detailed, and quantitative asset condition information such as cracks and corrosion in building elements. They also have an interest in using ICV to generate as-built models and digital twins of facilities where their records are poor, which unfortunately is most often the case. For example, a power plant owner may have out-of-date two-dimensional drawings from decades earlier when the plant was first constructed, and they need information about precise dimensions, locations, and tie-in points for modifications and additions to the plant that can be acquired using ICV.

• Asset managers—Many large asset management firms exist who provide the service to owners (such as health care providers) of managing their distributed physical assets, because the owners recognize that their core business and capabilities may not directly relate to the assets from which they are delivered, such as health care delivered from hospitals. Asset managers are interested in condition assessment, for example, as well as the modeling, visualization, and communications that interest owners. Typically, they might use applications of ICV such as building heat loss analysis.

• Engineers and architects—Engineers and architects design and analyze for owners and asset managers. They may focus on as-built models derived from ICV for design, visualization, and interference checking. They are also interested in highly specialized applications of ICV such as scan-to-FEA (3D scans conversion and structuring for finite element analysis), for example, for turbine blade condition assessment and reverse engineering of physical asset elements.

• Contractors—Contractors are called constructors in some countries and builders in others. They are responsible for marshaling the resources required and managing them to make what the architects and engineers have designed. ICV appeals to them as a tool for project cost, schedule, and quality control (for example, ICV helps with automated quantity tracking), for safety improvement (ICV may be used to detect dangerous conditions), and for risk mitigation (ICV can be used, for example, to document as-installed conditions to avoid or quickly resolve costly conflicts that occur when such information is disputed after the fact).

• Subcontractors—Distinguished by their higher level of specialization and narrower scope of work, subcontractors such as glazers use ICV to control exterior cladding unit dimensions so that they fit the unique connections points geometry of their target locations on the exterior of buildings.

• Fabricators—The work of fabricators can be similar to subcontractors, except that (1) it is normally located in facilities (fab shops) rather than the construction site, (2) the product might be described as engineered materials, and (3) typically they implement more automation and robotics. They are interested in using ICV to help control fabrication and, like contractors, to mitigate risks.

• Equipment and software vendors—Companies such as FARO, Trimble, Bentley, and others serve as technology providers and vendors, provide support, and service for the scanners, cameras, devices, and software used to acquire vision data. There are many dozens of players in this field, and the landscape and products they provide are evolving at such a rapid pace that we do not try to organize it in this book. Instead, we provide the readers with the key words, knowledge, and insight they need to investigate and evaluate the most recent offerings.

• Consultants—This class of stakeholders is rapidly evolving at the time of writing of this book, with new companies being announced almost weekly.

• Academia—Many of the discoveries that have led to new imaging devices such as structured lighting kits and to the most widely used algorithms have come out of the academic research environment. Some of the consultants and software-as-a-service vendors are also emerging from entrepreneurial ecosystems centered at leading research institutions globally. Nimble academics in construction and infrastructure engineering and management often partner with owners, vendors, and constructors as well to advance development and deployment of computer vision applications.

It is also important to outline the industry sectors relevant to ICV, because the domain knowledge, supply chains, project delivery models, special skills and equipment required, players, and types of repetition all distinguish them to the extent that companies tend not to cross the barriers between them despite the fact that we might consider them all related to or part of the overall industry of construction. For example, the US military divides construction into horizontal and vertical sectors. In North America, the sectors are typically split into residential, commercial, heavy, and industrial. Management consultants, estimating-data providers, and government statisticians will divide the industry into many more sectors such as types of residential, commercial, and industrial. This matters to ICV, because at the higher layers in Fig. 1.1, ICV service providers and applications will cluster into sectors. Examples include (1) terrestrial, its very large geometric and geographic scale attracts consultants who focus on light imaging detection and ranging (LIDAR) mounted on UAVs and on lower resolution and accuracy applications such as heavy construction and mining cut and fill quantities as well as terrain features; (2) buildings, with applications requiring smaller scale and more accuracy, providers in this area may focus on 2D computer vision for monitoring, or on quality and quantity control applications using 3D tripod-mounted laser scanners; and (3) fabrication process control service providers; they may prefer precision structured lighting scanners or SLAM devices that stream dimensional data directly and efficiently to real-time robotic applications. Companies offering products and services tend to specialize in one or a couple of these areas, and they are not typically competent in the other areas. This is important to know when buying their products and services.

Drivers and trends in the industry that may impact ICV include a steady movement toward more automation and integration of IT (information technology), yet this is counteracted by the resistance of the industry to share data such as 3D design models. Since 2010, BIM has gone past the tipping point and become widely adopted, which, when coupled with an emerging class of engineers who think and design directly in 3D, bodes well for increased applications of ICV. AI (artificial intelligence) is in its third surge on the innovation curve, and developments such as deep learning are making ICV applications such as automated equipment detection, identification, and tracking viable. Collaborative delivery of projects by project stakeholders who are widely dispersed geographically drives the use of remote presence and thus ICV that supports monitoring, augmented reality, and remote-control applications. Yet, mitigating against increased use of ICV is the persistent industry fragmentation, both vertically and horizontally, which serves as an impediment to data sharing, trust, innovation, supply chain visibility, and standards. All of these are required for the broader application of ICV.

Uncertainties around the industry abound as well are as follows:

Will automation be used to augment or replace labor? The outcome may drive the sort of ICV applications that are accepted and adopted widely.

Will circular economy principles drive infrastructure development? For example, if they do, then we will see more adaptive reuse and recycling of infrastructure assets and building elements, which will require ICV for deriving as-built information useful for planning for reuse projects.

Will IT and social media integration continue unimpeded? Crowd sourcing to support asset image data acquisition strategies is key to some emerging business models for ICV. Resistance or reversal of this trend will impede such models.

Will virtual replace face-to-face?

The science breakthroughs in computer vision and machine learning arrive at such a staggering rate, that it is impossible to make any reliable mid-long-term ICV plans.

Will online collaborative applications and environments lead to more trust and cooperation or less?

Will the trend toward more prefabrication continue?

So, while we have just finished describing the current business environment in term of its stakeholders, sectors, drivers, trends, sources of change (external and internal), and key uncertainties, we can only guess what the future holds for us. What we do know is that there are known inefficiencies in current business approaches and processes that can be addressed with ICV.

1.3. Documented inefficiencies of traditional approaches

Humans are unequaled by machines and computers at quickly assessing situations in scenes in all their complexity of context, emotion, conflict, ambiguity, and intent. We see hidden patterns. We can immediately recognize what is important and react or respond appropriately. We can plan complex future actions, communicate them to people and machines, and then command or motivate them to take action. We can anticipate primary, secondary, and tertiary waves of effects of our actions. We can appreciate the beauty in an image. Machines, and computer vision in particular, cannot do any of these things. However, they can be used to do many things well that people are not good at. Computer vision is useful for the following:

• Quantification—While humans can guess linear, area, volumetric, and other spatial properties, computer vision can be used to precisely and accurately qualify those properties. For example, computer vision can be used to measure soil and pavement lift quantities and depths accurately and precisely, while humans can only estimate visually or with point measurements. ICV can also be used to easily measure particle dimensions individually and collectively for vast quantities of particles and aggregates.

• Objectivity—Computer vision can be used to measure facial features and expressions more objectively than most humans can. Computer vision algorithms hold no biases. They can be used to evaluate quality as well, such as roughness and levelness, according to clear and understandable formulas.

• Consistency—It has been demonstrated many times that humans evaluate infrastructure surface conditions inconsistently over time. We may evaluate unchanged conditions differently from one evaluation repetition to another, because we are affected by our emotional state, context, and previous suggestions. Computer vision does not suffer from these traits.

• Uninterrupted attention—It is humanly impossible to maintain perfect, uninterrupted attention over long periods of time, so events often go unobserved. Computer vision can be continuous, uninterrupted, and relentless. This can be useful for identifying unsafe situations on a construction site or counting traffic on a road.

• Complete records—While we may observe an event, we may forget to record it or record it incorrectly because of trivial transcription errors. Computer vision systems can be constructed to record every observation and event. Of course, this can also create problems in terms of information overload and data management if not managed correctly.

• Speed—For some quantification, recognition, and classification tasks, computer vision can be quicker than humans. For example, fitting a parametric function to a set of points in 2D or 3D perfectly is hard for humans and sometimes requires iteration, such as in situations when we are creating as-built BIMs from 3D scan point cloud data. With some human direction, computer vision can fit objects or cuboids to a cloud of points quickly and precisely.

• Augmentation of reality—Computer vision can be useful for providing layers of information that can be used to augment reality in visual environments in ways that are not possible for humans.

• Following processes exactly—We know that operational and project success partly relies on following well-defined processes consistently and without exception. Unfortunately, even with automated workflow engines to keep us in line, humans are not good at repeating processes consistently. Computer vision can be used for process activities and to follow workflows repeatedly, without limit.

Given all these advantageous characteristics, it is not surprising that ICV is being used to create a myriad of new tools and processes.

1.4. ICV: new tools and new processes

An increasing number of companies are using 3D imaging devices as their cost is reducing and computing power is becoming inexpensive. Examples have been described in the preceding sections. In addition to Fig. 1.1, there are ways in which new tools and processes are emerging that are worth understanding, so that developments can be anticipated for planning and decision-making. The space can be broken down into the following factors to better understand how ICV businesses structure themselves, their strategies, and their offerings, and to be able to compare available services in the market for purchase:

• Level of automation

• Application

• Abilities

• Delivery model

• Dominant platform for data source

• Learning curve

• Intended users

1.4.1. Level of automation

This can be quite deceptive. Process control systems that measure dimensions for example, can be automatic and autonomous. However, processes such as dimensional measurement and scan-to-BIM from 3D point clouds can often require many hours of highly trained engineers' time using commercial software packages that only automate underlying computational and recoding functions. Other software, for example, for activity analysis of construction workers, might require vast amounts of crowd sourcing input or postprocessing manual corrections to be used for commercial applications. It is often not obvious to a layman what is automated and what is the related impact on application success. In addition to technology capabilities, the automation level may also be driven by application requirements. For fabrication process controllers, two levels of automation can be defined. Simple, highly automated applications (apps) are required, if the intended users are fabrication workers with limited knowledge in 3D imaging software. Less automated (and therefore more flexible) systems can be useful if the intended users are highly trained engineers, for example. Businesses focus on some balance therefore between the following two extremes for level of automation:

(1) Manual processing of data in conventional software environments—Companies that fall in this category require high level of effort to execute subsequent value adding operations, such a deformation or progress location, measurement, and quantification

(2) Software with functions that automate processes—Companies that fall in this category do not require a high level of effort to execute subsequent value-adding operations.

1.4.2. Application scope or use case

Within the infrastructure and construction industry, 3D imaging technologies have been used for the applications described earlier. Geometric and geographic scale matter as well. It is uncommon for tools and processes to be useful or for companies to be capable across scale boundaries and industry sectors.

1.4.3. Capabilities

Capabilities and applications have also been described in the preceding sections and in Fig. 1.1. No tool on the market has unlimited capabilities. However, scope of applications and level of autonomy are consistently being increased, and that will be expected to continue well into the future.

1.4.4. Delivery model

For tools and processes, delivery models differ substantially, with respective deployment and security advantages and disadvantages. Delivery model is another indicator of the core business strategy for many companies. Companies tend not to be good at more than one or two delivery models in the ICV sector. Delivery models include

(1) SaaS (software as a service)

(2) Platform deployed software

(3) Services

(4) Hardware

1.4.5. Dominant data source

The primary source of acquisition can dictate the quality and the diversity of possible postprocessing functionalities. One of the important strategic assets of information and services layer companies is to be able to be data capture method agnostic. Otherwise, companies and their customers can be caught easily in the trap of obsolescence. Data capture methods from a business perspective include

(1) UAV with LIDAR or structure from motion (SFM)

(2) SLAM

(3) Laser scanners

(4) Structured lighting scanners

(5) SFM alone

(6) Combined data capture methods

1.4.6. Learning curve

The ICV technology provider market offers solutions that range from very easy to learn and use to very complicated to do either. The level of complexity is not necessarily a sign of better performance across any key performance indicator, so the learning curve should be used to prioritize solutions that are easy to learn and use.

1.4.7. Intended users

Stakeholders discussed in section 1.2 comprise most of the population of intended users of ICV tools and processes. However, individual crafts people and small subcontractors are also emerging as users. Similar to the hand tools used in mechanical shops, some tools are meant for the casual consumer and some are meant for industrial grade applications and are priced accordingly. New tools and processes will continue to be released across all sectors.

1.5. Synergies of ICV and other technologies

ICV tools and processes are synergistic with many technologies, such as BIM, robotics, 3D printing, augmented reality, UAVs, cloud-based computing and communications, workflow engines, artificial intelligence, geographic information systems (GIS), global positioning systems (GPS), and animation. Much of these synergies have already been alluded to in preceding sections. Many synergies will be described in the subsequent chapters. Their synergy is derived from common modalities such as 2D, 3D, and 4D spatial data, data processing methods (such as translation and rotation and scaling), and data fusion advantages.

The synergy of ICV tools with two core technology families, i.e., Digital Twins (back-end) and automation (front-end), is of particular importance. Digital Twins rely on ICV to capture scene geometry and enrich it into a model, to capture additional attributes from the scene (e.g., material types, condition data), and to keep the model up to date over time. Conversely, ICV uses Digital Twins to guide dimensional control applications and check compliance in many use cases. Automation uses ICV to deliver robotic solutions, automation systems and processes, key performance indicator monitoring tools, and others. This is why ICV is becoming a core built environment subdiscipline and a required set of skills needed for young professionals in their future career.

1.6. What is not ICV technology?

ICV is not the technologies listed above, with which it is synergistic. That can be confusing, as specific applications that combine these technologies often blur the boundaries, especially augmented reality, which uses combinations of BIM, ICV, animation, inertial motion units, retinal display optics, and other technologies. Materials tracking technologies, and others that gather spatial data, are not ICV. Essentially, the difference is that ICV involves acquiring, processing, and understanding 2D or 3D data. Images and any scans of scenes that are greater than one dimension using a wide variety of sensors are the source data.

1.7. What challenges can be expected?

Some challenges have plagued ICV since its beginning. ICV can require massive amounts of processing power and continues to press the boundaries of processing power capabilities that are reaching the limits of Moore's law. For the first time in decades, in the near future, we may reach the point where we need to ration ICV applications.

Because ICV can be uninterrupted, it can create an information overload. Prioritizing, designing, and managing massive vision databases will be a challenge. Practical physical and financial constraints such as energy consumption of data farms will become very real. As libraries have known for millennia, collections must be continuously cataloged, weeded, and renewed for relevance and usefulness.

Ethical challenges abound for a technology that can be used to monitor and record people and events ubiquitously. In France, a percentage of traffic cameras have been purposefully destroyed by those who perceive such surveillance systems as impinging on their privacy, freedom, and human dignity. It is likely that more social movements may emerge, and we need to take these social concerns seriously.

In a related discovery in recent years, vision systems trained with biased data sources create biased results. "Researchers tested features of Microsoft and IBM's face-analysis services that are supposed to identify the gender of people in photos. The companies' algorithms proved near perfect at identifying the gender of men with lighter skin, but frequently erred when analyzing images of women with dark skin. The skewed accuracy appears to be due to underrepresentation of darker skin tones in the training data used to create the face-analysis algorithms ¹ " Relevance to ICV includes many of the algorithms emerging for recognition of vehicle and infrastructure features that are trained for limited geographic regions and cultural milieus.

The degree of acceptable accuracy is another challenge. Many published advances describe accuracies of 90% or more. However, in some applications, even very small false alarm rates become overwhelming to operators who then shut down the systems, and even a few modeling errors in an as-built model could create expensive and time-consuming subsequent rework if over relied on.

In summary, we need to be optimistic about the advantages but cautious when we use ICV.

¹  Joy Buolamwini and Timnit Gebru, 2018. Proceedings of Machine Learning Research, Conference on Fairness, Accountability, and Transparency Gender Shades: Intersectional Accuracy Disparities in Commercial Gender Classification, 81, 1–15.

Chapter 2

Surveying, Geomatics, and 3D Reconstruction

Pingbo Tang ¹ , Steven Vick ² , Jiawei Chen ¹ , and Stephanie German Paal ³       ¹ School of Sustainable Engineering and The Built Environment, Arizona State University, Tempe, AZ, United States      ² Department of Engineering, University of Cambridge, Cambridge, United Kingdom      ³ Zachry Department of Civil and Environmental Engineering, Texas A&M University, United States

Abstract

This chapter provides an overview of various surveying and computer vision techniques that can produce geometric and visual information necessary for civil infrastructure condition assessment and maintenance planning. One main objective of surveying infrastructure with computer vision is to acquire detailed and accurate geometric and visual data and construct digital models within given time and resource limits of the application scenarios. Aiming at this objective, this chapter defines domain requirements and resource limits in a quantitative manner and examines various computer vision techniques' capabilities for satisfying these domain requirements. After studying this chapter, readers should be able to select and apply appropriate surveying techniques for given domain applications that have specific domain requirement.

Keywords

3D reconstruction; Data collection planning; Laser scanning; Photogrammetry; Requirement modeling; Spatial data quality

2.1 Executive summary

2.2 Introduction

2.3 The evolution of surveying infrastructure: from theodolites to mobile phone cameras

2.4 Surveying user requirements

2.4.1 Typical use cases

2.4.2 Data quality requirements

2.4.3 Time and resource requirements

2.5 Overview of the major ICV-based surveying tools

2.6 Terrestrial laser scanning and point cloud registration

2.6.1 Data collection principles of laser scanning and field data quality checking methods

2.6.2 Factors that influence TLS data collection and best practices of TLS field operations

2.6.3 Field data quality visualization and checking methods for individual scans

2.6.4 3D laser scanning point cloud registration and registration errors

2.6.5 Quality checking and control of registered data for reliable DT generation of civil infrastructures

2.7 Photogrammetric and videogrammetric 3D reconstruction

2.7.1 Photogrammetric theory

2.7.1.1 Feature