University of Southern California Dissertations and Theses

A data driven software platform for process automation, planning and inspection of Contour Crafting large-scale robotic 3D printing system

(USC Thesis Other)

A data driven software platform for process automation, planning and inspection of Contour Crafting large-scale robotic 3D printing system

PDF

Download a page range

Download transcript

Copy asset link

Request this asset

Transcript (if available)

Content 1 | P a g e A Data Driven Software Platform for Process Automation, Planning and Inspection of Contour Crafting Large-Scale Robotic 3D Printing System By: Omid Davtalab A Dissertation Presented to the FACULTY OF THE GRADUATE SCHOOL UNIVERSITY OF SOUTHERN CALIFORNIA In Partial Fulfillment of the Requirements for the Degree DOCTOR OF PHILOSOPHY (CIVIL ENGINEERING) December 2018 2 | P a g e Table of Contents List of Figures ............................................................................................................................ 4 List of Tables ............................................................................................................................. 6 Acknowledgements .................................................................................................................... 7 Executive Summary ................................................................................................................... 8 Chapter 1: Automated Process Planning...................................................................................11 1. Introduction ....................................................................................................................11 2. Statement of the Problem...............................................................................................13 2.1 Integration of BIM and Automated Construction.......................................................13 2.2 Material Procurement and Site Logistics Planning for Automated Construction System ..........................................................................................................................13 3. Background of the Study ................................................................................................14 4. Critique of Past Approaches ...........................................................................................15 4.1 Integration of BIM and Automated Construction.......................................................15 4.2 Material Procurement and Site Logistics Planning for Automated Construction System ..........................................................................................................................16 5. Hypothesis .....................................................................................................................17 5.1 Hypothesis 1 for Integration of BIM technology with Automated Construction Systems ........................................................................................................................17 5.2 Hypothesis 2 for Material Procurement and Site Logistics Planning of Automated Construction Systems ....................................................................................................18 6. Research Plan and Procedures .....................................................................................19 6.1 Integration of BIM and Automated Construction.......................................................19 6.1.1 Methodology ....................................................................................................19 6.1.1.1 Extracting BIM Model Data using IFC Data Model .............................21 6.1.1.2 Toolpath Optimization ........................................................................24 6.1.2 Results .............................................................................................................30 6.1.2.1 BIM Model Design .............................................................................30 6.1.2.2 Data Extraction and Analysis in POCSAC Software...........................32 6.1.2.3 Automated Construction using Contour Crafting ................................35 6.1.3 Conclusion .......................................................................................................36 6.2 Material Procurement and Site Logistics Planning for Automated Construction System ..........................................................................................................................41 6.2.1 Methodology .............................................................................................42 6.2.1.1 Material Procurement Variables ..................................................43 6.2.1.2 Site Logistics Variables ...............................................................46 3 | P a g e 6.2.1.3 Material Procurement Optimization System ................................49 6.2.1.4 Site Logistics Optimization System .............................................55 6.2.2 Results ......................................................................................................60 6.2.3 Conclusion ................................................................................................62 7. Future Research Plan and Procedures ..........................................................................63 Chapter 2: Automated Inspection System .................................................................................68 1. Introduction ....................................................................................................................68 2. Statement of the Problem...............................................................................................69 3. Background of the Study ................................................................................................70 4. Critique of the Past Approaches .....................................................................................71 5. Research Objective and Questions ................................................................................73 6. Research Plan and Procedures .....................................................................................74 6.1 Automated Concrete Layers Detection ..................................................................77 6.1.1 Methodology ..................................................................................................77 6.1.1.1 Image Capturing System for Data Collection ...................................77 6.1.1.2 Image Data Preprocessing ..............................................................77 6.1.1.3 Image Data Augmentation ...............................................................78 6.1.1.4 Image Pixel-wise Segmentation and Labeling for Concrete Layers Detection Model Training ............................................................................80 6.1.1.5 Model Training using Convolutional Neural Network (CNN) ............82 6.1.2 Results ..........................................................................................................86 6.2 Automated Detection of Defects in Layers ............................................................88 6.2.1 Methodology ..................................................................................................88 6.2.1.1 Image Preprocessing ......................................................................88 6.2.1.2 Edge Detection ...............................................................................92 6.2.1.3 Layer Lines formation ......................................................................94 6.2.1.5 Defect Detection .............................................................................98 6.2.1.6 Evaluation Process ....................................................................... 100 6.2.2 Results ........................................................................................................ 101 7. Limitations and Future Research Work ........................................................................ 105 Bibliography ............................................................................................................................ 108 Appendices ............................................................................................................................. 112 A.1 IFC Data Model ......................................................................................................... 112 A. 2 TSP Algorithm and Conversion Steps of Raw 2D Graphs ......................................... 115 A.3 G-Code Generation ................................................................................................... 123 4 | P a g e List of Figures Figure 1 - (a) Contour Crafting machine (b) 3D printed concrete element (c) Contour Crafting using sulfur concrete for planetary applications [10, 11] .................................... 9 Figure 2 – Overall framework proposed for Planning and Operations Control Software for Automated Construction (POCSAC). ....................................................................... 10 Figure 3 - History of Proposed Automated Construction Systems [36] ......................... 11 Figure 4- Methodology and Framework Flow ................................................................ 19 Figure 5 - Sequence of Steps for Extracting Essential Data from BIM Models .............. 21 Figure 6 - 1-Bedroom Apartment Floor Plan Layout ...................................................... 26 Figure 7 - 3D Model Designed in Revit 2017 ................................................................. 26 Figure 8 - 2D Graph Model extracted using IFC Parser ................................................ 27 Figure 9 - Dimensions of the first layer including openings ........................................... 27 Figure 10 - JSON File summary extracted using IFC Parser ........................................ 28 Figure 11 - 2D Layer Layout Including both Artificial and Non-Artificial Nodes ............. 30 Figure 12 - 3D Model Designed in Revit (ceiling is hided) ............................................. 32 Figure 13 (a) Extracted Exterior and Interior Walls from the IFC Model (b) Sample Preliminary Graph Nodes (c) Graph Dimensions ......................................................... 33 Figure 14 - New Graph Generated in POCSAC with Updated Node Indices ................ 33 Figure 15 - Printed Case Study Model using Contour Crafting Machine ....................... 36 Figure 16 - Components of CPP-CC Model .................................................................. 42 Figure 17 - Construction Site Overall Layout ................................................................. 49 Figure 18 - Deterministic Inventory System with Minimum Inventory Level used in CPP- CC Model ...................................................................................................................... 51 Figure 19 - Total Cost tradeoff between Cost of Ordering and Cost of Space .............. 53 Figure 20 - Predetermined Spaces on The Construction Site ....................................... 56 Figure 21 - 3D Rendered Mode in Revit for Site Layout Facilities ................................. 62 Figure 22 - Revit Model sample including Mechanical and Electrical Components [33] 64 Figure 23 - Drone System providing real-time construction site data [34,35] ................ 66 Figure 24 - Pixel-wise segmentation using SegNet Model [45] ..................................... 72 Figure 25 - Schematic view of the proposed system ..................................................... 75 Figure 26 - Binary Mask Ground Truth Creation for Training Data Labeling Process a) Sample image captured from Nozzle facing y axis, b) Sample image captured from Nozzle facing x axis....................................................................................................... 81 Figure 27 - CNN Network Architecture designed for concrete layers extraction ............ 84 Figure 28 - Sample of Model Pixel-wise segmentation.................................................. 88 Figure 29 - a) Original Single Channel Image b) Transformed Image using Bilateral Filter c) Pixel Intensity Histogram for Original and Transformed Images....................... 92 Figure 30 - Preserving maximum value pixels in gradient direction [43]........................ 93 Figure 31 - Edge detection process with and without texture removal .......................... 94 Figure 32 - Line Representation Coordinate System used for Hough Transformation .. 96 5 | P a g e Figure 33 - Example of Hough Transform Process [44]. a) Line passing from preserved pixels b) Chart of line values passed from each pixel to detect intersections and upvoted pairs .............................................................................................................................. 97 Figure 34 - Iteration process for outlier detection over the entire image ........................ 99 Figure 35 - Sample 1: An image of 4 concrete layers with defect in the bottom layer . 103 Figure 36 - Sample 2: An image of 5 colorful concrete layers with defect in the 2nd bottom layer ................................................................................................................ 104 Figure 37 - Sample 3: An image of 9 curvy concrete layers with no defect ................. 105 Figure 38 - Texture removal issue caused by higher distance of image capturing spot .................................................................................................................................... 106 Figure 39 - Revit model in IFC format ......................................................................... 113 Figure 40 - Non-Artificial Nodes Considered for a Wall intersection ............................ 120 Figure 41 – Artificial Nodes for a Wall Intersection with Orthogonal Rotations ............ 120 Figure 42 – Directions of Printing in Contour Crafting System [11] ............................. 123 Figure 43 - Pseudocode for generating robot G-Code ................................................ 124 6 | P a g e List of Tables Table 1 - Sequence of the Nodes in the Optimal Solution Toolpath .............................. 34 Table 2 - Total Edge Counts and Distances Analysis based on The Sequence of Optimal Solution ............................................................................................................ 35 Table 3 - Wall Sizes and Material Quantities ................................................................. 44 Table 4 - Material Consumption Ratios during Automated Construction Process ......... 45 Table 5 - Automated Construction Process Facilities Size and Type ............................ 48 Table 6 - Size and Geometry Centers of Available Locations on the Construction Site 57 Table 7 - Travel Cost Matrix between All Facilities ........................................................ 57 Table 8 - Initial Chromosome String used in Genetic Algorithm .................................... 60 Table 9 - Solution Matrix representing Material Ordering Quantities during Automated Construction Process .................................................................................................... 60 Table 10 - Solution Chromosome String Generated for site layout facilities .................. 61 Table 11 - Model Performance Metrics for both Concrete Layers and Non-concrete pixels ............................................................................................................................. 87 Table 12 – Proposed Defect Detection Approach Performance .................................. 105 Table 13 - G-code Main Commands for 3D Printing Process ...................................... 124 7 | P a g e Acknowledgements Firstly, I would like to express my sincere gratitude to my advisor Professor Khoshnevis for his continuous support, guidance and encouragements through my Ph.D. study. Having such a creative adviser with great passion and immense knowledge gave me a great motivation every day to keep on working hard while enjoying the Ph.D. road as his student. His guidance helped me all the time not only for my research and writing of this thesis but also my viewpoints about the life in different aspects. I would not be the person I am today if it weren't for his influence and supervision. Besides my advisor, I would like to thank the rest of my thesis committee: Dr. Lee and Dr. Razaviyayn for their insightful comments and guidance which helped me to broaden my research from various perspectives and keep working on the right path. I would like to thank my lovely family for their invaluable sacrifices and supporting me throughout all these years. I know I would never be able to reach this point of my life without their everlasting love, support and kind efforts. I would also like to thank my grandmother for her love and support in all these years which is incredibly admirable. I would also like to thank Dr. Chris Mattmann, director of Information Retrieval and Data Science group (IRDS) at USC, with whom I had the honor to work with through the last year of my PhD and learn a lot from him and other people at USC IRDS. Last but not the least, I would also like to thank all my lab-mates specially Ali Kazemian for providing me samples that I utilized in several parts of my research. I would also like to express my gratitude to my friend and lab-mate, Dr. Xiao Yuan whose advice helped me a lot during my PhD program. 8 | P a g e Executive Summary Automated construction through concrete 3D printing is deemed as a revolution in construction industry. This novel idea of scaling up additive manufacturing techniques for automated building construction has been topic of academic and industrial research for several years. Many advantages are offered by automated building construction such as superior construction speed and higher degree of customization. There are considerable efforts on use of robots for various construction purposes such as pre- fabrication [1, 2] or non-load-bearing structures [3]. However, the focus of this thesis is on automated in-situ building construction using additive manufacturing techniques. Contour Crafting (CC) is the first additive fabrication technology developed for in-situ construction of custom-designed structures [4]. It can radically reduce the construction cost by reducing the workforce required for the construction process. Various advantages such as better surface quality, higher fabrication speed and broader choice of materials are offered by the CC technology compared to other layered fabrication systems [5, 6]. CC is also considered as one of the most promising approaches for construction of human settlements on other planets such as the Moon and Mars [7, 8, 9] (a) (b) 9 | P a g e (c) Figure 1 - (a) Contour Crafting machine (b) 3D printed concrete element (c) Contour Crafting using sulfur concrete for planetary applications [10, 11] A review of past and ongoing projects reveals that so far most of the efforts in this area are focused on robot development and hardware improvement [11], as well as material performance [12]. However, new software platforms are also highly essential for realizing a fully automated and reliable building construction system. Hardware and material related topics have been improved extensively over the past years while software related problems and needs are less discussed. The main objective of this thesis is to provide a software platform for large-scale robotic 3D printing process using layer by layer additive manufacturing system. The software is named Planning and Operations Control Software for Automated Construction (POCSAC). The thesis is divided into two major chapters that covers the preprocess and 3D printing process stages of an automated large-scale 3D printing system. In chapter one, a proposed framework presents details of interoperation between different components of an automated construction system and BIM platform such that maximum benefit is realized through synergy of the two technologies. Chapter one parts that cover preprocess tasks are: a) Integration of Building Information Modeling (BIM) for robotic construction, b) Material Procurement and site logistics planning for an automated construction project. 10 | P a g e In the second chapter, an automated inspection system is proposed to be used during the 3d printing process of concrete layers. The outcomes from the second chapter are related to distinguishing concrete layers from surrounding objects in an image frame and then detecting defects of concrete layers which are then inserted back into the construction report generated from the BIM integrated software component proposed in the first chapter of this study. Figure 2 represents the overall framework proposed for covering preprocess tasks as well as inspection task during the 3D printing process: Figure 2 – Overall framework proposed for Planning and Operations Control Software for Automated Construction (POCSAC). 11 | P a g e Chapter 1: Automated Process Planning 1. Introduction In the recent years, computer-automated construction has been the point of focus for several research groups. Different robots and machines are designed and developed to practice automated construction using robotic approaches. Automated construction undertakes rapid construction that takes a very short time to construct a structure such as buildings. Besides speed computer-automated construction offers many advantages including superior construction speed and higher degree of customization [1]. Construction process planning optimization for automated construction has an important role in the overall system efficiency. As shown in figure 3, Contour Crafting (CC) is the first additive fabrication technology that can build custom-designed mega-scaled structures [36]. Figure 3 - History of Proposed Automated Construction Systems [36] This technology scales up 3D printing to the size of buildings and other civil structures [2]. CC can radically reduce the construction cost by drastically reducing the workforce 12 | P a g e required for normal construction process. Various advantages such as better surface quality, higher fabrication speed and broader choice of materials are offered by the CC technology compared to other layered fabrication systems. Contour Crafting has received numerous major awards including NASA technology international grand prize in 2014 and it is considered as one of the most promising approaches for construction of human settlements on other planets such as the Moon and Mars [3-5]. Process planning and optimization of an automated construction project requires an integrated system to address the issues and requirements in major project stages including design and construction. The proposed integrated system should provide a framework in which a project is designed using advance designing tools and finally constructed using automated robotic approaches. This problem requires several sub- problems to be solved in order to provide a suitable process planning and optimization for automated construction systems. How to use current designing tools for automated construction systems and how to come up with the optimum construction process (including robot toolpath, material procurement and site logistics planning) are instances of the queries that the proposed system should address. Building information modeling (BIM) have been evaluated by many researchers in the recent years and it has been considered as a technology that improves the efficiency of a project during the whole process from design to operation [7]. Recent endeavors in this area such as automating the checking of compliance with various building codes based on the specific region that the project is located, automated cost estimation, scheduling, clash detection between different disciplines, energy analysis and simulations are other examples and applications of numerous benefits 13 | P a g e offered by BIM technology. A BIM model can provide various information such as geometry, geographic information, site layout facilities, and properties and quantities of every building elements and components used in the project [7]. Moreover, BIM can provide and facilitate the coordination between design, fabrication and construction partners collaborating on a construction project [8]. 2. Statement of the Problem 2.1 Integration of BIM and Automated Construction Most conventional construction projects are currently designed and carried out using Building Information Modeling (BIM) software platforms. There has been a lot of advancements in BIM technology and a lot of research studies have evaluated the benefits of using BIM technology for conventional construction projects’ life cycle from design to operation [6]. However, there is still lack of data interoperability between BIM platforms and automated construction systems such as Contour Crafting. A proper framework for integrating BIM technology and automated construction system should be defined and required components should be developed in order to overcome this issue. 2.2 Material Procurement and Site Logistics Planning for Automated Construction System Automated construction systems using Contour Crafting or other robotic approaches promise rapid construction that could take as little as one day to complete an average size single family residential building. This means that the rate of flow of materials to the site is considerably higher than a regular construction process due to high material consumption in a relatively short time. In automated construction systems such as CC, inefficiency in material procurement and storage scarcity due to inappropriate site 14 | P a g e logistics planning can result in major problems such as total construction time delay, and over-curing of deposited sections to the extent that bonding with freshly deposited sections becomes weaker than ideal. Now the key question is how the procurement process along with site logistics should be designed to guarantee the delivery of all needed material and components to the construction site on a timely manner where the construction time could be as short as one day? This motivates the development of a framework and proper algorithms to address the issues and requirements in material procurement and site logistics planning of an automated construction process. 3. Background of the Study Building Information Modeling (BIM) is widely used in construction industry from design to operations phases of a building’s lifecycle. Facilities management (FM) departments can utilize BIM technology for coordinated, consistent, and computable building information and knowledge management [37]. Although the term BIM has been defined many times before, but a clear definition prior to the discussion is yet necessary. Based on the definition by Gu et al. [38], “Building Information Modeling (BIM) is an IT enabled approach that involves applying and maintaining an integral digital representation of all building information for different phases of the project lifecycle in the form of a data repository”. Current conventional construction projects are mostly designed using Building Information Modeling (BIM) platforms such as Revit which have been widely adopted in Architecture, Engineering and Construction (AEC) industry. AEC industry have reached numerous technological advancements in the area of BIM such as generating construction schedules through automatic data extraction from BIM 15 | P a g e models [9], automated site layout planning [10], adopting BIM for operational activities, and so many other cases that are presenting the advantage of using BIM technology for all phases from design to construction. Procurement and delivery of materials and components for conventional construction systems has been studied broadly in the past. Research studies have investigated material procurement and site logistics planning for conventional construction site considering different aspects such as its relation with labor productivity [16-18], or using integration of GIS and BIM technologies to provide updated status of the materials within the supply chain. One of the major causes of productivity degradation in conventional construction projects is shortage or over-supply of materials [19]. Material shortage or excessive amounts of materials on construction sites are negative impacts of using a poor material procurement and site logistics system. Due to the relatively lengthy construction activities which could take several months to complete, ample time is usually allowed to plan and deploy the related logistics in the conventional construction projects. This fact facilitates proper material procurement and storage along with appropriate site logistics on construction sites. 4. Critique of Past Approaches 4.1 Integration of BIM and Automated Construction Almost all these researches are focused on conventional construction systems. Developing automated construction systems is certainly dependent to this technology and a framework should be established to connect automated construction technologies such as Contour Crafting to BIM technology. 16 | P a g e There have been several research studies focused on materials and robotic systems used in automated construction processes such as Contour Crafting. However, there is still no connection between common building design and construction software platforms and automated construction systems. In the first step of this proposal, a framework is proposed to overcome data interoperability issues between BIM technology and automated construction systems. Different requirements such as 3D Model data parsing and machine toolpath optimization need to be developed as key components of this framework. By adopting this framework, automated construction systems are able to utilize the tremendous advancements in BIM technology. Two components of BIM integrating with automated construction system are data extraction from BIM models using a proper data schema and toolpath optimization tools to provide optimum toolpath for machines. 4.2 Material Procurement and Site Logistics Planning for Automated Construction System There are many algorithms used by researchers for site layout planning. These algorithms can be categorized into three main groups [24]: a) Partial Layout, b) Entire Layout and c) Layout Improvements. In partial layout algorithms a facility is assigned to all possible locations to generate possible layouts from which the best layout is selected. Then, another facility is added to the chosen best layout from the previous part and the explained procedure is repeated [24 and 25]. The entire layout algorithm assigns each facility according to a predetermined allocating strategy [26]. Finally, in layout improvements algorithm, new layouts are generated based on the first layout assumption of all the facilities 17 | P a g e with respect to optimal facility locations based on the cost functions [27]. Besides of all these mentioned methods, artificial intelligence approaches have been widely adopted in recent years to provide a solution for site layout planning problems. Examples of the studies using these methods can be observed in different research studies [28]. Although Material procurement and site logistics planning have been studied widely for conventional construction systems, the current research and developed strategies do not address the issues and requirements for automated construction systems. 5. Hypothesis 5.1 Hypothesis 1 for Integration of BIM technology with Automated Construction Systems Integration of BIM technology and automated construction systems requires a framework consisting of at least the following components: 1- Industry Foundation Classes (IFC) Data Model Parser to extract the required information such as 3D geometry of the structures and materials used in the design model from BIM models 2- A machine operation optimization system, which would be toolpath optimization in case of 3D printing based technologies, where the 2D graphs representing the model layers are analyzed to provide the optimum sequence determining the nozzle traverse sequence 3- A G-code generating system which provides proper commands for the robots to construct the whole structure by combining the optimal toolpath generated for all layers to be deposited 18 | P a g e 4- A toolpath graphical simulation system to serve as a visual verification of the generated toolpath. A framework with the described requirements can successfully integrate the BIM technology and automated construction systems using robotic approaches. Evaluation of Hypothesis 1: It is proposed to evaluate the hypothesis by developing all the components required in the framework for integration of BIM technology and automated construction systems. Several test case projects are to be used to check if the proposed framework is able to properly extract required information from a 3D BIM model and finally provide the final commands for a robot used in an automated construction system. 5.2 Hypothesis 2 for Material Procurement and Site Logistics Planning of Automated Construction Systems For the reasons stated earlier, current material procurement and site logistics planning systems developed for conventional construction projects cannot be utilized for automated construction systems. A new model, that is capable of concurrently integrating material procurement variables with dynamic site layout planning variables to optimize final automated construction planning decisions would be needed to be applicable to automated construction site logistics planning. Evaluation of Hypothesis 2: In the new proposed model a proper goal function in which both material procurement and site logistics planning costs are minimized and all relevant constraints are satisfied, will be used to compare various solutions and evaluate the optimality of the final generated solution. The result will be compared with the sub-optimal solution generated by the conventional framework. 19 | P a g e 6. Research Plan and Procedures 6.1 Integration of BIM and Automated Construction 6.1.1 Methodology In this part a framework for integrating BIM technology and automated construction systems is proposed. This framework has two main components that are: a) data extraction from BIM models using IFC data models and b) Toolpath optimization. The data extraction module from IFC data models (i.e. partially developed for architectural elements) is used to divide all disciplines (architectural, structural, mechanical, and etc.) and provide the extracted information for toolpath generators. The current IFC data parser is developed for slicing the 3D architectural model into 2D layers that are given to toolpath generator for finding the optimum machine toolpath for printing the walls. The methodology can be presented using the chart represented in figure 4: Figure 4- Methodology and Framework Flow 20 | P a g e Required components to develop this framework for integrating BIM and automated construction system are summarized here: 1- Providing new Parameters in BIM Models that are required for automated construction process planning 2- IFC data extraction from BIM models and retrieving required information 3- Providing unique 2D graphs resulted from slicing 3D model based on IFC geometry data along with other parameters (such as material setting times) from extracted model data and automated construction tools 4- Generating optimal toolpath for each unique 2D graph In the first section of the proposed research the potential of inserting new parameters to current BIM models that contain key information related to automated construction process will be presented. The next step is to overpass the issues of interoperability requirements between BIM design platforms and automated construction robotic tools by using IFC models for data extraction. This framework will connect these two sides by automating the data extraction and analysis functions from BIM models using proper algorithms to generate structure layout graphs needed for robot toolpath optimization tools. Once the optimum tool path is provided for the whole structure it is translated to numerical control (NC) programming language mainly known as G-code for controlling the robots utilized in the automated construction process. Finally, the estimated total time required for the automated construction is calculated for the next part of the research related to material procurement and site logistics planning. 21 | P a g e Figure 5 shows different steps considered in the methodology to extract essential information from BIM models required as inputs for process planning of an automated construction system. The geometry data is converted to a set of 2D graphs representing the 3D model and finally a TSP solver algorithm is used to find the optimal path for each graph. These steps can be represented as follows: Figure 5 - Sequence of Steps for Extracting Essential Data from BIM Models 6.1.1.1 Extracting BIM Model Data using IFC Data Model The Industry Foundation Classes (IFC) is a data model developed by buildingSMART, an international organization known as International Alliance for Interoperability(IAI) formerly, which facilitates the data exchange interoperability between various software platforms in the construction industry such as Revit, Navisworks, AutoCAD. IFC data schema is developed to overcome the issues of data interoperability between different software applications that are utilized in design, construction, procurement, maintenance and operations phases of a BIM Model Preparation IFC Data Parsing (Geometry and Material Data) 3D Model Slicing into 2D Graphs Exporting Layer Graphs in JSON format 22 | P a g e construction project. This data schema similarly handles data exchanges for other software such as 3DMax, SketchUp, SolidWorks that are frequently used in 3D Printing industry. All these software applications are able to export the designed models to IFC format. IFC schema having object oriented data structure, is used in AEC industry and regularly used for collaboration between different BIM applications. Data interoperability ensures that all disciplines will work efficiently together with the same information (exchanged using IFC data schema) representing the designed model. The benefits of using this approach to communicate between different software applications are significant reduction in budget and schedule issues along with probable errors resulted from lack of data interoperability. The data dictionary in IFC files makes the data exchange conceivable between different software applications. In Contour Crafting process or any other automated construction system, there is a need for data schema from which required information related to the geometry and materials information of the designed structures can be extracted. BIM Models represented using IFC data models can be used to retrieve various model characteristics such as materials and geometry data. Following the information extraction from model, the data is then transformed to a graph or set of graphs provided in appropriate format such as JSON for a toolpath optimizer system to generate the optimum toolpath for the robot. This can be turned to a graph for hybrid 3D Printing system, which means several robots work together to construct the building with all disciplines included such as structural, mechanical, electrical and plumbing. 23 | P a g e BIM models have a significant potential to expand for automated constructions systems by inserting new data and parameters into models such as material setting time which is important for the overall planning of an automated construction system. However, rather than the necessity of inserting new data fields in BIM models, there is still a lack of data interoperability between automated construction process using robotic approaches and BIM software. BIM models are only used for regular conventional design and construction process while they have the potential to be used for automated construction using robotic approaches. This study is aiming to cover this gap in the first step and utilize the advancements in BIM industry for automated construction. To extract the data from BIM models, different methods are presented by several researchers and also the software companies such as Autodesk. One of the most convenient and successful approaches to read BIM model data is to use of Industry Foundation Classes (IFC) data models. BIM models can be converted into IFC data which is a text based structure representing all data nested in a model such as geometry, material type and hierarchy of elements along with project metadata such as IFC version and model center location. Existing parameters in BIM platforms such as Revit are related to conventional construction systems. New parameters can facilitate the overall process planning in automated construction if they get inserted into BIM models. As an instance, the waiting time for each layer in additive manufacturing is significantly important to measure the overall process duration. This data for each material can be added in model and hence, it can be easily extracted from IFC data without requiring any 24 | P a g e further action for material data extraction using other external database systems. On the other hand, when a structure is designed in BIM platforms such as Revit with the goal of being constructed by automated systems, the designer can easily observe various information about its constructability, total duration and other parameters if such a tool exists to analyze the model and provide that information. Finally, an IFC data parser is developed which is capable of extracting required information (such as geometry and materials) and slicing 3D model into 2D graphs. Since going through all the details of IFC parsing part is out of the main scope of this research, more details regarding IFC data parsing, type of IFC model used for this research and other elements are provided in the appendix section. 6.1.1.2 Toolpath Optimization The tool-path optimization problem is generally considered as the NP-hard combinatorial optimization problem. A toolpath optimization system is an important element in Contour Crafting process since it can considerably reduce the total nozzle travel time by generating optimum paths. This system should be able to satisfy all different requirements in the process while minimizing the construction time of each structure. A former research study for Contour Crafting toolpath optimization system is presented in [11,12]. This study is performed by a former CC lab student (Jing Zheng) and provide a solution for printing architectural elements (exterior and interior walls). Several algorithms such as Chine Postman Problem (CPP), Nearest Point (NP), and Travel Salesman Problem (TSP) are compared and TSP shows better performance in terms of reaching optimal solution among all the other algorithms. Hence, the algorithm selected for toolpath optimization part of this study is TSP. 25 | P a g e The raw graphs generated by data extraction from IFC data models should be converted into new graphs compatible with standard structure required in TSP optimization system that is proposed for Contour Crafting. All stages for converting the raw graphs to problem standard structure are provided in the appendix. Raw 2D graphs should be converted to new graphs to satisfy all different requirements existing in the system. The whole 3D model is now sliced into several layers by dividing the maximum height to extruding layer thickness. Total Number of Layers = L 𝑇 = 𝑀𝑎𝑥 (𝐻 ) 𝐸𝑥𝑡𝑟𝑢𝑠𝑖𝑜𝑛 𝑇 ℎ𝑖𝑐𝑘𝑛 𝑒𝑠𝑠 (𝑑 ℎ) In these graphs, Intersections are vertices while walls are edges. Tool Path optimization is applied to each layer separately. The raw graphs from parsed IFC files only include deposition edges, derived from physical objects such as walls in the model that should be printed. However, in the final converted graph there should be two types of edges: a) deposition edges and b) idle edges. Idle edges are those edges that the nozzle will travel without printing any object. The following example is considered and discussed to show how idle edges are calculated. 26 | P a g e Figure 6 - 1-Bedroom Apartment Floor Plan Layout A one-bedroom apartment is designed and presented as a case study to show how different parts in this research are integrated and finally provide required commands so that the presented structure can be constructed using Contour Crafting system. The 3D model is prepared in Revit 2017 and exported as IFC file. Figure 7 - 3D Model Designed in Revit 2017 There are several doors and window used in the model which makes the 2D layers non-identical to each other. After parsing the model IFC file, the following graph is one of the generated raw graphs from model: 27 | P a g e Figure 8 - 2D Graph Model extracted using IFC Parser Walls, doors and window lengths are as follow: Figure 9 - Dimensions of the first layer including openings In JSON format, this graph is represented as below: 28 | P a g e Figure 10 - JSON File summary extracted using IFC Parser The JSON file, i.e. summarized here, can be divided into 3 major parts: 1- Nodes/Vertices: Showing the intersections or cut points in the graph (windows, doors, etc.) 2- Edges: These are only deposition edges referring to walls in the raw graph 3- Material Ratios: These are derived from the parameters inserted in the model and filled based on the specific material mixture selected during the design process There are currently 20 nodes and 18 edges existing in the raw graph. This raw graph should be transferred into the new graph that can be used as input for toolpath optimization using TSP approach. As mentioned before, all steps for converting the raw graphs to problem standard structure are provided in the appendix. TSP algorithm which is discussed in the next section is selected for solving the toolpath optimization problem in this research. In this algorithm, the objective function is to minimize the final cost function that is the sum of all existing edges in 29 | P a g e the solution. Since in TSP algorithm there is no guaranty that any specific edge exists in the solution, there should be an additional effort to make sure that certain edges (deposition edges) be considered in the final solution. Selection of edges is only based on the cost function and also the fact that the solution should traverse all the nodes one and only one time. The only way to guarantee that the deposition edges are included in the solution is to assign large enough negative costs so that their existence force the cost function to be minimized. The absolute value of the negative cost should be large enough so that the optimization system definitely considers it in the final solution. The following formula is used to change the deposition edges’ costs before passing the graph to tool-path optimization system: 𝐶𝑜𝑠𝑡 𝑜𝑓 𝑑𝑒𝑝𝑜𝑠𝑖𝑡𝑖𝑜𝑛 𝑒𝑑𝑔𝑒 𝑏𝑒𝑡𝑤𝑒𝑒𝑛 𝑛𝑜𝑑𝑒 𝑖 𝑎𝑛𝑑 𝑗 = 𝐶 (𝑖 , 𝑗 ) = − (𝑀𝑎𝑥 (𝑅𝑜𝑡𝑎𝑡𝑖𝑜𝑛 𝐶𝑜𝑠𝑡𝑠 , 𝐼𝑑𝑙𝑒 𝐶𝑜𝑠𝑡𝑠 ) ∗ 𝑛𝑢𝑚𝑏𝑒𝑟 𝑜𝑓 𝑖𝑑𝑙𝑒 𝑒 𝑑 𝑔𝑒𝑠 ) = 𝐶 𝐷 Equation 1 For any other edges, the cost will remain as the idle cost calculated based on either rotation costs between artificial nodes or Euclidean distance cost effected by the µ factor defined previously related to the speed in x and y directions. Using the concepts described in appendix for rotations, idle edges and deposition edges, the problem can be modeled as a symmetric TSP where are there 2𝐸 𝑇 nodes in total in the system where 𝐸 𝑇 is the total number of deposition edges existing in the raw graph. This is because at each intersection, 𝑛 − 1 new artificial nodes are added to the system (where 𝑛 is equal to the number of walls at each intersection), so every deposition edge will have exactly two specific nodes in the new graph. Following the conversion steps, the raw graph shown above will be transferred to the following graph: 30 | P a g e Figure 11 - 2D Layer Layout Including both Artificial and Non-Artificial Nodes The nodes in red are the artificial nodes added at each intersection based on the number of existing walls. Idle edges are not shown here since from each node there is an edge to all other nodes except those nodes that form deposition edges. The cost of all deposition edges (i.e. shown in the graph) are equal to 𝐶 𝐷 which is a large negative number and the cost of idle edges 𝐶 𝑖𝑗 are calculated based on the Euclidean distance of the edge end nodes and the effect of robot speed ratio in x and y directions. 6.1.2 Results 6.1.2.1 BIM Model Design A case study is presented to further clarify some details of proposed framework using POCSAC software for different phases from design to robotic construction. A one-bedroom apartment is designed and presented in the case study to show how different parts in the presented framework work together and finally provide the 31 | P a g e required commands so that the intended structure can be constructed using the POCSAC software in a robotic construction scenario. The 3D model of a one-bedroom apartment is prepared in Autodesk Revit, as a commonly used design platform in AEC industry. The model includes the building shell and interior walls along with the electrical and mechanical systems to illustrate the integrated presentation of different disciplines within the same model as one of the BIM technology benefits. POCSAC is then able to detect and classify model objects and components according to their discipline. A specific robot family for layer by layer fabrication of the exterior and interior walls is provided and added to the model. This family consists of all parameters related to robot operations such as acceleration figures, linear deposition rate, rotation and nozzle traversal speeds between extrusion stop and start points. The robot family also provides the maximum foot print that can be built in a single run by the robot. There are several doors and windows used in the model which make the 2D layers of the walls non-identical to each other. The required construction material properties for wall construction are also included. The designed model is exported in IFC format and sent to POCSAC software for further analysis. Figure 12 shows the 3D model where ceiling components are hidden to make all interior architectural, mechanical, lighting fixtures and electrical components visible. The slab under the building, the dimensions of which are dictated by the work envelope of the construction machine, demonstrates the maximum dimensions which could be printed in a single run by the robot. 32 | P a g e Figure 12 - 3D Model Designed in Revit (ceiling is hided) 6.1.2.2 Data Extraction and Analysis in POCSAC Software After parsing the IFC model, exterior and interior walls information are extracted by POCSAC. The 3D model is sliced into 2D layers based on the provided layer thickness. Figure 13-a represents the extracted part of the model following one of the generated preliminary graphs from the model shown in Figure 13-b. Walls, doors and windows dimensions are presented in Figure13-c. (a) 33 | P a g e (b) (c) Figure 13 (a) Extracted Exterior and Interior Walls from the IFC Model (b) Sample Preliminary Graph Nodes (c) Graph Dimensions There are 20 nodes and 18 edges in the preliminary graph. This graph should be transferred into a new graph that can be used as input for toolpath optimization using TSP approach. Following the conversion steps, the preliminary graph shown above will be transferred to the following new graph with new vertex indices added as artificial nodes. After adding artificial nodes (shown in red), there would be 36 nodes in total and previous node numbers are changed accordingly. The new graph is presented in Figure 14. Figure 14 - New Graph Generated in POCSAC with Updated Node Indices Idle edges are not shown here since from each node there is an edge to all other nodes, except for nodes that form deposition edges. The cost of all deposition edges (i.e. shown in the graph) are equal to C_D which is a large negative number and the 34 | P a g e cost of idle edges Cij are calculated based on the Euclidean distance of the edge end nodes and the effect of robot speed ratio in x and y directions. Next, the graph is passed to the toolpath optimization module in the POCSAC software to generate the optimum toolpath and also minimum/maximum setting times of the layers to be used for laboratory testing and finalizing material mixture. The optimum toolpath output is representing a cycle which is a sequence of all the nodes in the new graphs starting from node 1 (usually the bottom-left corner of the building) and ending in node 1 with minimum cost while including all deposition edges that are required to be traversed and printed. The sequence of the optimum path for the test case, starting from node 1 and completing the cycle by going back to node 1 at the end, is shown in Table 1. Table 1 - Sequence of the Nodes in the Optimal Solution Toolpath N1 N2 N3 N4 N5 N6 N7 N8 N9 1 2 3 32 31 30 28 26 4 N10 N11 N12 N13 N14 N15 N16 N17 N18 5 6 7 8 9 36 35 34 33 N19 N20 N21 N22 N23 N24 N25 N26 N27 10 11 12 13 14 15 16 18 17 N28 N29 N30 N31 N32 N33 N34 N35 N36 N37 19 22 20 21 23 24 25 27 29 1 Table 2 shows the total counts of each edge category (deposition, idle and rotation) and total distance calculated for each category. Since deposition edges are in common for all different solutions and do not have any contribution for selecting the optimum toolpath, total summation of idle and rotation edges is then calculated. 35 | P a g e Table 2 - Total Edge Counts and Distances Analysis based on The Sequence of Optimal Solution Edge Category Total Counts Total Distance Deposition 18 169 Idle 6 41.7 Rotation 11 22 Radian 6.1.2.3 Automated Construction using Contour Crafting The final G-Code for all the layers is generated using the pseudocode shown in Figure 39 represented in appendices section. The finalized material mixture data is inserted into POCSAC for controlling the material dosing system. POCSAC controls both the robot and the material preparation unit during the automated construction process. Deposition, idle and rotation speed are also extracted from the BIM IFC model based on the provided robot family which includes all these parameters. Finally, for the test case presented in this section using POCSAC and Contour Crafting system, a scaled model (1 foot to 1 inch) is printed using the prototype CC machine and is shown in Figure 15. The prototype CC machine has work envelope dimensions of 5 m x 8 m x 3 m, corresponding to a 120 m2 printing zone [39]. 36 | P a g e Figure 15 - Printed Case Study Model using Contour Crafting Machine A layer by layer comprehensive automated construction report is also generated by POCSAC from all recorded parameters observed during the construction process. This report is added to the IFC model for future learning purposes. The test case studied in this paper briefly demonstrates the application of the BIM-Integrated framework including POCSAC software to connect the BIM technology with an automated robotic construction system such as Contour Crafting. 6.1.3 Conclusion Construction-scale 3D printing is still in its infancy and there are various related engineering challenges yet to be solved. For instance, inspection and quality control during construction is a critical aspect of automated construction which has not yet been investigated by researchers. Due to the absence of any published data on a comprehensive quality control system for automated construction, the quality aspect was not included and discussed within the proposed framework. As such, lack of connection with inspection and quality control system remains as one of the current limitations of POCSAC software. However, it seems that depending on the details of 37 | P a g e such quality control system and types of parameters and sensors used, quality aspects could be combined with proposed BIM-integrated framework. Such integration will make it possible to detect any unexpected problem occurring during the construction process and to include it in construction documents. Some parameters which could possibly be monitored for measuring quality of construction include rheological parameters of fresh printing concrete and early age compressive strength of concrete. Moreover, RGB depth cameras such as Kinect could be adopted to check the printed layers during the 3D printing process to make sure the structure is properly printed as the process continues. With regard to construction materials, automated material preparation offers new possibilities to concrete construction. Based on project needs, the properties of material which is deposited by the robot could be accurately controlled at any time and vary throughout the project. Some of the properties which could be engineered and variable in different parts of a structure include concrete density, compressive strength, thermal conductivity, and color. Also, considering the fact that temperature and humidity affects fresh properties of concrete including setting time, changes in mixture design could be automatically applied based on real-time local temperature and humidity measurements. Other limitations of the current POCSAC version are related to optimization algorithms for toolpath generation of robots. With respect to toolpath optimization system adopted in POCSAC software, a more comprehensive system could be developed. An automated construction system such as Contour Crafting has the ultimate goal of constructing a building including all different disciplines such as 38 | P a g e mechanical and electrical components. For the toolpath optimization part related to placing other components such as mechanical or electrical equipment during the automated construction process, algorithms such as time window TSP can be considered. This algorithm is used when visiting a node should happen in a specific predetermined time interval. The time of putting these components should take place in a precise time interval which is related to the time required for printing the layers supporting this component. POCSAC software is currently designed for specific robotic control system used in Contour Crafting process. This fact brings some limitations for use of the software for other large-scale 3D printing systems which have their own requirements and constraints. The commands that are generated from the software are programmed specifically for a certain type of robot. In the future, a tool for getting user inputs related to understandable command for their specific robotic control system can be developed to generate the final commands accordingly. This feature will eliminate the need to change hardcoded commands to be able to use it for another robotic control system. Finally, in addition to robotic control, another limitation is related to material procurement and site layout planning using required facilities for an automated construction process that are not integrated with current POCSAC software. To assure the feasibility of the automated construction in a short time, POCSAC should be able to generate the optimal construction process including site logistics and material procurement strategy using the previous generated information. Automated construction systems using Contour Crafting or other robotic approaches promise 39 | P a g e rapid construction that could take as little as one day to complete an average size single family residential building. This means that the rate of flow of materials to the site is considerably higher than a conventional construction process due to high material consumption in a relatively short time. In automated construction systems, inefficiency in material procurement and storage scarcity due to inappropriate site logistics planning can result in major problems such as longer total construction time, and formation of cold joint between successive deposited layers. POCSAC software has the potential to cover site logistics and material procurement planning of automated construction process as well as other parts using related BIM model data. It should be noted that although a small-scale case study is presented here, evaluation of such comprehensive framework requires its implementation in full-scale automated building construction. This would help the researchers to realize the feasibility of implementing proposed elements in this platform and also to understand other challenges and issues to be addressed in future studies. BIM has been advantageous for the AEC industry and has resulted in improved efficiency of many conventional construction projects. However, details of BIM implementation for automated construction has not yet been investigated in any appreciable depth. Considering the new capabilities and features to be provided by a well-developed automated construction system, integration of these two technologies, namely, BIM and automated construction, is beneficial. In this paper, a framework is proposed to integrate the BIM technology into automated building construction by Contour Crafting. 40 | P a g e The proposed framework provides resolutions for interoperation between different components of an automated construction system and the BIM-integrated software platform such that the maximum benefit is realized through synergy of the two technologies. Within the proposed framework, related data are extracted from initial BIM model and then are analyzed so that proper commands for operations and robotic control are generated. The POCSAC software platform is the major enabler for a seamless integration of BIM and Contour Crafting. Furthermore, data analysis using IFC models are the key linking point to resolve data interoperability issues between BIM and automated construction systems. Inclusion of some important information in BIM models related to automated construction, such as robot families and new material parameters, are also proposed in this study. Such modifications will result in a BIM model which also serve as a centralized database for automated construction project. Finally, the concept of automated report generation during the automated construction process is described. The report is later added to the model and would be quite valuable and beneficial as a future reference. As an instance, various reasons of system failures that are added to the report can be used by engineers to modify the construction system accordingly to prevent recurrence of such issues. Integrating BIM technology with Automated Construction systems can be beneficial in many regards. BIM has a lot of advancements in AEC industry which should not be ignored when automated construction is used. IFC parsers are the key linking point to resolve data interoperability issues between BIM and automated construction systems using robotic approaches. The specific IFC parser which is 41 | P a g e developed in this research is able to properly segment a 3D model into 2D graphs. Different parameters that are included in the IFC parser knowledge base help the system to extract required information used in TSP solver and construction planning such as components’ geometries and material details. IFC parsing system can be improved to be able to segment more complex structures intro constructible layers using robotic approaches. A final G-code generating algorithm is required to properly translate the solutions from toolpath generating system into a language that robots can understand and use to construct the building exactly compatible with the designed BIM model. 6.2 Material Procurement and Site Logistics Planning for Automated Construction System In this part of the proposal, a proper material procurement and site logistics planning for automated construction using robotic approaches is proposed. In the proposed study the material resources that are related to concrete structure are considered to be studied. The proposed model is capable of integrating material procurement variables with dynamic site layout planning variables concurrently to optimize final automated construction planning decisions. In each construction project there is a critical tradeoff between ordering in small amounts, in which material scarcity probability and project delays are increased, and on the other side ordering in large quantities where inventory costs and required storage spaces are increased [20]. In automated construction using CC technology more constraints and considerations in terms of time are applied to this problem due to the tight and critical schedule. The other important point to consider in rapid 42 | P a g e automated construction is the relation between material procurement and material storage space based on site layout planning. Although proper material procurement process can guarantee the delivery of all needed materials on the site but scarcity of site space can totally make this plan impractical. The methodology used in CPP-CC model is described in the next section. 6.2.1 Methodology The proposed model is named Construction Process Planning for Contour Crafting (CPP-CC) which considers material procurement and dynamic site layout planning variables simultaneously. The flow chart below (figure 16) shows the methodology system proposed for material procurement and site logistics planning for automated construction. Figure 16 - Components of CPP-CC Model In CPP-CC model, space availabilities are extracted directly from the IFC model along with material ratios and total demands as described in previous sections. The BIM model, in which site boundaries are existing, is parsed and the space availabilities are extracted. Material Procurement variables are optimized and 43 | P a g e required space for each material is calculated. All these spaces requirements are then checked with space availabilities on the construction site to make sure space constraints are satisfied. In case there are not enough space on the site, for those materials that need more space, material procurement optimization system will be used again to provide another solution. The cycle continues till material procurement variables satisfy site constraints. Once all material procurement variables are finalized and site requirements are set, site logistics planning optimization system is used to find the best facilities’ layout on the construction site. Material procurement variables and layout planning variables are described in the next sections. 6.2.1.1 Material Procurement Variables The variables that are explored in this part are demanding ratio of all required materials in the contour crafting process and also the ordering quantity and time periods for each specific material independently. The demand of each material is directly related to the automated construction process and the sequence of printing. CC machine starts printing the building and if all the required materials (water, cement, aggregates, etc.) are replenished properly, it is able to finish the project in the optimum time calculated in tool path optimization system. Overall wall sizes and total quantities are directly extracted from IFC model. Geometry and material data is extracted as explained before in tool path optimization part. Considering the same test case described in previous chapter, table 3 represents different wall sizes in the model along with total concrete volume required to build each component. 44 | P a g e Table 3 - Wall Sizes and Material Quantities Wall Schedule Family and Type Width Length Area (SF) Volume (CF) Basic Wall: Retaining - 12" Concrete 1' - 0" 45' - 0" 460 460 Basic Wall: Retaining - 12" Concrete 1' - 0" 20' - 0" 200 200 Basic Wall: Retaining - 12" Concrete 1' - 0" 45' - 0" 429 429 Basic Wall: Retaining - 12" Concrete 1' - 0" 20' - 0" 172 172 Basic Wall: Interior - 4 7/8" 0' - 4 7/8" 11' - 6" 112 45.51 Basic Wall: Interior - 4 7/8" 0' - 4 7/8" 20' - 6" 177 71.89 Basic Wall: Interior - 4 7/8" 0' - 4 7/8" 8' - 6" 57 23.14 Basic Wall: Interior - 4 7/8" 0' - 4 7/8" 3' - 0" 15 6.09 Basic Wall: Interior - 4 7/8" 0' - 4 7/8" 3' - 0" 15 6.09 Total concrete required for constructing the whole structure is approximately 53 cubic yards. The material consumption can be analyzed in two ways: 1- Based on the tool path sequence generated in the previous chapter 2- Assuming constant consumption ratio during the printing process Since there are few idle edges in the overall sequence which require few seconds to be traversed each, assuming constant consumption ratio during the printing process is considered as a precise estimation. CC Machine has a constant printing flow rate during the whole process for decomposition edges. Total length of all edges (including idle, decomposition and rotation edges) is 191 feet based on the calculations mentioned in chapter 2. Assuming a constant printing speed of 1.0 (ft/s), the total time required to print one layer is 191 seconds. There are 120 layers in total for a 10ft height wall, the final time required to print the whole structure would be as follow: 45 | P a g e Equation 2 𝑇 𝑡 𝑜 𝑡𝑎𝑙 = 𝑇 𝐿𝑎𝑦𝑒𝑟 ∗ 𝑁𝑢𝑚𝑏𝑒𝑟 𝑜𝑓 𝐿𝑎𝑦𝑒𝑟𝑠 = 191 ∗ 120 = 22920 𝑠 = 6.36 ℎ𝑟𝑠 This means that the automated construction process is finished in almost a quarter of a day. Using the weight ratios extracted from IFC model file, the material consumption rates with constant rate assumption are presented in table 4 as follow: Table 4 - Material Consumption Ratios during Automated Construction Process Material Weight Ratio (%) Consumption Ratio (Kg/minute) Chemical Mixtures 5 2.36 Water 10 4.7 Cement 25 11.76 Fine Aggregate 60 28.23 Now that the demand measurements are performed for each individual material that is utilized during the construction process, ordering quantities and time slots should be determined. Once they are fixed, site logistics planning system can find the optimal facilities layout on the construction site. Two factors that should be considered here are the space availability on construction site and order quantities of materials. In conventional construction systems, the procurement process is usually planned without considering site logistics constraints since the process is going to be accomplished in an ample time. Order quantities are usually low enough so that the site spaces are able to handle all different material orders during the construction process. Since there is a high volume of material flow into the site during the automated construction process that could take as little as 46 | P a g e one day to be finished, integrating site logistics and material procurement variables is a vital task for planning an automated construction process. 6.2.1.2 Site Logistics Variables Before describing each optimization system, it is necessary to describe different facilities considered for the automated construction process. This is needed since facilities to store materials are not the only facilities needed during the construction process. These facilities should be defined to be deducted from the overall space availability on the construction site. Space availability is a key factor to determine if the material procurement variables that are calculated are compatible with the site space constraints. In a study by Li et al. [21], a list of facilities required for a conventional construction site are presented as: 1- Site office 2- Falsework workshop 3- Labor residence 4- Storeroom 5- Carpentry workshop 6- Reinforcement steel workshop 7- Side and main gates 8- Utilities control room including electrical and water 9- Concrete batch workshop For Contour Crafting, several facilities from this list are ignored due to the automation nature of the CC machine and minimum human involvement in the construction process. It is assumed that workshops for falsework, carpentry, and 47 | P a g e reinforcement steel are not required in CC project. Instead of concrete batch workshop, several silos for raw concrete materials should be located on the site. Labor residence facility is not required for CC construction process. One storeroom can be located on the site for storing maintenance equipment of the CC machine in case any problem occurred during the construction process. Control room for electrical and water utilities can be replaced with a single room for placement of generators providing electrical power for the CC machine. There should also be a water tank located on the site near continuous mixture to provide required water for concrete mixture. Based on the given explanations, the following facilities are considered essential for contour crafting construction project: Facility 1: Material Silos and Storing Space Facility 2: Water tank (Locating this facility close to the continuous mixer is advantageous, though it is not necessary) Facility 3: Maintenance Equipment Space for CC Machine Facility 4: Electrical Power Equipment Space Facility 5: Site Trailer (This facility is not necessarily required in automated construction process. However, it can exist if there are several projects on the site being constructed together. For a single construction project that took a very short time, it is assumed that the office trailer is not required.) Facility 6: Side and Main gates The size of first two facilities related to material spaces and water tank are determined from material procurement optimization system. The sizes of other facilities are predetermined and defined for each project. For the test case study 48 | P a g e used in this proposal, it is assumed that there is no need for site trailer. There are two gates at the north and west sides of the construction site for vehicles access. Other facility sizes are represented in table 5. Table 5 - Automated Construction Process Facilities Size and Type Facility Size Type Maintenance Equipment 5' x 5' Movable Electrical Power Equipment 8' x 8' Movable North Gate 76'-0" Fixed West Gate 61'-0" Fixed Maintenance equipment and electrical power equipment facilities are movable which means they can be located in any empty space available on the construction site. The north and the west gates are fixed since their location is already determined on the north and the west sides of the site. Figure 17 shows the overall sketch of the construction site used in the test case study. 49 | P a g e Figure 17 - Construction Site Overall Layout 6.2.1.3 Material Procurement Optimization System In this section, the optimization problem, variables, constraints and the objective function are defined. There are two variables for each material to be determined in this parts: a) Order Quantity and b) Ordering Times. For each material used during the construction process, these two variables should be determined to be able to estimate the required space on the construction site that should be assigned to be able to handle the maximum flow of each individual material. The maximum order quantity determines the minimum space required on the construction site. In order to solve this problem, a deterministic inventory system can be adopted. This system finds the answers to both questions for material procurement process; the 50 | P a g e amount of each material to order and the proper time lags to order. Below are the variables required for developing a deterministic inventory model [22 and 23]: 1. Cost of Ordering (CO): This includes the delivery costs and material costs. Different vehicles and different number of orders result in a different cost of ordering. However, since the total cost of materials in constant, it can be omitted from the optimization objective function. Small-order quantities will increase the number of delivery times and hence cost of ordering will be increased as well. However, low number of orders requires more space on the site since the amount of materials that are delivered each time are enlarged. 2. Cost of Space (CS): For each material, based on the amount of the order, there should be enough space available on the site. Space scarcity will increase this cost to infinity. Moreover, for some materials such as silos for aggregates or water tank for water, more space requires bigger silo or water tank size so the cost of space will be increased. So there is always a tradeoff between cost of ordering and the cost of space which should be optimized here. 3. Holding Cost (HC): This cost is representing the cost of warehousing over the period of time, which would be zero in contour crafting project since the project is estimated to be finished in a very short period of time (e.g. one day). 4. Shortage Cost (SC): In Contour Crafting project, due to the compact schedule and short waiting time between layers, shortage of materials is not allowed in the system. It should also be noticed that material shortage during the automated construction process has a tremendous cost for the whole system. The final extruded layer will be over-solidified and next layer cannot be printed on it, or concrete might get stock in the nozzle system because of the delay in material delivery to the site. To avoid such problems and provide the ability in the system to handle material shortages, ordering 51 | P a g e times are planned in a way to handle up to one-hour delay in material deliveries. It means that there should always be enough materials on the site to be able to continue the project for at least one hour. This means that the inventory level should always be more than or equal to: Equation 3 a. 𝐼𝑛𝑣𝑒𝑛𝑡𝑜𝑟𝑦 𝐿𝑒𝑣𝑒𝑙 𝑚𝑖𝑛 = 1 ℎ𝑟 ∗ 𝑎 (𝑚𝑎𝑡𝑒𝑟𝑖𝑎𝑙 𝑐𝑜𝑛𝑠𝑢𝑚𝑝𝑡𝑖𝑜𝑛 𝑟𝑎𝑡𝑖𝑜 / ℎ𝑟 ) However, the allowed delay time can be set differently based on the project. Higher times require more frequent orders, but the process suffers from lower level of risk for material shortages. A deterministic inventory system that can avoid material shortage with a specific minimum inventory level is shown in figure 18 which is also used in the proposed framework. Figure 18 - Deterministic Inventory System with Minimum Inventory Level used in CPP-CC Model In figure 18, 𝑄 𝑖 represents the size of each ordering while the horizontal axis represents the time lags between each order. The total demand value at 𝑇 𝑡𝑜𝑡𝑎𝑙 , i.e. 52 | P a g e total consumption of each material, should be equal to the total summation of each material ordering 𝑄 𝑖 values. 5. Salvage Value (SV): This item is the cost of leftover materials when there is no further need for them. In order to simplify the deterministic inventory model for CC project, it is assumed that the calculated amount of materials is not exceeding the actual required materials for the construction. Hence, the salvage value can be assumed as zero. 6. Discount Rate (DR): As it is usual in all markets, ordering more amount of an item will result in a lower unit price. However, in contour crafting project, the ordering amounts of construction materials are all within short time as little as one day, thus there is no need to use this variable in this model and the unit prices are assumed to be constant. All the materials are ordered together prior to the start of construction process. Based on the cost factors described above, the only two factors that are considered in the cost function of the optimization algorithm is the first two factors that are cost of ordering and space. The trade-off between these two factors are shown in figure 19 as follow: 53 | P a g e Figure 19 - Total Cost tradeoff between Cost of Ordering and Cost of Space All variables in this problem are non-negative integers and they are selected from a finite set. Available material delivery vehicles have specific sizes and costs. Order quantities should be selected within this range. The optimization cost will force the order quantities to be as close as possible to the maximum quantity that a vehicle can deliver. Solving discrete optimization problem finding the best solution among a finite set of feasible values is categorized as combinatorial optimization problem. The mathematical representation of the optimization problem discussed in this section is presented here. Equation 4 𝑔𝑜𝑎𝑙 : 𝑎𝑟𝑔𝑚𝑖𝑛 ∑ ∑ 𝐷𝐶𝐹 𝑚 (𝑄 𝑚 ,𝑡 ) 𝑇 𝑡𝑜𝑡𝑎𝑙 𝑡 =0 𝑀 𝑚 =1 + ∑ 𝑆𝐶𝐹 𝑚 (max (𝑄 𝑚 ,0 , 𝑄 𝑚 ,1 , … , 𝑄 𝑚 ,𝑇 𝑡𝑜𝑡𝑎𝑙 )) 𝑀 𝑚 =1 𝑠𝑢𝑏𝑗𝑒𝑐𝑡𝑒𝑑 𝑡𝑜 : 𝑄 𝑚 ,𝑡 ∈ [0, (𝑇𝐷 ) 𝑚 ] ∑ 𝑄 𝑚 ,𝑡 𝑇 𝑡𝑜𝑡𝑎𝑙 𝑡 =0 = (𝑇𝐷 ) 𝑚 54 | P a g e 𝐷𝐶𝐹 𝑚 (𝑄 𝑚 ,𝑡 ) = { 𝐷𝐶 1 , 𝑄 𝑚 ,𝑡 ≤ 𝑞 1 𝐷𝐶 2 , 𝑞 1 < 𝑄 𝑚 ,𝑡 ≤ 𝑞 2 … 𝑆𝐶𝐹 𝑚 (max (𝑄 𝑚 ,0 , … , 𝑄 𝑚 ,𝑇 𝑡𝑜𝑡𝑎𝑙 )) = { 𝑆𝐶 1 , max (𝑄 𝑚 ,0 , 𝑄 𝑚 ,1 , … , 𝑄 𝑚 ,𝑇 𝑡𝑜𝑡𝑎𝑙 ) ≤ 𝑑 1 𝑆𝐶 2 , 𝑑 1 < max (𝑄 𝑚 ,0 , 𝑄 𝑚 ,1 , … , 𝑄 𝑚 ,𝑇 𝑡𝑜𝑡𝑎𝑙 ) ≤ 𝑑 2 … 𝐷𝑒𝑓𝑖𝑛𝑖𝑡𝑖𝑜𝑛𝑠 : 𝑄 𝑚 ,𝑡 : 𝑉𝑎𝑟𝑖𝑎𝑏𝑙𝑒𝑠 ∶ 𝑂𝑟𝑑𝑒𝑟 𝑄𝑢𝑎𝑛𝑡𝑖𝑡 𝑦 𝑜𝑓 𝑀𝑎𝑡𝑒𝑟𝑖𝑎𝑙 𝑚 𝑎𝑡 𝑡𝑖𝑚𝑒 𝑡 𝐷𝐶𝐹 𝑚 ∶ 𝐷𝑒𝑙𝑖𝑣𝑒𝑟𝑦 𝐶𝑜𝑠𝑡 𝐹𝑢𝑛𝑐𝑡𝑖𝑜𝑛 𝑜𝑓 𝑀𝑎𝑡𝑒𝑟𝑖𝑎𝑙 𝑚 − 𝐵𝑎𝑠𝑒𝑑 𝑜𝑛 𝑡 ℎ𝑒 𝑜𝑟𝑑𝑒𝑟 𝑞𝑢𝑎𝑛𝑡𝑖𝑡𝑦 , 𝑐 ℎ𝑜𝑜𝑠𝑒𝑠 𝑡 ℎ𝑒 𝑚𝑖𝑛𝑖𝑚𝑢𝑚 𝑐𝑜𝑠𝑡 𝑜𝑓 𝑑𝑒𝑙𝑖𝑣𝑒𝑟𝑦 𝑣𝑒 ℎ𝑖𝑐𝑙𝑒 𝑆𝐶𝐹 𝑚 ∶ 𝑆𝑝𝑎𝑐𝑒 𝐶𝑜𝑠𝑡 𝐹𝑢𝑛𝑐𝑡𝑖𝑜𝑛 𝑜𝑓 𝑀𝑎𝑡𝑒𝑟𝑖𝑎𝑙 𝑚 − 𝐵𝑎𝑠𝑒𝑑 𝑜𝑛 𝑡 ℎ𝑒 𝑚𝑎𝑥𝑖𝑚𝑢𝑚 𝑜𝑟𝑑𝑒𝑟 𝑞𝑢𝑎𝑛𝑡𝑖𝑡𝑦 , 𝑐 ℎ𝑜𝑜𝑠𝑒𝑠 𝑡 ℎ𝑒 𝑚𝑖𝑛𝑖𝑚𝑢𝑚 𝑐𝑜𝑠𝑡 𝑜𝑓 𝑠𝑡𝑜𝑟𝑎𝑔𝑒 (𝑇𝐷 ) 𝑚 ∶ 𝑇𝑜𝑡𝑎𝑙 𝑑𝑒𝑚𝑎𝑛𝑑 𝑞𝑢𝑎𝑛𝑡𝑖 𝑖 𝑡𝑦 𝑜𝑓 𝑚𝑎𝑡𝑒𝑟𝑖𝑎𝑙 𝑚 𝐷𝐶 𝑖 = 𝐷𝑒𝑙𝑖𝑣𝑒𝑟𝑦 𝑐𝑜𝑠𝑡 𝑜𝑢𝑡𝑝𝑢𝑡 𝑆𝐶 𝑖 = 𝑆𝑝𝑎𝑐𝑒 𝑐𝑜𝑠𝑡 𝑜𝑢𝑡𝑝𝑢𝑡 The problem is solved using linear integer programming package in MATLAB. The final solution provides all the variable quantities 𝑄 𝑚 ,𝑡 in a 𝑚 x 𝑡 matrix as follow: 𝒕 𝟎 𝒕 𝟏 𝒕 𝟐 … 𝒕 𝑻𝒐𝒕𝒂𝒍 𝒎 𝟏 𝑄 1,0 𝑄 1,1 𝑄 1,2 … 𝑄 1,𝑇 𝒎 𝟐 𝑄 2,0 𝑄 2,1 𝑄 2,2 … 𝑄 2,𝑇 𝒎 𝟑 𝑄 3,0 𝑄 3,1 𝑄 3,2 … 𝑄 3,𝑇 𝒎 𝟒 𝑄 4,0 𝑄 4,1 𝑄 4,2 … 𝑄 4,𝑇 … … … … … … 𝒎 𝑴 𝑄 𝑀 ,0 𝑄 𝑀 ,1 𝑄 𝑀 ,2 … 𝑄 𝑀 ,𝑇 Deterministic inventory system provides different answers for quantity of each material to be ordered in a specific time interval sequence. The solution from the 55 | P a g e deterministic inventory system is considered as the ordering policy during the construction process. 6.2.1.4 Site Logistics Optimization System In the previous section, ordering quantities and their times are calculated for each material. The maximum quantity for each material determines the space required at the construction site to be assigned for storage. As mentioned in the previous part, it is assumed that silos are used for cement and fine aggregates, and a water tank is used to keep waters. Additive chemical mixtures are also stored in a small 75 CF container which is able to keep total amount of chemical mixtures used in the test case project. Based on the information provided table 5, all the facilities for material storage or other purposes required for the automated construction process using Contour Crafting approach are finalized. All facilities determined, the next step is to position these facilities on the construction site. To do this, available locations on the construction site must be identified. It is obvious that these locations are case dependent and differ based on the site condition in each construction project. Among the all exiting methods for site layout planning problem mentioned in background study part, the generic algorithm is selected to be studied for this study. The advantage of genetic algorithm over other approaches is that it avoids getting stuck in a local optimum by choosing feasible random layouts each time for facilities [29]. In other words, genetic algorithm provides a high chance that the global optimal solution is found. This method is discussed later and its possible implementation for Contour Crafting project is provided. 56 | P a g e As mentioned before, possible locations in the site layout are selected and the next step is to define the constraints of the site layout based on the existing site conditions and requirements. Gate facilities are the ones that are predetermined based on the site access condition. Thus, their location is fixed during the solution finding process. The available locations on the test study site are chosen based on the required facilities and the available spaces on the construction site. Figure 20 shows the 13 predetermined locations considered for the test case project. Each facility should be finally assigned to one of the available locations on the construction site for site logistics planning. The red labeled locations are already filled with fixed facilities representing the Contour Crafting machine (the building area), and the two north and west gates. Figure 20 - Predetermined Spaces on The Construction Site Location indices and their sizes are represented in table 6 as follow: 57 | P a g e Table 6 - Size and Geometry Centers of Available Locations on the Construction Site Space Number Size Origins 𝑳 𝒙 𝑳 𝒚 𝑪 𝒙 𝑪 𝒚 1 10'-0" 10'-0" 10'-0" 52'-6" 2 16'-6" 15'-0" 27'-6" 50'-0" 3 16'-6" 15'-0" 48'-6" 50'-0" 4 10'-0" 10'-0" 66'-0" 52'-6" 5 10'-0" 15'-0" 69'-0" 30'-6" 6 10'-0" 10'-0" 66'-0" 8'-6" 7 16'-6" 15'-0" 48'-6" 11'-0" 8 16'-6" 15'-0" 27'-6" 11'-0" 9 10'-0" 10'-0" 10'-0" 8'-6" 10 10'-0" 15'-0" 7'-0" 30'-6" 11 45'-0" 20'-0" 38'-0" 30'-6" 12 76'-0" 2'-0" 38'-0" 61'-0" 13 2'-0" 61'-0" 0'-0" 30'-6" The next step is to create the travel cost matrix between all facilities. As an example, the power required to transfer raw materials to the continuous mixer or the power required to pump concrete mixture from continuous mixer to Contour Crafting machine is considered as travel cost between facilities. In other words, travel cost matrix provides the cost per foot ratios between all the facilities in the system. Table 7 shows the assumptions considered the travel cost matrix between the facilities required for automated construction process. Table 7 - Travel Cost Matrix between All Facilities Facilities* CCM CM G1 G2 ME EPE T1 S1 S2 S3 CCM 0 4p* 0 0 0 p/2 0 0 0 0 CM 4p 0 0 0 0 p/2 2p 2p 2p 2p G1 0 0 0 0 0 0 p/4 p/4 p/4 p/4 G2 0 0 0 0 0 0 p/4 p/4 p/4 p/4 58 | P a g e ME 0 0 0 0 0 0 0 0 0 0 EPE p/2 p/2 0 0 0 0 p/2 p/2 p/2 p/2 T1 0 2p p/4 p/4 0 p/2 0 0 0 0 S1 0 2p p/4 p/4 0 p/2 0 0 0 0 S2 0 2p p/4 p/4 0 p/2 0 0 0 0 S3 0 2p p/4 p/4 0 p/2 0 0 0 0 *p = cost/ft unit Facility* Index CC Machine CCM Continuous Mixer CM North Gate G1 West Gate G2 Maintenance Equipment ME Electrical Power Equipment EPE Water Tank T1 Cement Silo S1 Fine Aggregate Silo S2 Chemical Mixture Mini-Silo S3 Travel distances are calculated based on the center-to-center Euclidean distances between the locations that facilities are assigned. It is necessary to mention that travel distances are representing the required distances for different activities such as delivering the materials from silos to the continuous mixer. Since there are 10 different facilities and 13 available spaces on the site, 3 artificial facilities which has zero cost to all facilities are added to the system. It is explained later why this is required for solving the problem using genetic algorithm system. It should be considered that the travel distance and cost from facility x to facility y is the same as the travel distance and cost from facility y to facility x. Based on this fact, both cost and distance matrices are symmetric. 59 | P a g e Another requiring matrix for genetic algorithm is called the permutation matrix in which all the values of a row are zero except one that is equal to 1. Rows and the columns of this matrix are facilities and available locations accordingly. This matrix shows that each facility will be placed in one and only one location. The next part of the solution is to use different operators such as crossover adjusted edge recombination and mutation adjusted symmetric genes exchange [30]. The mathematical representation of the problem to be solved using genetic algorithm is as follow: Equation 5 argmin ∑ ∑ ∑ 𝛾 𝑥𝑖 𝑐 𝑥𝑖 𝑑 𝑖𝑗 𝑗 𝑥 𝑖 𝑠𝑢𝑏𝑗𝑒𝑐𝑡𝑒𝑑 𝑡𝑜 : 1 − ∑ 𝛾 𝑥𝑖 𝑥 = 1 𝑓𝑜𝑟 𝑖𝜖 {1,2, … , 𝑛 } 𝑤 ℎ𝑒𝑟𝑒 𝑛 𝑖𝑠 𝑡 ℎ𝑒 𝑡𝑜𝑡𝑎𝑙 𝑛𝑢𝑚𝑏𝑒𝑟 𝑜𝑓 𝑙𝑜𝑐𝑎𝑡𝑖𝑜𝑛𝑠 2 − 𝑖𝑓 𝛾 𝑥𝑖 = 1 (𝑆 (𝐹𝑎𝑐𝑖𝑙𝑖𝑡𝑦 𝑥 ) ≤ 𝑆 (𝐿𝑜𝑐𝑎𝑡𝑖𝑜 𝑛 𝑖 )) Notations and definitions: 𝑐 𝑥𝑖 = 𝑐𝑜𝑠𝑡 𝑓𝑡 𝑟𝑎𝑡𝑖𝑜 𝑑𝑒𝑟𝑖𝑣𝑒𝑑 𝑓𝑟𝑜𝑚 𝑐𝑜𝑠𝑡 𝑚𝑎𝑡𝑟𝑖𝑥 𝑑 𝑖𝑗 = 𝑐𝑒𝑛𝑡𝑒𝑟 − 𝑡𝑜 − 𝑐𝑒𝑛𝑡𝑒𝑟 𝑑𝑖𝑠𝑡𝑎𝑛𝑐𝑒 𝑏𝑒𝑡𝑤𝑒𝑒𝑛 𝑙𝑜𝑐𝑎𝑡𝑖𝑜𝑛 𝑖 𝑎𝑛𝑑 𝑗 𝛾 𝑥𝑖 𝑖𝑠 𝑑𝑒𝑟𝑖𝑣𝑒𝑑 𝑓𝑟𝑜 𝑚 𝑝𝑒𝑟𝑚𝑢𝑡𝑎𝑡𝑖𝑜𝑛 𝑚𝑎𝑡𝑟𝑖𝑥 (𝑓𝑎𝑐𝑖𝑙𝑖𝑡𝑦 𝑥 𝑖𝑠 𝑎𝑠𝑠𝑖𝑔𝑛𝑒𝑑 𝑖𝑛 𝑙𝑜𝑐𝑎𝑡𝑖𝑜𝑛 𝑖 ) 𝑆 𝑖𝑠 𝑎 𝑓𝑢𝑛𝑐𝑡𝑖𝑜𝑛 𝑐 ℎ𝑒𝑐𝑘𝑖𝑛𝑔 𝑖𝑓 𝑓𝑎𝑐𝑖𝑙𝑖𝑡𝑦 𝑥 𝑖𝑠 𝑖𝑛 𝑡 ℎ𝑒 𝑠𝑒𝑡 𝑜𝑓 𝑎𝑙𝑙 𝑠𝑚𝑎𝑙𝑙𝑒𝑟 𝑓𝑎𝑐𝑖𝑙𝑖𝑡𝑖𝑒𝑠 𝑡 ℎ𝑎𝑛 𝑙𝑜𝑐𝑎𝑡𝑖𝑜𝑛 𝑖 By generating new populations from the random initiation of feasible layouts, the model will be merged to find the optimum answer based on reducing the travel times, which is considered as the goal function. The final answer is represented in a string 60 | P a g e layout known as a single “chromosome” in genetic algorithms. An instance of the initial chromosome string used in the algorithm is represented in table 8: Table 8 - Initial Chromosome String used in Genetic Algorithm Locations 1 2 3 4 5 6 7 8 9 10 11 12 13 Facility 5 6 11 7 12 10 9 2 8 13 1 3 4 The underlined facilities (11,12 and 13) are the empty artificial facilities added to the chromosome. 6.2.2 Results The results are divided into two parts. Material procurement results are presenting the order quantities and times for each specific material. There are 6 time slots considered for the whole process schedule (i.e. calculated based on project total time) represented in table 9. The time interval starts from zero (prior to the start of project) till the last hour of the process. Each time unit shows the delivery time of an order. For each material, order quantities in each time slot are represented in the table which provides the overall ordering timeline along with quantities during the whole automated construction process. Table 9 - Solution Matrix representing Material Ordering Quantities during Automated Construction Process Material Quantities Time Slots (1hr) Material Rounded Total Quantity T=0 T=1 T=2 T=3 T=4 T=5 T=6 Chemical Mixtures 100 (Gallons) Q0 = 50 Q1 = 0 Q2 = 0 Q3 = 50 Q4 = 0 Q5 = 0 Q6 = 0 Water 500 (Gallons) Q0 = 500 Q1 = 0 Q2 = 0 Q3 = 0 Q4 = 0 Q5 = 0 Q6 = 0 Cement 111 (Bags) Q0 = 40 Q1 = 0 Q2 = 40 Q3 = 0 Q4 = 31 Q5 = 0 Q6 = 0 Fine Aggregate 150 (CF) Q0 = 150 Q1 = 0 Q2 = 0 Q3 = 0 Q4 = 0 Q5 = 0 Q6 = 0 61 | P a g e Next, site logistics planning results are determining the layout of all facilities on the construction site which is resulted from the genetic algorithm used in section 2.5. The final solution is presented below in Table 10 as a single chromosome string in which each facility is assigned to one location considered on the construction site. Table 10 - Solution Chromosome String Generated for site layout facilities Locations 1 2 3 4 5 6 7 8 9 10 11 12 13 Facility 8 2 9 7 13 11 12 5 10 6 1 3 4 The 3D rendered model in figure 21, provided using Autodesk Revit, illustrates the layout of required facilities on site. 62 | P a g e Figure 21 - 3D Rendered Mode in Revit for Site Layout Facilities 6.2.3 Conclusion Automated construction systems using robotic approaches such as Contour Crafting technology, require precise material procurement and site logistics planning system. This is a new construction paradigm that is different from conventional construction process in which material procurement and site logistics are usually planned independently. The procurement process in an automated construction system should be designed with respect to site conditions to guarantee the delivery of all needed material and components to the construction site on a timely manner. The solution for the multi objective optimization problems are significantly affected by the local vendors’ information such as delivery and material ordering rates. Site conditions (such as congested site vs. open site) are also highly effective factors for 63 | P a g e the final layout of facilities or material quantity orders that are limited to available spaces. 7. Future Research Plan and Procedures In chapters 1 and 2, material procurement and site logistics planning strategies and methods are proposed for an automated construction system using robotic approaches such as Contour Crafting. It was demonstrated that these approaches are able to provide a systematic solution for a feasible material procurement and site logistics planning for an automated construction process considering different constraints and requirements. However, the provided solutions and algorithms are related to the main shell structure of a building or a structure including exterior and interior walls. The ultimate goal of such automatic construction systems is to encompass all other disciplines existing in a building including mechanical, electrical, plumbing and fire protection (MEPF) systems. The main objective of this research is to develop a framework in which a multifunction robot or a number of robots working together are able to construct a building that includes all MEPF components required for common buildings. The process planning and optimization of such comprehensive process requires a highly sophisticated material procurement and site logistics planning system. Revit can contain MEPF systems all together in a single model as shown in figure 22. Besides of architectural and structural components, mechanical, electrical and plumbing systems can be designed and analyzed for different purposes such as energy efficiency. 64 | P a g e Figure 22 - Revit Model sample including Mechanical and Electrical Components [33] IFC parsing system should have a more comprehensive knowledgebase in which parameters related to MEPF systems are defined as well. The data parsing system should be able to extract and categorize different components depending on their discipline category (MEPF). Each set of components (e.g. electrical components) are then sent to a specific toolpath optimization that is providing the optimum toolpath for a robot assigned for locating or constructing the components in that specific discipline such as electrical. For the toolpath optimization part related to placing other components such as mechanical equipment during the automated construction process, algorithms such as time window TSP will be considered. This algorithm is used when visiting a node should happen in a specific predetermined time interval. For placing other components during the construction process, (e.g. electrical plugs), several layers should be printed first and be solidified enough so that other components can be located in the wall. This means that the time of putting these components using other 65 | P a g e robots on the site should take place in a precise time interval which is related to the time required for printing the layers supporting this component. This is the reason that an algorithm such as time window TSP can be used in this part. Next, technologies such as Contour Crafting that are able to construct a building in a short time (e.g. one day), have the capability to construct a large number of buildings and structures in a short reasonable time. A new system for both material procurement and site logistics planning should be developed for this paradigm. Automated Construction Process of colony of buildings (e.g. 100 number of buildings) using single or multi robots has different constraints and requirements. As an instance, material silos can be moved from one building into the next project or while one building’s concrete shell is printed, the previous buildings can get other required services such as placing doors and windows. Lastly, a comprehensive automated construction system requires a lot of different objects and things to be connected to each other and communicate so that the whole system works properly. Internet of Things (IoT) has been the point of focus for many researchers in the recent decade. Home automation and collaboration of different robots and sensing systems working together on the same structure are instances of this technology [31]. The main concept of IoT is interacting of objects such as RFID sensors, mobile phones, robots, computers and software platforms that are able to cooperate with each other to reach a specific goal [32]. Sensors, smart devices, and software systems are the key aspects that collaborate in an IoT framework for data acquisition and analysis to finally provide required information for a decision making system designed for a specific aim. 66 | P a g e As an instance, one of the benefits of using automated construction systems such as Contour Crafting is to provide safe living shells for those people who lose their home in a natural disaster such as earthquake or tsunami. Aerial photography by an airplane or a drone can be used in the region where new buildings are planned to be constructed. Such photos can provide information on the site conditions and based on the current conditions of the sites, proper locations for building new buildings or routing the material vehicles are determined. Next, based on the site conditions and available spaces a decision making system can choose between various predesigned BIM models stored on the cloud system of new buildings to be constructed at the site. Using the location of the materials, an automatic system can search for available vendors in a reasonable distance from the site to provide required materials for construction. Using live traffic information such as the one provided by the Google maps system, delivery times to the site can be planned precisely to avoid any delay and issue during the automated construction process. To conclude, an overall Internet of Things (IoT) framework for Automated Construction System should be provided to explore the benefits using this technology or explore probable issues that might happen in this system. Figure 23 - Drone System providing real-time construction site data [34,35] 67 | P a g e Different devices (including sensors), gateways and cloud systems should be connected together to provide such a platform for automated construction system. Sensors that are connected to a cloud storage system can provide dynamic real-time data such as site conditions (temperature, humidity), robot conditions (e.g. any issue in the robot’s performance) and material storages’ conditions (e.g. no material left because of accident in tanks or silos). Different analysis Application program interfaces (API) such as Google Vision APIs can be utilized to analyze all these data retrieved from various objects to make proper decisions to control the project. 68 | P a g e Chapter 2: Automated Inspection System 1. Introduction One of the main advantages that an automated construction system using 3D printing approach brings is omitting human involvement in the process. These approaches have the potential to be adopted for space construction as well with no human involvement needed. Implementing such a system requires a software platform in which all related aspects of an automated construction process including model analysis, toolpath optimization, material procurement and site logistics planning. While all aforementioned software components are considered as preprocessing modules utilized before beginning the construction process, an automated inspection system is definitely needed during the construction process as well. In the previous chapter, software components that are associated with model data extraction and analysis, toolpath optimization, material procurement and site logistics optimization were discussed. In this chapter, a newly developed automated structure inspection system is introduced. This inspection system should be able to detect any performance and quality issues during the 3D printing process of the structures and report them to the human supervisor to take proper actions be taken to prevent further complications and expenses. 69 | P a g e 2. Statement of the Problem In conventional construction projects, structure inspection is usually considered as a post construction activity. Activities during the construction process such as concrete casting are monitored by labors and engineers to make sure they have been accomplished properly. Also, the inspection task for purposes such as crack detection in concrete walls or pavements is usually performed years after the construction process. There is also an ample time in conventional construction projects to fix any problem that rises either during or after the construction process. However, automated construction systems using robotic approaches promise rapid construction that could take as little as one day to complete a project. Moreover, it is always better and more appropriate to have a continuous 3D printing construction process because of the structural benefits that come with it such as a sturdier binding between the layers and avoiding concrete material getting solid in different parts of the robot such as nozzle system. This means there should be an inspection system that is capable of detecting any issue arisen during the process in a quick time so that required actions may be taken to avoid further problems, risks and enormous costs. Failures can be monitored and detected directly using various sensors and measurement instruments added to different components of the system. However, problems related to the structure being printed layer by layer by the robot requires external devices (such as cameras) that can observe and detect defects during the 3D printing process and report them so that proper actions are performed. Real-time inspection of printed layers during the 3D printing process is significantly important 70 | P a g e to guarantee successful robotic construction. Problems such as layer deformation, collapsing and surface quality defects are instances that should be inspected continuously during the construction process. 3. Background of the Study Automated inspection systems for structure health monitoring have been the point of focus in the recent years for infrastructures such as dams and bridges as well as for buildings. Several research labs have been recently adopting various artificial intelligence methods to implement an automated inspection system in order to detect effects such as cracks in built structures including dams, bridges and nuclear power plants. In the recent years artificial intelligence approaches such as image processing have been adopted extensively for such scenarios. Recent advancements in machine learning and deep learning techniques have proven the fact that they can be successfully employed for a lot of purposes including object detection and structural health monitoring, the technical areas that are the points of focus in this chapter. Deep convolutional neural networks (CNN) have been applied to various computer vision problems with successful outcomes in recent years. In a paper focused on structural health monitoring, Chen et al. [39] propose a framework using convolutional neural network (CNN) and a Naïve Bayes data fusion scheme for crack detection in constructed structures. In another paper, Zhang et al. [40] proposed a supervised deep convolutional neural network for road crack detection using images taken by a low-cost cellphone camera. 71 | P a g e Pixel-wise segmentation has been the point of focus for several research groups in the recent years. Badrinarayanan et al. [46] proposed a deep convolutional encode- decoder architecture for image segmentation named SegNet. In this architecture, which is proposed for both road scenes and indoor environments, each image is encoded first using namouras convolutions and pooling layers and then decoded by means of upsampling and SoftMax layers. Decoding architecture is appended into the network because the final output should be pixel-wise classification in which all pixels in the original input image are classified into classes such as cars, trees, roads, etc. 4. Critique of the Past Approaches Previous works on structural health monitoring topic using image processing techniques are all focused on already-built structures using conventional methods. These methods are not yet applied in automated construction systems using the layer by layer 3D printing approach. Moreover, these models cannot be used for detection of another object or phenomenon because of two reasons: First, pixel-wise classification for detecting concrete layers requires related training data that are images and their corresponding pixel-wise labels. This requirement implies the fact that current models such as Senet or those for crack detections cannot be used for another purpose. For instance, although the proposed SegNet architecture has an acceptable performance on classifying pixels into the classes usually existing in urban streets and indoor scenes, but the model cannot be used for purposes such as concrete layers detection. Figure 24 represents the results of an 72 | P a g e experiment which has been performed by passing two sample images of concrete layers to SegNet model. Figure 24 - Pixel-wise segmentation using SegNet Model [45] 73 | P a g e As shown in Figure 24, concrete layers are not classified properly, and the output classes only cover 12 classes proposed for most probable objects on urban street scenes. Second, in addition to the need for training data described above, each model has its own architecture that is designed for a specific task. For instance, the architecture of models that are designed for human face detection are different from the ones designed for dog or cat detection in image scan. Moreover, each model has its own convolution layers, hyper parameters and other components that are optimized for its specific purpose. These components are used for fine tuning a model based on various metrics such accuracy, number of training data and other constraints. Providing a model for specific purposes, i.e. pixel-wise segmentation of concrete layers vs all surrounding objects, requires testing various network architectures and optimizing different model components based on the model performance using specific training and testing datasets. 5. Research Objective and Questions The main objective of this section is to design an inspection framework to detect defects of concrete layers in large-scale concrete 3D printing through an automated approach using image processing techniques. The research questions to answer this objective are: 74 | P a g e 1- How to design an automated inspection framework that fits in the BIM-integrated software proposed in chapter 1 of this thesis? 2- In this proposed automated inspection framework, how the concrete layers are distinguished from surrounding objects in an image taken during the 3D printing process? 3- In this proposed automated inspection framework, how it is detected if there is any defect in the distinguished concrete layers and where its exact location is? Next section explains our proposed framework to answer the stated research questions. 6. Research Plan and Procedures Our automated inspection framework has two modules: a) Concrete layer detection and b) defect detection. In real time, concrete layer detection module received images from cameras installed on nozzle system as input. Using a novel deep learning-based segmentation approach described in section 6.1 the segments in the images that are associated with the concrete layers are recognized and the rest pixel are turned into white (zero). The output of the concrete detection module is then received by defect detection module where concrete layer defect is detected, and its position is highlighted using a novel algorithm explained in section 6.2. It should be pointed out here than this thesis particularly focuses on bending defection of inter- layer concrete lines. The explained framework fits into the BIM-integrated software as follows: In pre- construction phase, BIM models are analyzed and G-Codes for robot tasks as well 75 | P a g e as material procurement and site logistics plans are generated via the proposed process in first part of the thesis. During the construction phase, according to the stated outputs (e.g. robot toolpath and material preparation units tasks) during the pre-construction phase, the concrete layers are printed. During the 3D printing process, as mentioned previously, based on the images taken in real-time the inspection system evaluates the printed layers and the output is provided in the software in which a human inspector could observe and take required action. Figure 25 represents the schematic view of the entire system. Figure 25 - Schematic view of the proposed system The novel contribution of part 2 include: 1) an automated inspection framework that fits into the software proposed before 2) A deep-learning based segmentation approach to detect printed layers 3) A defect detection approach to detect bending defection in printed concrete layers along with their exact location. 76 | P a g e Implementing an automated inspection system in which defects related to the concrete structures (e.g. concrete layers extraction and defect detection) being printed by a robot requires the following components: 1- A trained model which receives an image as input and then detects and separates the concrete structure from all other existing objects in the field including the robot and other surrounding substances is required. The output from this model will be used as an input for the model described below. 2- Defective Layer detection model in which defective layers are distinguished is required. This model receives its input from the aforementioned model and detects the spots where layer defects occurred. To validate the effectiveness and robustness of the proposed method, both models are evaluated using the images of actual automated construction process. The performance of each model is then evaluated separately. The first model is evaluated by its accuracy for distinguishing a concrete structure from other surrounding objects. The performance of the second model is evaluated based on the accuracy of detecting locations of defective layers within an image. The validation dataset is unseen and not used in the training process of the first model. A variety of conditions in terms of surrounding environment and concrete labels (e.g. defective and good condition) are present in the validation set to make sure the models have acceptable performance under a variety of conditions. The models are fine-tuned by finding the optimal values for hyper parameters with better performance. 77 | P a g e 6.1 Automated Concrete Layers Detection 6.1.1 Methodology In this part the methodology to create a model that receives images as input and is capable of distinguishing concrete layers from other surrounding objects on construction site is presented. All required components for implementation of the model are described in this section. 6.1.1.1 Image Capturing System for Data Collection In this part the method for capturing images to be used for the training task is described. The data used for training the model that is implemented for concrete layers detection is taken using 13 MP cameras. The cameras are set to capture square size images of 1000 by 1000 pixels. However, the pipeline designed for receiving images as inputs to the model is not limited to the camera type or image size. Several preprocessing steps that are described in this chapter are applied to each image for size and color adjustments as well as segmentation and noise reduction. Images are taken from cameras (Raspberry Pi 3 Model B plus Camera Fisheye Wide Angle 5MP 1080p) that are implemented on four sides of the printing nozzle of the gantry system so that they cover all sides of a structure during the layer by layer 3D printing process. Cameras are installed on the nozzle system in a way that they do not cause any clashes with the structure and other components of the robot. 6.1.1.2 Image Data Preprocessing Raw images that are captured by the cameras, described in previous section, are 3 channel (RGB) types. To reduce the complexity and size of the input data the 78 | P a g e images are all converted into single gray channel. This helps to reduce the complexity of the model, input data size and higher computation speed in both training and processing stages. Moreover, concrete layers may come in different colors such as the layers shown in Figure 25. Converting RGB channels into a single channel using the weighted transformation will provide single channel images in which the effect of concrete colors variety is significantly reduced. The next step after channel transformation is resizing step. All images are also resized into 512 by 512 pixels images. Due to the fact that the original images have the same height and width, layer attributes such as inter-layer lines and concrete surface textures are not changed by scaling down. Also, the resizing of images will not cause any problem for the detection task. In case the original images were not taken in 1:1 ratio, scaling the images would increase or decrease the layers line slopes (if they are not vertical or horizontal in camera perspective). This change will have an effect on defect detection that is described along in this chapter. 6.1.1.3 Image Data Augmentation One of the common methods in deep learning or machine learning methods in data augmentation. Using data augmentation techniques, new data will be generated which basically increases the number of training data. The most important benefit of data augmentation is providing a new data for training task which is not existing in the actual dataset and this will help in covering other probable conditions in model training. Common techniques to create augmented data are flipping, rotating, zooming, cropping or changing the color of the images. Moreover, adding artificial random shadows and changing model light intensity are other common approaches 79 | P a g e in data augmentation. The last two are considered when capturing images can happen in various environmental conditions and times. For training the deep learning model described in this section several techniques are adopted to generate new augmented data including: a) horizontal flipping, b) rotation +-45 degrees, c) adding artificial random rectangular shadows, and d) changing brightness. The reason behind choosing augmentation techniques is to cover probable conditions that can be generated using the existing data. For horizontal flipping, there is a high chance that the same image symmetric to the y axis will be given to the model. Moreover, horizontal flipping will not change characteristics of the surrounding objects usually appearing on the construction sites such as silos, people, gantry robot beams and etc. Rotations are applied since concrete layers may not be always seen from a straight perspective and various layers’ line slopes are probable in the image. Here, after rotating an image, all empty pixels that are generated because of the rotation will be turned into white pixels and the picture is again resized to the original size as before. This process will also zoom out a little from the original image which is considered as a beneficial instance in data augmentation. Adding artificial shadows to the images is done due to the high probability of real shadows in real construction sites. Shadows of different objects on the concrete layers should not change the model ability to classify them as a wrong object and this is the main reason behind adding artificial shadows to the model. Adding 80 | P a g e artificial shadows is done by defining random four points inside an image boundary and making the image part within the four points boundaries darker. The brightness technique is applied to all pixels within an image. Here, half of the images were randomly selected to be brighter and the other half to be darker than the original images to provide equal chance for both augmentations. Using 100 original images and applying the above techniques for creating augmented new images results in providing a database with 1000 images. Total images = 100 (original) + 100 (flipped) + 400 (rotations) + 200 (random artificial shadows added to original and flipped images) + 200 (random artificial lighting added to original and flipped images) = 1000 Images 6.1.1.4 Image Pixel-wise Segmentation and Labeling for Concrete Layers Detection Model Training One of the most important steps of implementing machine learning or deep learning models for classification purposes is the labeling process. This step basically identifies the output that is expected from the model once it is trained. Understanding the desired output will help how the labels or classes for input data should be defined. Here, the labeling approach that is utilized for preparing the training data for concrete layers detection model is described. Each image captured for model training is segmented into two classes using a binary 0 and 1 class types Concrete layers boundary is manually set and then all pixels that fall inside the boundary line are set to 1 while other are set to zero. We defined two semantic classes for our problem: concrete (i.e., 1) and non-concrete (i.e., 0). Accordingly, a label was assigned to each pixel in an image (1 or 0). Using this labeling 81 | P a g e system, once the model is trained, it will be able to classify image segments into two classes and distinguish concrete layer pixels. Figure 26 provides an example in which both classes are present. In this figure, two sample segmentations are provided for an image that includes image of concrete layers which is taken from the nozzle facing y axis and x axis. Augmented images that are provided using the techniques described in previous section are all automatically labeled using the transformed label mask from original images. Figure 26 - Binary Mask Ground Truth Creation for Training Data Labeling Process a) Sample image captured from Nozzle facing y axis, b) Sample image captured from Nozzle facing x axis Using this approach, we are able to extract only printed concrete layers from images and pass the mask applied image to the next module for defect detection. 82 | P a g e 6.1.1.5 Model Training using Convolutional Neural Network (CNN) In this section, a Convolutional Neural Network (CNN) model architecture is proposed. The performance of the model is improved by fine-tuning several hyperparameters and observing performance metrics including accuracy, harmonic average of precision and recall (known as f1 score) and miss rate of the model. To detect the defections in images, as first step, we detect the concrete via semantic pixel-wise segmentation of the images. To do so, we designed and trained a deep convolutional neural network architecture inspired by SegNet [45]. SegNet is a deep convolutional network architecture designed for image segmentation. Our network has three components: an encoder, a decoder, and a pixel-wise classification layer. Figure 27 shows the proposed architecture and its corresponding components. For the encoder part, which is basically the feature extraction part of the network, what was used in this work are 8 convolutional layers with filter size of 5X5 and 32, 64 or 128 channels in sequence, with batch normalization and rectified linear unit (ReLU) activation function [50]. Convolution layers refers to those layers that basically extract the features and patterns from the input images. The input data is convolved using convolution filters that come as an n by n bucket size of weights where n is usually an odd number such as 3 or 5. The weights inside convolution filters are usually randomly initialized and then optimized through the training process. Activation functions define the output of neurons given a set of inputs. The weighted sum of previous layer neurons is given as an input to the activation function and then the output is resulted based on the specific activation function used for that neuron. ReLU transforms the input to the maximum of either zero or the input itself. This transformation cancels out the negative 83 | P a g e values received as input. Batch normalization or batch norm controls the amount of covariance shift for hidden layer unit values. Batch norm normalizes the outputs from the activation functions. It is used for increasing the learning speed as well as expanding the model capability for input to output mapping in case of having various input data distributions. For instance, if a model is trained for only dark concrete colors, without batch normalization, probably it will not have a good performance for mapping X to Y values for those concrete types that are not dark. In other words, batch normalization reduces overfitting by adding slight regularization effects. Batch normalization adds a slight noise to each hidden layer’s activations and this will help in using higher learning rates since activations are not exploded (i.e. going very high or low). Encoders perform the task of downsampling. Downsampling layers are located after convolutional layers to reduce the feature map dimensionality for computational efficiency. For downsampling 2x2 max pooling layers with stride of size 2 are used. Max pooling layers preserves the maximum value from a cluster of neurons in previous layer. Using max pooling the number of parameters and required computations are reduced in the network while the overfitting is also avoided. The encoder extracts the features from images by performing convolutions and producing a set of feature maps. In the decoder, identical to the encoder, we used 8 convolutional layers. However, instead of down sampling layers we have upsampling layers to produce sparse feature maps. Decoders perform upsampling task using the memorized max-pooling values from the encoder feature maps. A batch normalization step is applied to each map and the final probabilities are generated for each pixel. The output of the decoder is the same size as 84 | P a g e the input image. The sigmoid layer then carries out pixel-wise classification which is done for each pixel separately. Figure 27 represents the CNN network architecture designed for concrete layer detection. Figure 27 - CNN Network Architecture designed for concrete layers extraction As mentioned before, the entire dataset size contains 1000 images resulted from combining original images and augmented data. The training dataset includes 800 RGB images and the test dataset contains 200 RGB images which is 20% of the entire dataset. To avoid overfitting which might for instance be caused by putting all augmented images in training set and original images in testing set, each cluster of training set (i.e. original images and augmented images including flipped, rotated, random shadow and lighting images) is split into the same 20% and 80% rations and then combined together to provide testing and training sets. 85 | P a g e All images are resized using the aforementioned size. As mentioned in the previous section, the original images are manually annotated, and their mask is used for other augmented images to have semantic pixel-wise segmentation of the images. Optimization of a network should be performed using a loss function that can provide a reasonable comparison between the generated output from the model and the actual labels. In this model with the purpose of pixel-wise segmentation, cross-entropy loss [49] is a good candidate to be used as the network objective function for training. The cross-entropy loss function determines the difference between the classification output of each pixel with its true label. In a binary pixel-wise classification, i.e. concrete vs. non-concrete pixels, the cross-entropy loss function can be calculated as follow for each pixel output: 𝑙𝑜𝑠𝑠 𝑐𝑟𝑜𝑠𝑠 −𝑒𝑛𝑡𝑟𝑜𝑝𝑦 = −(𝑦𝑙𝑜𝑔 (𝑝 ) + (1 − 𝑦 )log(1 − 𝑝 )) 𝑤 ℎ𝑒𝑟𝑒 : 𝑦 𝑖𝑠 𝑡 ℎ𝑒 𝑡𝑟𝑢𝑒 𝑙𝑎𝑏𝑒𝑙 𝑓𝑜𝑟 𝑒𝑎𝑐 ℎ 𝑖𝑚𝑎𝑔𝑒 ∈ {0,1} 𝑝 𝑖𝑠 𝑡 ℎ𝑒 𝑜𝑢𝑡𝑝𝑢𝑡 𝑔𝑒𝑛𝑒𝑟𝑎𝑡𝑒𝑑 𝑓𝑜𝑟 𝑒𝑎𝑐 ℎ 𝑝𝑖𝑥𝑒𝑙 ∈ {0,1} The Adam optimization algorithm, which is an extension to stochastic gradient descent that has recently seen broader adoption for deep learning applications in computer vision [46] was used to train the network (i.e., minimizing the cross-entropy loss) in the experiment. One major benefit of using the Adam optimization method compared to other optimization algorithms is that the model can get to the stage of generating good results in a faster manner. 86 | P a g e The implementations were carried out using TensorFlow and Keras with TensorFlow backend [47,48]. In training process, 1000 epochs with batch-size of 20 are used that result in having 50 batches in total. At beginning of each epoch, the images were first shuffled and then divided into batches for training. The results of the final model proposed for concrete layers pixel-wise segmentation are presented in the next section. 6.1.2 Results To evaluate the performance of the trained network, three commonly used performance measures were used: (1) accuracy (percentage of the pixels that were classified correctly), (2) precision (with true positives relating to concrete layers), and (3) recall. Miss rate is also measured for both classes. These three metrics are calculated as follows: Equation 6 𝑇𝑜𝑡𝑎𝑙 𝐴𝑐𝑐𝑢𝑟𝑎𝑐 𝑦 : 𝐴𝐶𝐶 = 𝑇𝑃 + 𝑇𝑁 𝑇𝑃 + 𝐹𝑃 + 𝑇𝑁 + 𝐹𝑁 Equation 7 𝐹 1 𝑆𝑐𝑜𝑟𝑒 : 𝐹 1 = 2 𝑃𝑃𝑉 . 𝑇𝑃𝑅 𝑃𝑃𝑉 + 𝑇𝑃𝑅 Equation 8 𝑀𝑖𝑠𝑠 𝑅𝑎𝑡𝑒 : 𝐹𝑁𝑅 = 𝐹𝑁 𝑇𝑃 + 𝐹𝑁 𝑤 ℎ𝑒𝑟𝑒 : 𝑃𝑜𝑠𝑖𝑡𝑖𝑣𝑒 𝑟𝑒𝑝𝑟𝑒𝑠 𝑒 𝑛𝑡𝑠 𝑏𝑒𝑛𝑑𝑒𝑑 𝑐𝑜𝑛𝑐𝑟𝑒𝑡𝑒 𝑙𝑎𝑦𝑒𝑟𝑠 𝑎𝑛𝑑 𝑛𝑒𝑔𝑎𝑡𝑖𝑣𝑒 𝑟𝑒𝑝𝑟𝑒𝑠𝑒𝑛𝑡𝑠 𝑙𝑎𝑦𝑒𝑟𝑠 𝑤𝑖𝑡 ℎ 𝑛𝑜 𝑏𝑒𝑛𝑑𝑖𝑛𝑔 𝑇𝑃 𝑖𝑠 𝑡 ℎ𝑒 𝑟𝑎𝑡𝑒 𝑜𝑓 𝑡𝑟𝑢𝑒 𝑝𝑜𝑠𝑖𝑡𝑖𝑣𝑒 𝑓𝑟𝑎𝑚𝑒𝑠 𝐹𝑃 𝑖𝑠 𝑡 ℎ𝑒 𝑟𝑎𝑡𝑒 𝑜𝑓 𝑓𝑎𝑙𝑠𝑒 𝑝𝑜𝑠𝑖 𝑡𝑖𝑣𝑒 𝑓𝑟𝑎𝑚𝑒𝑠 87 | P a g e 𝑇𝑁 𝑖𝑠 𝑡 ℎ𝑒 𝑟𝑎𝑡𝑒 𝑜𝑓 𝑡𝑟𝑢𝑒 𝑛𝑒𝑔𝑎𝑡𝑖𝑣𝑒 𝑓𝑟𝑎𝑚𝑒𝑠 𝐹𝑁 𝑖𝑠 𝑡 ℎ𝑒 𝑟𝑎𝑡𝑒 𝑜𝑓 𝑓𝑎𝑙𝑠𝑒 𝑛𝑒𝑔𝑎𝑡𝑖𝑣𝑒 𝑓𝑟𝑎𝑚𝑒𝑠 Total accuracy can provide a good understanding of the overall model performance while F1 score can express sensitivity and precision of the model in a combined mode which is known as harmonic mean of precision and recall. Table 11 summarizes the performance of model which is gained using 1000 epochs with batch sizes of 20. Table 11 - Model Performance Metrics for both Concrete Layers and Non-concrete pixels Classes Accuracy F1 Score Miss Rate Concrete Layers 90.5% 90.26% 2.22% Non-concrete 90.5% 90.73% 15.46% Based on the achieved results, miss rate for concrete layers shows a good performance for not missing concrete layer related pixels. However, some pixels related to non- concrete objects are misclassified as concrete layers. Most of the misclassified pixels relate to the ground area close to concrete pixels on which the layers were printed. Majority of other misclassified pixels relate to beams of the gantry system structure. One approach to avoid such a problem is to paint the beams with a black color which has extremely higher pixel value compared to normal concrete layers. Figure 28 shows the segmentation (ground truth vs. our result) results of the model for a sample taken while facing gantry x axis. 88 | P a g e Figure 28 - Sample of Model Pixel-wise segmentation 6.2 Automated Detection of Defects in Layers 6.2.1 Methodology In this part a framework for distinguishing bended layers from an image outputted from the previous CNN model is proposed. This framework has three main components, that are combined to provide the ability of detecting bended layers, as follows: a) image fata preprocessing, b) layers’ edge extraction and line formation, and c) bended layers detection using an outlier detection technique with a defined threshold. The following sections describe the above-mentioned components of the framework and finally the results are shown and discussed. Limitations and future research topics are presented through the end of this chapter. 6.2.1.1 Image Preprocessing The main purpose of the framework proposed in this chapter is to detect defects in concrete layers. To this aim, preprocessing techniques that are chosen should facilitate the task of layers line detections along the process so that the defective layers are detected. Images that are captured from the previous model are transformed from RGB channel to single channel using the weighted sum of RGB values described in data 89 | P a g e preprocessing step of section 6.1. The next preprocessing techniques described here are all applied to the single channel images. A problem that a raw image of concrete layers causes in line extractions is its texture. The texture of the concrete results in having a very noisy data points for edge detection task. This noisy space is due to the surface quality of the concretes. The defect detection approach presented here should not be limited because of the surface quality of the concrete which means if the surface quality is not very smoothed, the model should still be able to detect layer boundary lines. This means if this noise is not reduced, a lot of unwanted lines and edges will be detected in the next steps. To this aim, a noise reduction technique should be applied so that the aforementioned problems are avoided. However, if a comprehensive noise reduction technique be applied to the entire image, layer lines may also be ignored in edge detection task. This will result in dissolving the noise but smoothing away the edges at the same time. A noise reduction technique that can only affect the texture of an object but not its edges should be applied here so that the information (data points) related to layer lines are not lost. For removing noise resulted from concrete texture, an approach mainly known as bilateral filtering is utilized in this framework. Bilateral filtering is an approach proposed by Tomasi et al. [41] to smooth images while keeping edges through a nonlinear combination of neighborhood pixels. One of the main advantages of this method is that the computation process in not iterative and thus it can be used for fast response needed frameworks. While basic smoothing filters average the pixels on all regions, bilateral method is a way to do the same but prevent smoothing around the edges. It is 90 | P a g e the combination of domain and range filtering and this is the main reason that it is known as bilateral filter. Domain filtering weights pixels with coefficients that reduces by the distance while the weights used in range filter are dependent to image intensity (for gray pictures). When domain filter is applied to an image, pixel values are changed, and the new image is produced as follows: Equation 9 𝑖 (𝑥 ) = 𝑛 𝑑 −1 (𝑥 ) ∬ 𝑓 (𝑝 ) 𝑐 (𝑝 , 𝑥 )𝑑𝑝 ∞ −∞ 𝑤 ℎ𝑒𝑟𝑒 : 𝑖 (𝑥 ) 𝑖𝑠 𝑡 ℎ𝑒 𝑛𝑒𝑤 𝑖𝑚𝑎𝑔𝑒 𝑎𝑓𝑡𝑒𝑟 𝑎𝑝𝑝𝑙𝑦𝑖𝑛𝑔 𝑡 ℎ𝑒 𝑓𝑖𝑙𝑡𝑒𝑟 𝑛 𝑑 (𝑥 )𝑖𝑠 𝑡 ℎ𝑒 𝑛𝑜𝑟𝑚𝑎𝑙𝑖𝑧𝑎𝑡𝑖 𝑜𝑛 𝑡𝑒𝑟𝑚 𝑎𝑛𝑑 𝑒𝑞𝑢𝑎𝑙 𝑡𝑜 ∬ 𝑐 (𝑝 , 𝑥 )𝑑𝑝 ∞ −∞ 𝑓 (𝑝 ) 𝑖𝑠 𝑡 ℎ𝑒 𝑜𝑟𝑖𝑔𝑖𝑛𝑎𝑙 𝑖𝑚𝑎𝑔𝑒 𝑐 (𝑝 , 𝑥 ) 𝑖𝑠 𝑡 ℎ𝑒 𝐸𝑢𝑐𝑙𝑖𝑑𝑒𝑎𝑛 𝑑𝑖𝑠𝑡𝑎𝑛𝑐𝑒 𝑓𝑢𝑛𝑐𝑡𝑖𝑜𝑛 𝑏𝑒𝑡𝑤𝑒𝑒𝑛 𝑡 ℎ𝑒 𝑝𝑖𝑥𝑒𝑙 𝑝 𝑎𝑛𝑑 𝑖𝑡𝑠 𝑛𝑒𝑖𝑔 ℎ𝑏𝑒𝑟 ℎ𝑜𝑜𝑑 𝑐 𝑒𝑛𝑡𝑒𝑟 𝑥 Range filtering which is defined based on the similarity of the pixel values is defined as follows: Equation 10 𝑟 (𝑥 ) = 𝑛 𝑟 −1 (𝑥 ) ∬ 𝑓 (𝑝 ) 𝑠 (𝑓 (𝑝 ), 𝑓 ( 𝑥 ))𝑑𝑝 ∞ −∞ 𝑤 ℎ𝑒𝑟𝑒 : 91 | P a g e 𝑟 (𝑥 ) 𝑖𝑠 𝑡 ℎ𝑒 𝑛𝑒𝑤 𝑖𝑚𝑎𝑔 𝑒 𝑎𝑓𝑡𝑒𝑟 𝑎𝑝𝑝𝑙𝑦𝑖𝑛𝑔 𝑡 ℎ𝑒 𝑓𝑖𝑙𝑡𝑒𝑟 𝑛 𝑟 (𝑥 )𝑖𝑠 𝑡 ℎ𝑒 𝑛𝑜𝑟𝑚𝑎𝑙𝑖𝑧𝑎𝑡𝑖𝑜𝑛 𝑡𝑒𝑟𝑚 𝑎𝑛𝑑 𝑒𝑞𝑢𝑎𝑙 𝑡𝑜 ∬ 𝑠 (𝑓 (𝑝 ), 𝑓 ( 𝑥 ))𝑑𝑝 ∞ −∞ 𝑓 (𝑥 ) 𝑖𝑠 𝑡 ℎ𝑒 𝑜𝑟𝑖𝑔𝑖𝑛𝑎𝑙 𝑖𝑚𝑎𝑔𝑒 𝑠 (𝑓 (𝑝 ), 𝑓 ( 𝑥 )) 𝑖𝑠 𝑡 ℎ𝑒 𝑠𝑖𝑚𝑖𝑙𝑎𝑟𝑖 𝑡𝑦 𝑓𝑢𝑛𝑐𝑡𝑖𝑜𝑛 𝑏𝑒𝑡𝑤𝑒𝑒𝑛 𝑡 ℎ𝑒 𝑣𝑎𝑙𝑢𝑒𝑠 𝑜𝑓 𝑝𝑖𝑥𝑒𝑙 𝑝 𝑎𝑛𝑑 𝑖𝑡𝑠 𝑛𝑒𝑖𝑔 ℎ𝑏𝑒𝑟 ℎ𝑜𝑜𝑑 𝑐𝑒𝑛𝑡𝑒𝑟 Combination of both closeness and similarity functions will provide the bilateral filtering which is applied to each single pixel as follows: Equation 11 𝑏 (𝑥 ) = 𝑛 −1 (𝑥 ) ∬ 𝑓 (𝑝 ) 𝑠 (𝑓 (𝑝 ), 𝑓 ( 𝑥 )) 𝑐 (𝑝 , 𝑥 ) 𝑑𝑝 ∞ −∞ 𝑤 ℎ𝑒𝑟𝑒 : 𝑏 (𝑥 ) 𝑖𝑠 𝑡 ℎ𝑒 𝑛𝑒𝑤 𝑖𝑚𝑎𝑔𝑒 𝑎𝑓𝑡𝑒𝑟 𝑎𝑝𝑝𝑙𝑦𝑖𝑛𝑔 𝑡 ℎ𝑒 𝑏𝑖𝑙𝑎𝑡𝑒𝑟𝑎𝑙 𝑓𝑖𝑙𝑡𝑒𝑟 𝑛 (𝑥 )𝑖𝑠 𝑡 ℎ𝑒 𝑛𝑜𝑟𝑚 𝑎𝑙𝑖𝑧𝑎𝑡𝑖𝑜𝑛 𝑡𝑒𝑟𝑚 𝑎𝑛𝑑 𝑒𝑞𝑢𝑎𝑙 𝑡𝑜 ∬ 𝑠 (𝑓 (𝑝 ), 𝑓 ( 𝑥 )) 𝑐 (𝑝 , 𝑥 ) 𝑑𝑝 ∞ −∞ 𝑓 (𝑥 ) 𝑖𝑠 𝑡 ℎ𝑒 𝑜𝑟𝑖𝑔𝑖𝑛𝑎𝑙 𝑖𝑚𝑎𝑔𝑒 𝑠 (𝑓 (𝑝 ), 𝑓 ( 𝑥 )) 𝑖𝑠 𝑡 ℎ𝑒 𝑠𝑖𝑚𝑖𝑙𝑎𝑟𝑖𝑡𝑦 𝑓𝑢𝑛𝑐𝑡𝑖𝑜𝑛 𝑏𝑒𝑡𝑤𝑒𝑒𝑛 𝑡 ℎ𝑒 𝑣𝑎𝑙𝑢𝑒𝑠 𝑜𝑓 𝑝𝑖𝑥 𝑒𝑙 𝑝 𝑎𝑛𝑑 𝑖𝑡𝑠 𝑛𝑒𝑖𝑔 ℎ𝑏𝑒𝑟 ℎ𝑜𝑜𝑑 𝑐𝑒𝑛𝑡𝑒𝑟 𝑐 (𝑝 , 𝑥 ) 𝑖𝑠 𝑡 ℎ𝑒 𝐸𝑢𝑐𝑙𝑖𝑑𝑒𝑎𝑛 𝑑𝑖𝑠𝑡𝑎𝑛𝑐𝑒 𝑓𝑢𝑛𝑐𝑡𝑖𝑜𝑛 𝑏𝑒𝑡𝑤𝑒𝑒𝑛 𝑡 ℎ𝑒 𝑝𝑖𝑥𝑒𝑙 𝑝 𝑎𝑛𝑑 𝑖𝑡𝑠 𝑛𝑒𝑖𝑔 ℎ𝑏𝑒𝑟 ℎ𝑜𝑜𝑑 𝑐𝑒𝑛𝑡𝑒𝑟 𝑥 One parameter that should be set before applying the bilateral filter is the number of neighborhood pixels from a pixel to cover for calculations. This means how many surrounding pixels should be considered to iterate for both distance and similarity 92 | P a g e functions. This value is tuned based on the assumption that the pictures are captured in a range of 20cm to 100cm from the concrete layers. Figure 29 shows how the bilateral filter has changed the intensity histogram of an image including 3 concrete layers by smoothing the texture pixels and reserving edges. Figure 29 - a) Original Single Channel Image b) Transformed Image using Bilateral Filter c) Pixel Intensity Histogram for Original and Transformed Images This transformation in which concrete textures are smoothed while edges that are layer lines are preserved will facilitate the next step which is the edge detection process described in the next section. 6.2.1.2 Edge Detection Now that the noise reduction is applied in previous step, edge detection algorithm can detect and keep the edges in the image. Using the approach proposed by Canny [42], the angle of the gradient can be intended by calculating the gradient magnitude in both X and Y directions as follows in a local segment of an image: 93 | P a g e Equation 12 𝜃 = tan −1 𝐺 𝑦 𝐺 𝑥 This angle is perpendicular to the edge direction. Now that the gradient direction is set, for all pixels within the local windows of the image, a function should check whether the pixel value is the maximum among all other pixels in the gradient direction. Those pixels that are maximum will be kept and those that are not will be changed to zero. This process is known as non-maximum suppression which basically thins the edges and preserve those pixels with maximum values in gradient directions. Figure 30 illustrates how the pixels located near an edge are transformed based on their magnitude compared to neighborhood pixels along the gradient direction. Figure 30 - Preserving maximum value pixels in gradient direction [43] Here, the results of the edge detection step for two same images, one having bilateral filter applied and the other without bilateral filter, are shown. The result implies the fact that applying bilateral filter is an essential component in this framework. Figure 31 shows edge detection process of two same images. The first row relates to the image of 3 concrete layers with concrete texture removed and the second row is the same image while texture is preserved. 94 | P a g e Figure 31 - Edge detection process with and without texture removal After this transformation, pixels near edges are reserved and more noise from the picture is removed. The next task is to detect layer lines from the pixels preserved by edge detection algorithm. 6.2.1.3 Layer Lines formation Line formations should be applied to the segments of an image resulted from the previous step. Once the segmentation size is defined, the sliding windows will traverse over the entire image and form line within each window. One approach to form a line would be fitting a linear regression model using the coordination of all preserved pixels within a window. However, this approach will work 95 | P a g e properly if there is only one line existing inside the frame. In the case of having several lines in a frame being analyzed, such as having several layers inside the sliding window, this approach will fail. This is due to the fact that the model will consider all points together and the result would be a line fitted to all points. Another problem is that when there are several lines inside a frame, only the slope and exact position of the average line is represented. In this case, even if the lines are parallel and the average slope is a good representative of all the lines, the exact position of each line is missed. In some probable scenarios, for instance having three layers with two of them taking symmetric slopes to one being horizontal, fitting a linear regression model for all the lines will provide a slope of zero since two lines are symmetric to the third horizontal line axis. This implies that in case this slope is used for defect analysis, it will ignore the defect that has happened for two layers. Consequently, an approach that can detect all the lines and their precise parameters (slope and position) is required here to avoid the aforementioned problems. In a method called “Hough Transform” proposed by Hough [44], parameters of all the simple shapes such as lines or circles existing within a frame can be extracted. In this approach, even if there are several lines in an image, their precise slope and position can be provided successfully. This approach is chosen to be used in current framework since it will avoid the problematic scenarios described for concrete layers. Here the process of detecting several lines using Hough transform as well as the rationale behind choosing the parameters used in this transform for the purpose of detecting layer lines is described. 96 | P a g e A line can be represented in different formats. Considering an image frame with an origin at (0,0), each line can be represented by two parameters 𝜌 and 𝜃 as shown in figure 32. 𝜌 would be distance between the origin pixel and the line while 𝜃 is the angle between x axis and the perpendicular line from the origin to the line. Using these two parameters each line can be positioned precisely within an image frame. By choosing the origin to be at one of the frame corners (i.e. chosen to be top left here), the possible interval for the 𝜌 parameter is between zero and the diagonal length of the frame and for 𝜃 it would be zero to 180 degrees. Figure 32 - Line Representation Coordinate System used for Hough Transformation The main idea behind Hough transformation is passing all probable lines from each pixel preserved within a frame and record the two 𝜌 and 𝜃 parameters based on its distance and slope to the origin pixel. The number of lines that are passed from each pixel is dependent to the step size considered for 𝜌 and 𝜃 . Once all the possible lines are passed from each preserved pixel, a chart consisting all the line parameters passed from each pixel can be provided using 𝜌 and 𝜃 values as XY axes. The intersections in this chart shows the characteristics of those lines that are passing at least two pixels. 97 | P a g e More crowded intersections represent the lines passing more pixels. Each time an intersect happen, the pair consisting those specific 𝜌 and 𝜃 is upvoted by the number of lines intersecting. By setting a minimum number of pixels required in a line, those pairs that pass this threshold will be kept. Figure 33 illustrates an example with 3 preserved pixels in an image. Figure 33 - Example of Hough Transform Process [44]. a) Line passing from preserved pixels b) Chart of line values passed from each pixel to detect intersections and upvoted pairs As shown in the chart (figure-6-b), the intersection point represents the line at 𝜃 0 degree with 𝜌 0 from the origin which passes all the 3 points. All other pairs are related to the lines that pass only one point in this image. The step size for both 𝜌 and 𝜃 is defined based on the required accuracy. Also, the minimum number of pixels should be defined for the final filtering step. In this framework, considering the sliding window of size 50 by 50 pixels, the minimum line length is set to be 25 pixels which covers 50% of each axis and also the step size for 𝜌 is one pixel and for 𝜃 is one degree. Using this system, those short lines that are probably formed by concrete texture are removed and the final output for the line angles would be between 0 and 180 degrees with an accuracy of one degree which is an acceptable accuracy to detect defective cases. Another benefit of setting the minimum 98 | P a g e line length threshold to be 25 pixels is that the short lines in the edges are also removed and each layer line is represented by few lines rather than numerous short lines. This will help in having a more accurate evaluation of line slopes and reducing computation process for the final step which is defect detection. 6.2.1.5 Defect Detection In this section, the proposed procedure to detect bended concrete layers is described. The final output from this part is highlighted sections of an image where a defective case occurs. Using the outputs from the previous section that are line parameters, an algorithm should be defined to detect abnormal lines known as bended layers. Each image contains several printed layers and the layers are not necessarily starightlyface the camera. This implies the fact that the layers should be compared together rather than a fixed slope value such as zero representing horizontal direction. Moreover, a picture can be taken from different positions such as a corner of a wall which means there are cases where the layer lines are not parallel entirely and the angles might be different at each part of the image. This means that comparing all the lines existing in an image for outlier selection is not a proper approach to detect bended layers. Using a 50x50 pixel sliding windows, each 300x300 size image is segmented into 6 by 6 windows. Each column of size 50 includes 6 windows. By traversing through all the columns, the lines within each window are collected and appended in a list for each column separately. The result of this process is having 6 lists containing all the lines within the six columns of an image. These lists are then analyzed separately, and outliers are detected within each window. Figure 34 represents the iteration process through all the columns using a sliding window. 99 | P a g e Figure 34 - Iteration process for outlier detection over the entire image The main reason that columns are not analyzed as a single frame is to reduce the computation process. Analyzing an entire column specially in Hough transform step increases the calculations dramatically. This will cause slow responses from the system. Aside from time consuming calculations, the lines that are formed by passing a line from the pixels of a layer line to another layer are not important for defect detection. The last step in this section is to detect outliers in the column lists containing line angles. By averaging all the line angles and setting a constant threshold, outliers can be detected using the following formula: Equation 13 𝑎𝑏𝑠 ( 𝛼 − 1 𝐿 ∑ 𝜃 𝑙 𝑙 =𝐿 𝑙 =1 ) ≥ 𝛿 → 𝛼 ∈ 𝑂 100 | P a g e 𝑤 ℎ𝑒𝑟𝑒 : 𝛼 𝑖𝑠 𝑡 ℎ𝑒 𝑎𝑛𝑔𝑒𝑙 𝑜𝑓 𝑎 𝑙𝑖𝑛𝑒 𝑏𝑒𝑖𝑛𝑔 𝑡𝑒𝑠 𝑡 𝑒𝑑 𝜃 𝑙 𝑖𝑠 𝑡 ℎ𝑒 𝑎𝑛𝑔𝑒𝑙 𝑜𝑓 𝑙𝑖𝑛𝑒𝑠 𝑖𝑛𝑠𝑖𝑑𝑒 𝑒𝑎𝑐 ℎ 𝑐𝑜𝑙𝑢𝑚𝑛 𝑙𝑖𝑠𝑡 𝑏𝑒𝑡𝑤𝑒𝑒𝑛 − 90 𝑎𝑛𝑑 90 𝑑𝑒𝑔𝑟𝑒𝑒𝑠 𝛿 𝑖𝑠 𝑡 ℎ𝑒 𝑑𝑖𝑓𝑓𝑒𝑟𝑒𝑛𝑐𝑒 𝑡 ℎ𝑟𝑒𝑠 ℎ𝑜𝑙𝑑 𝑓𝑜𝑟 𝑜𝑢𝑡𝑙𝑖𝑒𝑟 𝑑𝑒𝑡𝑒𝑐𝑡𝑖𝑜𝑛 𝑂 𝑖𝑠 𝑡 ℎ𝑒 𝑠𝑒𝑡 𝑜 𝑓 𝑜𝑢𝑡𝑙𝑖𝑒𝑟𝑠 The angles of the lines are transformed from the [0,180] interval into the [-90,90] interval. This transformation is applied to avoid false positive outliers specially for near horizontal lines. For instance, if there are two lines that are both almost horizontal but one with an angle close to zero and the other one close to 180 degrees, although they are almost parallel, but they will be both detected falsely as outliers. However, if the angle with value close to 180 degrees is transformed to [-90,0] interval, the average value of these two lines is almost zero and none of them will be detected as outliers (assuming the threshold is not passed). Sensitivity of the system for outlier detection would be dependent on the defined angle threshold and for different requirements this parameter can be changed without the need of modifying any other component described in this framework. 6.2.1.6 Evaluation Process Evaluation of the proposed framework for detecting defective concrete layers is performed using 100 sample images of the actual printed samples and structures. Three performance metrics including total accuracy, f1 score and miss rate are measured to evaluate the overall performance of the proposed model. 101 | P a g e The last metric, miss rate, is an important factor specially for the purpose of this framework which is detection of bended layers. An important factor to be considered that in this framework is that the cost of false negative outputs from the model are significantly more than the false positive cases. False negatives relate to those frames where a layer defect occurred, but it was ignored by the model while false positives are those frames with no defect but wrongly classified as a positive case. This kind of system is exactly in contradiction with other models such as face detection security systems where false positive is way more costly than false negative outputs. Performance of the proposed framework for layer defect inspection system is validated using various images of actual concrete layers printed by Contour Crafting gantry system. For measuring the performance metrics, 100 images resized to 300x300 with concrete layers are selected and segmented into 50x50 windows. This will provide 36 windows inside each image and 3600 data rows in total. Windows are then labeled as true and false cases with true ones related to bended layers. Using the labels as ground truth, the performance of the model is measured by comparing the outputs with ground truth labels. 6.2.2 Results In this section, results related to concrete layer defect detection are presented. Three sample cases are selected and the outputs from all the framework components are shown in sequence. Finally, the overall performance of the model is measured by means of three evaluation metrics described in section 6.2.1.3 and the results are presented. 102 | P a g e The first example, shown in figure 35, contains 3 concrete layers. There is a defect in the bottom layer with length of more than 100 pixels. Thus, two segments of the image should be highlighted as positive defect cases. The sequence of tasks starts from top left image (original). First, the RGB channel is transformed into single gray channel. Second, concrete texture is removed as shown in the third image from left at the first row. Next, edge detection is applied into the image and the result is shown in the first image from left in the second row. Using the preserved pixels from edge detection, layer lines are formed using Hough transform approach as shown in the middle image in the second row. Finally, using the column-based outlier detection technique, bended concrete layers are detected and highlighted in the original image as shown in the last image of the second row. The same sequence holds for the next examples as well. 103 | P a g e Figure 35 - Sample 1: An image of 4 concrete layers with defect in the bottom layer The second example relates to colorful concrete layers. There are 5 colorful layers printed in this sample and there is a slight defect happened at the left side of the 2 nd layer from bottom. This example also shows that the proposed framework is not limited to the colors used in concrete mixture. Figure 36 shows the sequence of all transformation and analysis steps on second example with the same structure described for the first example. 104 | P a g e Figure 36 - Sample 2: An image of 5 colorful concrete layers with defect in the 2nd bottom layer In the third example 9 concrete layers with a curvy pattern are shown. The third example is a good case showing why the column-based technique is designed here to detect outliers by considering only those lines that fall inside the same image column. From right to left, the average magnitude of the slopes is decreasing. Figure 37 illustrates how an image including 9 curvy concrete layers are preprocessed and analyzed. There is no defect case in this image. 105 | P a g e Figure 37 - Sample 3: An image of 9 curvy concrete layers with no defect Table 12 provides the proposed framework performance for detecting bended layers from 3600 window segmentations in 100 images in total. Table 12 – Proposed Defect Detection Approach Performance Classes Accuracy F1 Score Miss Rate Defected Layers 97.50% 82% 5.83% Non-defected Layers 97.50% 98.65% 2.28% 7. Limitations and Future Research Work The proposed framework presented in the here is related to recognizing concrete layers in an image and then detecting defective layers. Although based on the experiments it seems that prior concrete layer detection model can work with an acceptable performance even when the image is captured from distances more than 1 meter. The second model to detect defect may fail in far distances. One of the main problems is that using the tuned parameters in texture removal and edge detection steps, some 106 | P a g e information is lost when the image is captured from far distances. Figure 38 shows how the information related to several parts of concrete layers are missed from a structure built for space applications at Contour Crafting lab. Figure 38 - Texture removal issue caused by higher distance of image capturing spot While the proposed framework can handle defect detection cases, there are several other problems that can happen to concrete layers. Quality concerns such as concrete layer surface quality, layer compression, layer expansions, ripple effects, accuracy of the as-built printed structure compared to designed model, and other subjects are still the cases that should be added to a comprehensive automated inspection system. 107 | P a g e Although the proposed framework for detecting the concrete layers is the first step in every scenario, detecting a specific problem requires designing a proper approach that can reach a robust performance with high accuracy. Fixing such unexpected issues that are probable to occur during the 3D printing process, should also be accomplished using an automated method. Using new technologies such as drones and other robotic systems can be used to create a totally automated system that can detect quality problems during the process as well as possibly fixing the problems in an automated way. 108 | P a g e Bibliography 1- Khoshnevis, Behrokh. "Automated construction by contour crafting—related robotics and information technologies." Automation in construction 13.1 (2004): 5-19. 2- Khoshnevis, Behrokh, et al. "Mega-scale fabrication by contour crafting." International Journal of Industrial and Systems Engineering 1.3 (2006): 301-320. 3- Khoshnevis, Behrokh, et al. "Lunar contour crafting—a novel technique for ISRU- based habitat development." 43rd AIAA Aerospace Sciences Meeting and Exhibit— Meeting Papers. 2005. 4- Mueller, Robert P., et al. "Automated Additive Construction (AAC) for Earth and Space Using In-situ Resources." Proceedings of the Fifteenth Biennial ASCE Aerospace Division International Conference on Engineering, Science, Construction, and Operations in Challenging Environments (Earth & Space 2016). American Society of Civil Engineers, 2016. 5- Khoshnevis, Behrokh, et al. "Contour crafting simulation plan for lunar settlement infrastructure buildup." ASCE Earth and Space Conference, Pasadena. 2012. 6- Azhar, Salman. "Building information modeling (BIM): Trends, benefits, risks, and challenges for the AEC industry." Leadership and Management in Engineering 11.3 (2011): 241-252. 7- R. Volk, J. Stengel, F. Schultmann, Building Information Modeling (BIM) for existing buildings-Literature review and future needs, Autom. Constr. 38 (2014) 109–127. 8- Dhillon, Raninder Kaur, Mayur Jethwa, and Hardeep Singh Rai. "Extracting Building Data from BIM with IFC." International Journal on Recent Trends in Engineering & Technology 11.2 (2014): 202. 9- Kim, Hyunjoo, et al. "Generating construction schedules through automatic data extraction using open BIM (building information modeling) technology."Automation in Construction 35 (2013): 285-295. 10- Kumar, Srinath S., and Jack CP Cheng. "A BIM-based automated site layout planning framework for congested construction sites." Automation in Construction 59 (2015): 24-37. 11- Zhang, Jing, and Behrokh Khoshnevis. Contour crafting process planning and optimization. Diss. University of Southern California, 2009. 12- Zhang, Jing, and Behrokh Khoshnevis. "Contour Crafting Process Plan Optimization Part I: Single-Nozzle Case." Journal of Industrial and Systems Engineering 4.1 (2010): 33-46. 109 | P a g e 13- Lin, Shen, and Brian W. Kernighan. "An effective heuristic algorithm for the traveling- salesman problem." Operations research 21.2 (1973): 498-516. 14- Helsgaun, Keld. "An effective implementation of the Lin–Kernighan traveling salesman heuristic." European Journal of Operational Research 126.1 (2000): 106-130. 15- Oberg, Erik, et al. Machinery's handbook. Vol. 200. New York: Industrial Press, 2004. 16- Thomas, H. R., Riley, D. R., and Messner, J. I. (2005). “Fundamental principles of site materials management.” J. Constr. Eng. Manage., 131(7), 808–815. 17- Thomas, H. R., Riley, D. R., and Sanvido, V. E. (1999). “Loss of productivity due to delivery methods and weather.” J. Constr. Eng. Manage., 125(1), 39–46. 18- Thomas, H. R., Sanvido, V. E., and Sanders, S. R. (1989). “Impact of material management on productivity—A case study.” J. Constr.Eng. Manage., 115(3), 370– 384. 19- Thomas, H. R., Riley, D. R., and Messner, J. I. (2005). “Fundamental principles of site materials management.” J. Constr. Eng. Manage., 131(7), 808–815. 20- Said, Hisham, and Khaled El-Rayes. "Optimizing material procurement and storage on construction sites." Journal of Construction Engineering and Management 137.6 (2010): 421-431. 21- Li, Heng, and Peter ED Love. "Genetic search for solving construction site-level unequal-area facility layout problems." Automation in Construction 9.2 (2000): 217- 226. 22- Baker, R. C., and Timothy L. Urban. "A deterministic inventory system with an inventory-level-dependent demand rate." Journal of the Operational Research Society (1988): 823-831. 23- Hillier, Frederick S. Introduction to operations research. Tata McGraw-Hill Education, 1995. 24- Sirinaovakul, Booncharoen. "An analysis of computer-aided facility layout techniques." International Journal of Computer Integrated Manufacturing 9.4 (1996): 260-264. 25- Lee, Robert C., and James M. Moore. "CORELAP-computerized relationship layout planning." Journal of Industrial Engineering 18.3 (1967): 195-200. 26- HASSAN, MOHSEN MD, and G. L. Hogg. "On constructing a block layout by graph theory." THE INTERNATIONAL JOURNAL OF PRODUCTION RESEARCH 29.6 (1991): 1263-1278. 27- Buffa, Elwood Spencer, Gordon C. Armour, and Thomas E. Vollmann. Allocating facilities with CRAFT. Harvard University, 1964. 110 | P a g e 28- Yeh, I-Cheng. "Construction-site layout using annealed neural network." Journal of Computing in Civil Engineering 9.3 (1995): 201-208. 29- Goldberg, David E. "E.(1989). Genetic algorithms in search, optimization and machine learning." Reading: Addison-Wesley (1990). 30- Li, Heng, and Peter ED Love. "Genetic search for solving construction site-level unequal-area facility layout problems." Automation in Construction 9.2 (2000): 217- 226. 31- Gubbi, Jayavardhana, et al. "Internet of Things (IoT): A vision, architectural elements, and future directions." Future Generation Computer Systems 29.7 (2013): 1645-1660. 32- Atzori, Luigi, Antonio Iera, and Giacomo Morabito. "The internet of things: A survey." Computer networks 54.15 (2010): 2787-2805. 33- M&E KIOST. “Sample Building with M&E System” 2015, www.mnekiost.blogspot.com/2015/11/1-what-is-building-service.html 34- Future Aerial Innovations (FAI), a drone mapping and survey system, www.futureaerial.com/mapping-survey 35- Equipment World, “Airware’s software launching a drone from a tablet on the jobsite”, www.equipmentworld.com/airware-acquires-caterpillar-partner-redbird-to-offer-full- drone-planning-solution-for-construction-mining 36- El Studio, 10/02/2015, “Mapping 20 years of 3D printing in Architecture”, www.elstudio.nl/?p=1904 37- Becerik-Gerber B., Jazizadeh F., Li N. and Calis G. Application areas and data requirements for BIM-enabled facilities management. Journal of construction engineering and management, 138(3):431-442, 2011. 38- Gu N. and London K. Understanding and facilitating BIM adoption in the AEC industry. Automation in construction, 19(8):988-999, 2010. 39- Chen, Fu-Chen, and Mohammad R. Jahanshahi. "NB-CNN: deep learning-based crack detection using convolutional neural network and naive Bayes data fusion." IEEE Transactions on Industrial Electronics 65.5 (2018): 4392-4400. 40- Zhang, Lei, et al. "Road crack detection using deep convolutional neural network." Image Processing (ICIP), 2016 IEEE International Conference on. IEEE, 2016. 41- Tomasi, Carlo, and Roberto Manduchi. "Bilateral filtering for gray and color images." Computer Vision, 1998. Sixth International Conference on. IEEE, 1998. 111 | P a g e 42- Canny, John. "A computational approach to edge detection." IEEE Transactions on pattern analysis and machine intelligence 6 (1986): 679-698. 43- Link: https://docs.opencv.org/3.0beta/doc/py_tutorials/py_imgproc/py_canny/ (Visited: 07/25/2018) 44- Tao, Xiaodong, and Alwyn Eades. "Errors, artifacts, and improvements in EBSD processing and mapping." Microscopy and Microanalysis 11.1 (2005): 79-87. 45- Badrinarayanan, Vijay, Alex Kendall, and Roberto Cipolla. "Segnet: A deep convolutional encoder-decoder architecture for image segmentation." IEEE transactions on pattern analysis and machine intelligence 39.12 (2017): 2481-2495. 46- Kingma, Diederik P., and Jimmy Ba. "Adam: A method for stochastic optimization." arXiv preprint arXiv:1412.6980 (2014). 47- Abadi, Martín, et al. "Tensorflow: a system for large-scale machine learning." OSDI. Vol. 16. 2016. 48- Chollet, François. "Keras: Deep learning library for theano and tensorflow." URL: https://keras. io/k 7.8 (2015). 49- J. Long, E. Shelhamer, and T. Darrell, “Fully convolutional networks for semantic segmentation,” in CVPR, pp. 3431–3440, 2015 50- LeCun, Yann, Yoshua Bengio, and Geoffrey Hinton. "Deep learning." nature 521.7553 (2015): 436. 112 | P a g e Appendices A.1 IFC Data Model IFC data file formats are mainly divided into three types: 1- .ifc type: the default IFC format 2- .ifcXML type: XML based structure for IFC format which is 3 to 4 times larger than a default .ifc file 3- .ifcZIP type: compressed version of .ifc or .ifcXML files which reduces their size significantly. BIM models can be exported as all three different types. In the presented data extraction system, the default IFC format (.ifc) is used since it has smaller size than ifcXML files and also has a distinct structure to parse compared to ifcZIP files. IFC files can store all kinds of information related to a project such as project location, units and measurements used in project design, facility zones, software version, and geometric and parametric information of all different components that are nested in a BIM model. The first part of IFC files, referred as meta data part, will represent general information and the following parts describe geometric and parametric information of every single element existing in the model as well as their dependencies. A part of a Revit model in IFC format is represented here to illustrate how IFC data structure can represent a 3D model and how the developed IFC Data Parser can extract required information from this file: 113 | P a g e Figure 39 - Revit model in IFC format The first part containing metadata information is called header, similar to many other mime types that have headers, such as HTMLs or XML files. Here is a portion of header part providing information regarding project creation date, data schema, and database local path: HEADER; /****************************************************************************************** * STEP Physical File produced by: The EXPRESS Data Manager * Module: EDMstepFileFactory/EDMstandAlone * Creation date: Tue May 10 18:26:06 2016 * Host: LAPTOP-***** * Database: C:\***\ifc * Database version: 5507 * Database creation date: Tue May 10 18:26:04 2016 * Schema: IFC2X3 * Model: DataRepository.ifc * Model creation date: Tue May 10 18:26:04 2016 114 | P a g e * Header model: DataRepository.ifc_HeaderModel * Header model creation date: Tue May 10 18:26:04 2016 ******************************************************************************************/ FILE_SCHEMA(('IFC2X3')); ENDSEC; The next section of IFC files include geometric and parametric data related to the physical components existing in the model. Moreover, measurement units, defined axes directions, project origin coordinates and other parameters are also represented in this part of IFC files, known as DATA Section. This information can be detected by the specific IFC tag used in the beginning of the line. The first lines are assigned to software name and version that the model is designed. The line below shows that the BIM model used as a test case in this paper is designed in Revit 2017: #5= IFCAPPLICATION(#1,'2017','Autodesk Revit 2017 (ENU)','Revit'); To show an example related to measurement units, the line below represents the conversion ratio between foot and meter defined in the project basics: #44= IFCMEASUREWITHUNIT(IFCRATIOMEASURE(0.3048),#42); This line has another nested element by the end referring to line #42 which is: #42= IFCSIUNIT(*,.LENGTHUNIT.,$,.METRE.); This line represents the conversion ratio between foot and meter set as 0.3048. This is basically the main idea behind IFC data structure, i.e. each line can have its own information directly represented or it can refer to other objects/lines in the file by 115 | P a g e referring to their GUID. GUID is generally is a unique reference number assigned as identifier in the field of computer software. In general, IFC files represent an entity-relationship based model which a hierarchy of all model objects that are connected based on their connectivity and relation in the model. All these entities are categorized into rooted and non-rooted entities that are defined in the model. As an example, ifcWall is an entity that represents existing of a wall component inside the model, in case a window is located in this wall, the hierarchy of the IFC file can precisely describe it. The existence of non-rooted objects is dependent to rooted elements. IFC files architecture can be divided into several main categories such as: 1- IfcObjectDefinition 2- IfcRelationship 3- IfcPropertyDefinition 4- IfcProduct 5- IfcProcesses 6- IfcResources 7- ifcContexts Following parsing the IFC files, the required information such as geometry, material types and parameters are extracted and exported as raw JSON files. The geometry information of each JSON file represents one layer of the model and its ID is showing where this layer is located from bottom since lower layers are printed first in an additive layer fabrication process. A. 2 TSP Algorithm and Conversion Steps of Raw 2D Graphs One of the most successful algorithms proposed for solving tool path optimization systems is Travel Salesman Problem algorithm, known as TSP. The main objective in this algorithm is to visit all the cities given in the problem with the distances 116 | P a g e between each pair of the cities, using the minimum cost path that traverse over all the cities. Assuming the nodes in a graph are labeled as 𝑁 𝑖 where 𝑖 is the index of each node, a TSP problem can be formulated as an Integer Linear Programming (ILP) problem as follow: Equation 14 argmin ∑ 𝑗 ∑ 𝑖 𝑒 𝑖𝑗 𝐶 𝑖𝑗 𝑠𝑢𝑏𝑗𝑒𝑐𝑡𝑒𝑑 𝑡𝑜 : 𝑖 , 𝑗 ∈ {1,2, … , 𝑛 }𝑤 ℎ𝑒𝑟𝑒 𝑛 𝑖𝑠 𝑡𝑜𝑡𝑎𝑙 𝑛𝑢𝑚𝑏𝑒𝑟 𝑜𝑓 𝑛𝑜𝑑𝑒𝑠 𝑖𝑛 𝑡 ℎ𝑒 𝑝𝑟𝑜𝑏𝑙𝑒𝑚 𝐶 𝑖𝑗 = 0 𝑖𝑓 𝑖 = 𝑗 𝐶 𝑖𝑗 = 𝐶 𝐷 𝑓𝑜𝑟 𝑑𝑒𝑝𝑜𝑠𝑖𝑡𝑖𝑜𝑛 𝑒𝑑𝑔𝑒𝑠 𝑤 ℎ𝑒𝑟𝑒 𝑖 𝑎𝑛𝑑 𝑗 𝑎𝑟𝑒 𝑡 ℎ𝑒 𝑒𝑛𝑑 𝑛𝑜𝑑𝑒𝑠 𝑏𝑒𝑙𝑜𝑛𝑔𝑖𝑛𝑔 𝑡𝑜 𝑎 𝑑𝑒𝑝𝑜 𝑠 𝑖𝑡𝑖𝑜𝑛 𝑒𝑑𝑔𝑒 𝐶 𝑖𝑗 = 𝑅 (𝐴𝑟𝑐 𝐶𝑜𝑠 (𝛼 𝑖𝑗 )) 𝑖𝑓 𝑖 𝑎𝑛𝑑 𝑗 𝑎𝑟𝑒 𝑡𝑤𝑜 𝑎𝑑𝑗𝑎𝑐𝑒𝑛𝑡 𝑛𝑜𝑑𝑒𝑠 (𝑎𝑡 𝑙𝑒𝑎𝑠𝑡 𝑜𝑛𝑒 𝑛𝑜𝑑𝑒 𝑖𝑠 𝑎 𝑎𝑟𝑡𝑖𝑓𝑖𝑐𝑖𝑎𝑙 𝑛𝑜𝑑𝑒 ) 𝑅 (𝐴𝑟𝑐 𝐶𝑜𝑠 (𝛼 𝑖𝑗 )) 𝑖𝑠 𝑡 ℎ𝑒 𝑓𝑢𝑛𝑐𝑡𝑖𝑜𝑛 𝑐𝑎𝑙𝑐 𝑢𝑙𝑎𝑡𝑖𝑛𝑔 𝑡 ℎ𝑒 𝑟𝑜𝑡𝑎𝑡𝑖𝑜𝑛 𝑐𝑜𝑠𝑡 𝑏𝑒𝑡𝑤𝑒𝑒𝑛 𝑡𝑤𝑜 𝑒𝑑𝑔𝑒𝑠 ℎ𝑎𝑣𝑖𝑛𝑔 𝑖 𝑎𝑛𝑑 𝑗 𝑎𝑠 𝑡 ℎ𝑒𝑖𝑟 𝑒𝑛𝑑 𝑝𝑜𝑖𝑛𝑡𝑠 𝑒 𝑖𝑗 = { 1 𝑖𝑓 𝑡 ℎ𝑒 𝑠𝑜𝑙𝑢𝑡𝑖𝑜𝑛 𝑝𝑎𝑡 ℎ 𝑡𝑟𝑎𝑣𝑒𝑟𝑠𝑒𝑠 𝑡 ℎ𝑒 𝑒𝑑 𝑔 𝑒 𝑤𝑖𝑡 ℎ 𝑖 𝑎𝑛𝑑 𝑗 𝑒𝑛𝑑 𝑛𝑜𝑑𝑒𝑠 0 𝑖𝑓 𝑡 ℎ𝑒 𝑒𝑑𝑔𝑒 𝑤𝑖𝑡 ℎ 𝑖 𝑎𝑛𝑑 𝑗 𝑒𝑛𝑑 𝑛𝑜𝑑𝑒𝑠 𝑖𝑠 𝑛𝑜𝑡 𝑝𝑟𝑒𝑠𝑒𝑛𝑡𝑒𝑑 𝑖𝑛 𝑡 ℎ𝑒 𝑠𝑜𝑙𝑢𝑡𝑖𝑜𝑛 𝑝𝑎𝑡 ℎ ∑ 𝑒 𝑖𝑗 𝑛 𝑖 =1 = 1 𝑖𝑛 𝑡 ℎ𝑒 𝑠𝑜𝑙𝑢𝑡𝑖𝑜𝑛 𝑝𝑎𝑡 ℎ, 𝑜𝑛𝑙𝑦 𝑜 𝑛𝑒 𝑜𝑡 ℎ𝑒𝑟 𝑛𝑜𝑑𝑒 𝑤𝑖𝑙𝑙 𝑏𝑒 𝑣𝑖𝑠𝑖𝑡𝑒𝑑 𝑓𝑟𝑜𝑚 𝑛𝑜𝑑𝑒 𝑖 ∑ 𝑒 𝑖𝑗 𝑛 𝑗 =1 = 1 𝑖𝑛 𝑡 ℎ𝑒 𝑠𝑜𝑙𝑢𝑡𝑖𝑜𝑛 𝑝𝑎𝑡 ℎ, 𝑒𝑎𝑐 ℎ 𝑛𝑜𝑑𝑒 𝑤𝑖𝑙𝑙 𝑏𝑒 𝑣𝑖𝑠𝑖𝑡𝑒𝑑 𝑓𝑟𝑜𝑚 𝑜𝑛𝑙𝑦 𝑜𝑛𝑒 𝑜𝑡 ℎ𝑒𝑟 𝑛𝑜𝑑𝑒 In the previous section, the raw graphs are converted to the new graphs that can be solved using TSP algorithm while it is guaranteed that the deposition edges are included in the final solution. Using the results from literature, Lin-Kernighan (LK) [13] TSP algorithm is considered for solving toolpath optimization problem in this research because of its better results compared other heuristic based approaches 117 | P a g e used to solve TSP problems to other algorithms such as CPP. LK algorithm is implemented based on aforementioned local optimization heuristics. In the original LK algorithm, for problems with less than 50 nodes, optimum solution was found almost 100%. For the ones with more than 100 nodes, this probability was between 20 and 30 percent while running the algorithm several times could increase this probability close to 100%. However, a modified version of LK algorithm presented by Helsgaun [14], there was a significant improvement in finding the optimum solutions in a reasonable short time. Given the distances between all the nodes, space requirements are linear or 𝑂 (𝑛 ) in this approach and the running time which is the main bottle neck of all TSP algorithms is approximately 𝑂 (𝑛 2.2 ) which can solve problems with more than 100k nodes. Since the graphs resulted from building layouts have usually less than 250 nodes, this algorithm can work very well for finding the optimum solutions in tool path optimization problems related to building graphs. Consequently, LK algorithm using Helsgaun implementation (known as LKH algorithm) is selected to be used for tool path optimization algorithm in this research. Several constraints and conditions should be considered while the optimum tool-path is generated: 1- Contour Crafting toolpath problem is a symmetric TSP problem since the distance between each pair of nodes is the same in both directions. 2- Each layer is extruded completely before its upper layer to avoid any collision between the nozzle and the extruded layers 118 | P a g e 3- Idle time between lower and upper layers should be in a specific time interval which is bounded by minimum time needed for the material strength to support the upper layer and the material setting time. In other words, the amount of time required for the nozzle to come back to the same edge and print the next layer should be set based on the material (e.g. concrete) used in the process to avoid any problem for layers’ adherence and support. 4- The problem is considered to be solved using the single nozzle system, in which the building boundaries are fitted within the machine size in all xyz directions. The LKH TSP algorithm used in this research, has been implemented in a virtual server where it can be accessed through any browser with no system setup required. The input raw graph in given as a JSON file and the system will do both transferring the raw graph and solving the TSP problem automatically. In the first step, the raw graph for the test case shown in figure 8 is transferred to LKH TSP problem based on the described steps in this section. LKH TSP algorithm is then applied to the new converted graph. The output in JSON format is representing a cycle which is a sequence of all the nodes in the converted graph starting from node 1 (usually the bottom-left corner of the building) and ending in node 1 with minimum cost and including all deposition edges that are required to be traversed and visited in the original problem. The results are presented in the next part. The last step after finding the optimum toolpath is to generate G-code for the machine. In general, it is a code that informs the machine what type of actions it should take step by step to complete the considered process [15]. The input file for this part is a JSON graph including the sequence of all the nodes from optimal solution along with the category of each edge determining what type of task the machine should perform to execute 119 | P a g e this part of the graph. The G-code generator should know if it should print (deposition edge), move to another node without printing (idle or rotation edge) or go to the next layer by moving in z direction. The scope of the research is not to dive into all the details for G-code generation since it is a common task in robotic field, but main tasks of the machine that are related to 3D printing system are mentioned. Following constraints and conditions in Contour Crafting process effect this transformation and calculations: Full Graph Generation & Idle Edges: The nozzle can travel from one node to any other node in the same layer. This means that the raw graph should be converted into a full graph where all the nodes are connected to each other. In this graph, there are two types of edges, deposition and idle. Deposition edges are the ones existing in the raw graph. Idle edges are new added edges in which the nozzle travels from one vertex to another one without printing any object. Since the shortest path from one node to another node is the straight line connecting these two nodes, the idle edge between the nodes is the Euclidean distance calculated based on their x-y coordinates since they are assumed to be in the same layer with the same z coordinate. Idle edge between two arbitrary nodes is defined as below: 𝐼𝑑𝑙𝑒 𝐸𝑑𝑔𝑒 𝑐𝑜𝑠𝑡 = √ ((𝑋 1 − 𝑋 2 )) 2 + ((𝑌 1 − 𝑌 2 )) 2 Equation 15 Rotation costs and Artificial Nodes: The rotation cost is defined as the time needed for the nozzle to rotate to be able to deposit the next edge. As an example, in an orthogonal wall intersection, when the 120 | P a g e nozzle finished one wall and wants to start printing the next wall, it has to rotate 90 degrees first and then be able to start printing. Assuming that the rotation is feasible and equal in both clockwise and counter clockwise directions, we can take into account the rotation costs by adding artificial nodes to the graph. Consider the following intersection of three walls: Figure 40 - Non-Artificial Nodes Considered for a Wall intersection In the raw graph, this intersection with 3 walls will have 4 nodes which is 1 node for the intersection and 3 nodes for end of each wall. Now, when rotation costs are considered in this system, the raw graph should be converted to the following graph having two extra artificial nodes added in the point of intersection: Figure 41 – Artificial Nodes for a Wall Intersection with Orthogonal Rotations 121 | P a g e As shown in figure 41, two artificial nodes are added in intersection to be able to assign rotation costs from each wall to the other. As an instance, the rotation from Node 2 to 2-A is set as zero since they are both in the same straight direction. But the other two rotation costs are equal to the time considered for 90 degree rotation (i.e. calculated using rotation function R(90)). In any part of the raw graph where two or more deposition edges are intersecting, the following formula applies to generate artificial nodes: 𝑁𝑢𝑚𝑏𝑒𝑟 𝑜𝑓 𝐴𝑟𝑡𝑖𝑓𝑖𝑐𝑖𝑎𝑙 𝑁𝑜𝑑𝑒𝑠 𝑅𝑒𝑞𝑢𝑖𝑟𝑒𝑑 = 𝑁𝑢𝑚𝑏𝑒𝑟 𝑜𝑓 𝑊𝑎𝑙𝑙𝑠 𝐼𝑛𝑡𝑒𝑟𝑠𝑒𝑐𝑡𝑖𝑛𝑔 − 1 Equation 16 The -1 in this formula is because of the fact that there is always at least one node considered at each intersection in the raw graph. Changing the Cost of Idle Edges: There are several factors that are considered in this part to change the cost of idle edges. This changing is applied to consider other parameters existing in the system such as the difference in nozzle travel speed in X or Y directions or the difference of energy consumption rate for traveling in x or y directions. Rotations are essentially important since the nozzle direction for deposition edges is aligned with the edge orientation. One problem we have is that these rotations cannot be predefined and given in the current system. It is a dynamic value which is dependent to the orientation of the current edge that the nozzle is traversing and also the angle it should rotate to align the nozzle head with the next edge. In other words, each time, depending on the sequence of the edges, we should find: 122 | P a g e cos −1 𝑒 1 . 𝑒 2 |𝑒 1 |. |𝑒 2 | Equation 17 𝑤 ℎ𝑒𝑟𝑒 𝑒 1 𝑎𝑛𝑑 𝑒 2 𝑎𝑟𝑒 𝑡 ℎ𝑒 𝑡𝑤𝑜 𝑎𝑑𝑗𝑎𝑐𝑒𝑛𝑡 𝑒𝑑𝑔𝑒𝑠 𝑖𝑛 𝑡 ℎ𝑒 𝑡𝑟𝑎𝑣𝑒𝑟𝑠𝑖𝑛𝑔 𝑠𝑒𝑞𝑢𝑒𝑛𝑐𝑒 𝑜𝑓 𝑡 ℎ𝑒 𝑠𝑜𝑙𝑢𝑡𝑖𝑜𝑛 This will give the angle between two edges (𝑒 1 𝑎𝑛𝑑 𝑒 2 ). Considering the rotation speed as a variable in our system, we are able to find the time required for each different angle. This is the cost that is assigned between artificial nodes at each intersection based on their direction towards each other. For the nozzle speed, assuming that the ratio of speed in y direction (𝑉 𝑌 ) over speed in x direction (𝑉 𝑋 ) is defined by µ, the idle edge costs can between two nodes 𝑁 𝑖 (𝑋 𝑖 , 𝑌 𝑖 ) to 𝑁 𝑗 (𝑋 𝑗 , 𝑌 𝑗 ) is calculated as below: Modified Idle Edge cost = √ (µ(𝑋 𝑖 − 𝑋 𝑗 )) 2 + ((𝑌 𝑖 − 𝑌 𝑗 )) 2 µ = 𝑉 𝑌 𝑉 𝑋 Equation 18 𝑤 ℎ𝑒𝑟𝑒 : 𝑉 𝑋 = 𝑁𝑜𝑧𝑙𝑙𝑒 𝑠𝑝𝑒𝑒𝑑 𝑡𝑜𝑤𝑎𝑟𝑑𝑠 𝑋 𝑎𝑥𝑖𝑠 𝑉 𝑌 = 𝑁 𝑜𝑧𝑙𝑙𝑒 𝑠𝑝𝑒𝑒𝑑 𝑡𝑜𝑤𝑎𝑟𝑑𝑠 𝑌 𝑎𝑥𝑖𝑠 Same modification can be applied to idle edges for considering the effect of energy consumption of robot in different directions. In general, this factor will change the scale of the whole graph in either x or y direction. The difference between the speed and energy using overall gantry and the nozzle assembly is shown in figure 42. The nozzle assembly is supposed to use less energy than moving in the gantry direction. 123 | P a g e Figure 42 – Directions of Printing in Contour Crafting System [11] Changing the Cost of Deposition Edges: The last important change and modification that should be applied to the raw graph structure before sending it to the tool-path optimizer system, is altering the cost of deposition edges, those edges that should definitely exist in the final solution of the TSP toolpath optimization algorithm since they are belonging to the edges that should be printed. No matter what optimal path is, nozzle should traverse all the deposition edges in the system for one and only one time and they cannot be omitted from the solution. In other words, idle edges are the only edges that play role in optimum solution and distinguishing one solution from another one since deposition edges are existing and considered as constant cost that are in common in all solutions. The final optimum tool-path is the one having minimum nozzle idle time. A.3 G-Code Generation Table 13 is providing the basic commands used to do the main tasks described for a 3D printing machine: 124 | P a g e Table 13 - G-code Main Commands for 3D Printing Process Command Parameters Related to G0 XYZ Position Deposition Edge G1 XYZ Position and Feed Idle and Rotation Edge G4 time Waiting G20 none Setting Units to Inch G28 none Setting Units to mm Here is the sudo code used for generating the G-code from the toolpath provided by toolpath optimization algorithm: Algorithm gCodeGenerator(Queue<JSON> graphs){ //input: queue of layers //output: G-code Commands for 3D printing the layers (tasks) String tasks; // all commands will be stored here while(graphs!= null){ currentGraph = graphs.remove(); //This will give the next Layer //inserted in the queue //Which are sorted from bottom to top //It will also reduce the graphs queue //size by one tasks += parseGraph(currentGraph); //ParseGraph is an algorithm in //which proper G-code commands are inserted //based on the JSON graph that is parsed here tasks += stepUp(); //after each layer, the machine should stepUp by delta h //to be able to start the next layer without collision //stepUp() will add the required command to the tasks string } } Figure 43 - Pseudocode for generating robot G-Code

Abstract (if available)

Abstract Automated construction through concrete 3D printing is deemed as a revolution in construction industry. This novel idea of scaling up additive manufacturing techniques for automated building construction has been topic of academic and industrial research for several years. Many advantages are offered by automated building construction such as superior construction speed and higher degree of customization. There are considerable efforts on use of robots for various construction purposes such as pre-fabrication or non-load-bearing structures. However, the focus of this thesis is on automated in-situ building construction using additive manufacturing techniques. ❧ Contour Crafting (CC) is the first additive fabrication technology developed for in-situ construction of custom-designed structures. It can radically reduce the construction cost by reducing the workforce required for the construction process. Various advantages such as better surface quality, higher fabrication speed and broader choice of materials are offered by the CC technology compared to other layered fabrication systems. CC is also considered as one of the most promising approaches for construction of human settlements on other planets such as the Moon and Mars. ❧ A review of past and ongoing projects reveals that so far most of the efforts in this area are focused on robot development and hardware improvement, as well as material performance. However, new software platforms are also highly essential for realizing a fully automated and reliable building construction system. Hardware and material related topics have been improved extensively over the past years while software related problems and needs are less discussed. ❧ The main objective of this thesis is to provide a software platform for large-scale robotic 3D printing process using layer by layer additive manufacturing system. The software is named Planning and Operations Control Software for Automated Construction (POCSAC). The thesis is divided into two major chapters that covers the preprocess and 3D printing process stages of an automated large-scale 3D printing system. ❧ In chapter one, a proposed framework presents details of interoperation between different components of an automated construction system and BIM platform such that maximum benefit is realized through synergy of the two technologies. Chapter one parts that cover preprocess tasks are: a) Integration of Building Information Modeling (BIM) for robotic construction, b) Material Procurement and site logistics planning for an automated construction project. ❧ In the second chapter, an automated inspection system is proposed to be used during the 3d printing process of concrete layers. The outcomes from the second chapter are related to distinguishing concrete layers from surrounding objects in an image frame and then detecting defects of concrete layers which are then inserted back into the construction report generated from the BIM integrated software component proposed in the first chapter.

Linked assets

University of Southern California Dissertations and Theses

Conceptually similar

PDF

Contour Crafting process planning and optimization

PDF

Contour crafting construction with sulfur concrete

PDF

Thermal and deformation analysis of multiphase sulfur concrete extrusion for planetary construction

PDF

Evaluation of interlayer bonding strength in contour crafted structures: experimental and statistical approach

PDF

Mixture characterization and real-time extrusion quality monitoring for construction-scale 3D printing (Contour Crafting)

PDF

3D printing of polymeric parts using Selective Separation Shaping (SSS)

PDF

Analyses of strength of layered structures fabricated by Contour Crafting

PDF

AI-driven experimental design for learning of process parameter models for robotic processing applications

PDF

Selective Separation Shaping (SSS): large scale cementitious fabrication potentials

PDF

Hybrid vat photopolymerization processes for viscous photocurable and non-photocurable materials

PDF

Energy control and material deposition methods for fast fabrication with high surface quality in additive manufacturing using photo-polymerization

PDF

Algorithms and systems for continual robot learning

PDF

Using building information modeling with augmented reality: visualizing and editing MEP systems with a mobile augmented reality application

PDF

Hybrid vat photopolymerization: methods and systems

PDF

Data-driven 3D hair digitization

$Metasurfaces in 3D applications: multiscale stereolithography and inverse design of diffractive optical elements for structured light$

PDF

Metasurfaces in 3D applications: multiscale stereolithography and inverse design of diffractive optical elements for structured light

PDF

Statistical modeling and machine learning for shape accuracy control in additive manufacturing

PDF

In-situ quality assessment of scan data for as-built models using building-specific geometric features

PDF

Machine learning methods for 2D/3D shape retrieval and classification

PDF

Face recognition and 3D face modeling from images in the wild

Asset Metadata

Creator Davtalab, Omid (author)

Core Title A data driven software platform for process automation, planning and inspection of Contour Crafting large-scale robotic 3D printing system

School Viterbi School of Engineering

Degree Doctor of Philosophy

Degree Program Civil Engineering

Publication Date 10/17/2018

Defense Date 08/21/2018

Publisher University of Southern California (original), University of Southern California. Libraries (digital)

Tag 3D printing,building information modeling (BIM),contour crafting,deep learning,image processing,machine learning,OAI-PMH Harvest,robot toolpath optimization,robotic,site layout planning,software

Format application/pdf (imt)

Language English

Contributor Electronically uploaded by the author (provenance)

Advisor Khoshnevis, Behrokh (committee chair), Lee, Vincent (committee member), Razaviyayn, Meisam (committee member)

Creator Email davtalab@usc.edu,omid.davtalab@gmail.com

Permanent Link (DOI) https://doi.org/10.25549/usctheses-c89-81702

Unique identifier UC11671708

Identifier etd-DavtalabOm-6867.pdf (filename),usctheses-c89-81702 (legacy record id)

Legacy Identifier etd-DavtalabOm-6867.pdf

Dmrecord 81702

Document Type Dissertation

Format application/pdf (imt)

Rights Davtalab, Omid

Type texts

Source University of Southern California (contributing entity), University of Southern California Dissertations and Theses (collection)

Access Conditions The author retains rights to his/her dissertation, thesis or other graduate work according to U.S. copyright law. Electronic access is being provided by the USC Libraries in agreement with the a...

Repository Name University of Southern California Digital Library

Repository Location USC Digital Library, University of Southern California, University Park Campus MC 2810, 3434 South Grand Avenue, 2nd Floor, Los Angeles, California 90089-2810, USA