University of Southern California Dissertations and Theses

Design and implementation of an enterprise spatial raster management system

(USC Thesis Other)

Design and implementation of an enterprise spatial raster management system

PDF

Download a page range

Download transcript

Copy asset link

Request this asset

Transcript (if available)

Content Design and Implementation of an Enterprise Spatial Raster Management System by David Robert Scobie A Thesis Presented to the Faculty of the USC Graduate School University of Southern California In Partial Fulfillment of the Requirements for the Degree Master of Science (Geographic Information Science and Technology) May 2016 Copyright © 2016 by David Robert Scobie iii Table of Contents List of Figures ................................................................................................................................ vi List of Tables ................................................................................................................................ vii Acknowledgements ...................................................................................................................... viii List of Abbreviations ..................................................................................................................... ix Abstract ........................................................................................................................................... x Chapter 1 Introduction .................................................................................................................... 1 1.1 Motivation ............................................................................................................................2 1.2 Short Term Data Management System Objectives ..............................................................3 1.3 Long Term Data Management Objectives ...........................................................................4 1.4 Data Scope of the Project .....................................................................................................6 1.5 Structure of this Document ..................................................................................................7 Chapter 2 Background .................................................................................................................... 8 2.1 Advancements in Raster Data Technology ..........................................................................8 2.1.1. Raster data services ....................................................................................................9 2.1.2. Raster compression ..................................................................................................10 2.1.3. Esri’s mosaic data model .........................................................................................10 2.2 Enterprise Spatial Data Management .................................................................................16 2.3 Foundations for Enterprise Raster Data Management Systems .........................................17 2.3.1. Emergence of tiled database raster management systems .......................................17 2.3.2. Example of a web-enabled database raster management systems ...........................18 2.4 Summary ............................................................................................................................20 Chapter 3 High Level Design Components .................................................................................. 21 3.1 Software Requirements ......................................................................................................21 3.1.1. Geodatabase model ..................................................................................................22 3.2 Hardware Architecture Requirements and Overview ........................................................23 3.3 Raster Data Types and Subtypes ........................................................................................24 3.3.1. Imagery ....................................................................................................................24 3.3.2. Elevation data...........................................................................................................25 3.3.3. Thematic data ...........................................................................................................25 iv 3.4 Data Acquisition and Dissemination .................................................................................26 3.4.1. Data dissemination protocol ....................................................................................28 3.4.2. Web server clients ....................................................................................................28 3.4.1. Web server configuration .........................................................................................29 3.5 Summary ............................................................................................................................30 Chapter 4 Low-Level Design Components ................................................................................... 31 4.1 Source Data Management and Processing .........................................................................31 4.1.1. Source data structure and naming conventions ........................................................31 4.2 Mosaic Dataset Management and Processing ....................................................................33 4.2.1. Mosaic data structure and naming conventions .......................................................33 4.2.2. Creating a source mosaic dataset .............................................................................35 4.2.3. Mosaic data design template and schema ................................................................36 4.2.4. Referenced and derived mosaics ..............................................................................37 4.3 Data Updates and Maintenance..........................................................................................38 4.4 Project Workflow Summary ..............................................................................................38 4.5 Summary ............................................................................................................................40 Chapter 5 Technology Assessment ............................................................................................... 42 5.1 Raster Workflow Assessments ..........................................................................................42 5.1.1. Elevation and LiDAR workflow enhancements ......................................................42 5.1.2. Thematic workflow enhancements: Alberta Wet Area Mapping ............................44 5.2 Data Management and Interface Appraisal ........................................................................47 5.2.1. Web service management and dissemination ..........................................................48 5.2.2. Data administration ..................................................................................................49 5.3 Hardware and Data Performance Assessment ...................................................................51 5.3.1. Display Performance ................................................................................................51 5.4 Achievement of Short Term Data Management Objectives ..............................................52 5.4.1. Successful implementation of a database related raster management technology ..52 5.4.2. Established raster data management structure and guidelines .................................52 5.4.3. Accommodate server based raster dissemination requirements ..............................53 5.4.4. Consolidate all of Talisman Energy’s North American raster data in a single location ...................................................................................................................54 v 5.4.5. Confirm that workflow improvements related to these technology upgrades satisfy objectives ...............................................................................................................56 5.5 Achievement of Long Term Data Management Objectives ..............................................57 5.5.1. Prepare for Data Consolidation for Repsol Integration ...........................................57 5.5.2. Adapt to emerging technologies and business objectives ........................................57 5.6 Summary ............................................................................................................................57 Chapter 6 Conclusion .................................................................................................................... 60 6.1 Future Tasks to be Completed and Proposed Enhancements ............................................60 6.2 Final Comments .................................................................................................................62 REFERENCES ............................................................................................................................. 63 vi List of Figures Figure 1 GIS patterns in utilities and energy .................................................................................. 1 Figure 2 Global Repsol Operations. Repsol’s new assets after the acquisition of Talisman Energy ..................................................................................................................................... 5 Figure 3 The Mosaic Data Model.. ............................................................................................... 11 Figure 4 Mosaic Dataset Raster Functions ................................................................................... 14 Figure 5 Image Analysis Window. ............................................................................................... 16 Figure 6 TEODOOR System user data access methods. ............................................................. 19 Figure 7 Web-based spatial raster data servers and clients within Talisman Energy. .................. 29 Figure 8 Source Data Structure and Hierarchy.. ........................................................................... 34 Figure 9 Mosaic Dataset Folder Hierarchy. .................................................................................. 35 Figure 10 Raster Data Life-Cycle at Talisman Energy ................................................................. 41 Figure 11 Raw Thematic Data ...................................................................................................... 45 Figure 12 Processed Thematic Data. ............................................................................................ 46 Figure 13 Mosaic Dataset Interface.. ............................................................................................ 48 Figure 14 Image Service Deployment and Access. ...................................................................... 49 vii List of Tables Table 1 Mosaic dataset tables. ...................................................................................................... 12 Table 2 Software used to access process and disseminate spatial raster data at Talisman Energy ................................................................................................................................... 22 Table 3 Talisman Energy’s ArcGIS 10.2 Desktop Hardware Specifications ............................... 23 Table 4 Common business considerations relating to spatial raster data at Talisman Energy .... 27 Table 5 Talisman Energy multi-tier server environment for raster data services ......................... 30 Table 6 Source data folder hierarchy ............................................................................................ 32 Table 7 Mosaic dataset attribute template for enterprise raster data sources at Talisman Energy ................................................................................................................................... 36 Table 8 Project milestones ............................................................................................................ 39 Table 9 Implementation steps completed ..................................................................................... 40 Table 10 Comparison of traditional and mosaic dataset elevation workflow. .............................. 43 Table 11 Thematic raster processing workflow comparison ........................................................ 47 Table 12 Examples of stored source raster data using a defined structure ................................... 50 viii Acknowledgements I am grateful to all of the teachers in life who have helped build my passion to learn. I would like to thank my employer, Talisman Energy, who allowed me to write a thesis using cutting edge hardware and software. I am grateful for my team lead, Roland Schwietz, who has always been supportive of my pursuits and has been an amazing mentor for me in my career and life. I am also grateful to Nola Lewis who provided me with a life-changing opportunity allowing work on a diverse range of rewarding projects within the Geospatial services group at Talisman Energy. I would also like to thank the USC faculty and in particular to all the professors who have taught me so much and inspired me to continue my journey of learning. Specifically, I would like to thank Karen Kemp for her patience during this project. She has provided great personal and professional mentorship. ix List of Abbreviations API Application Programing Interface DBMS Database Management System EDM Enterprise Data Management EULA End User Licence Agreement GIS Geographic information system GISci Geographic information science O&G Oil and Gas SDK System Development Kit SOS Sensor Observation Service SSI Spatial Sciences Institute UAV Unmanned Aerial Vehicle USC University of Southern California WAM Wet Area Mapping WMS Web Mapping Service x Abstract Spatial data is used in many industries and it is common for large organizations to internally manage and employ spatial data to assist with operations. The intent of this thesis is to identify and accommodate existing technical architecture strategies, user requirements, and platform software in order to design an enterprise data management solution specific to raster spatial data used within an international oil and gas exploration organization. Company specific data management objectives are identified and assessed in terms of their viability with respect to data acquisition, management, and dissemination. Raster data model improvements were considered necessary to provide web and third party applications with centrally hosted data. Raster technology innovations are examined within the context of enterprise data management practices, providing use case examples that leverage database raster management technology. High and low-level components for the design of the enterprise data management solution are described in detail. Components specific to the mosaic data model and its associated data dependencies are detailed as low-level design components. Subsequent to the implementation of the system, a technology assessment was undertaken to identify the raster workflow benefits associated with raster data model enhancements. Critical analysis of both the strengths and weaknesses of the system with respect to current and future business operations is provided. In summary, the original objectives are revisited to assess their achievement and future considerations for the continuing management of the system are investigated. 1 Introduction Chapter 1 Geographic Information Systems (GIS) software and data can support decision making across industries using a vast range of technologies and solutions. GIS functions within an organization typically aim to optimize operational development and management using a combination of data sources. The application of spatial software and data ranges from simple visualizations to complex outputs to generated using analytical processes. In the oil and gas (O&G) industry, spatial data can be used to optimize operations and planning activities by supporting the dissemination of information and an extensive range of surface and subsurface operations, as illustrated by Figure 1 (Meehan et al. 2012). Successful corporate GIS functions, including those in O&G industry, depend on efficient and logical management of organizational spatial data. Figure 1 GIS patterns in utilities and energy. Source: Meehan et al. 2012 This document provides a summary of the complexities and benefits associated with the implementation of a database related raster management system required upon completion of a 2 data and software upgrade executed at Talisman Energy in 2014. On January 1, 2016, Talisman Energy Inc. changed its name to Repsol Oil & Gas Canada Inc. For the purposes of this thesis, all references to Talisman Energy Inc. relate to the organization prior to January 1, 2016. This introductory chapter examines project motivation and scope and the concept of organization-specific data management. 1.1 Motivation Talisman Energy Inc., a subsidiary of Repsol, is an integrated global energy company that uses spatial data and software to support a variety of global operations. Successful GIS integration at Talisman Energy requires the development of innovative data acquisition, management and dissemination solutions. Significant GIS activity at Talisman Energy is linked to mapping and analysis functions used to support decision-making across multiple disciplines and business units including exploration, development and operational activities. In order to communicate the project motivation, it is also important to understand how Geographic Information Systems (GIS) are used within Talisman Energy and how spatial raster data is leveraged throughout the organization. Prior to the implementation of the system described in this document, a widely distributed archive of GIS data at Talisman Energy included over 10 TB of data spanning multiple continents. The lack of data management workflows and structure related to this distributed raster data archive had become increasingly problematic. A large amount of the raster data related to both past and active projects had not been adequately maintained or inventoried. Consequently, spatial raster data within the organization generally lacked metadata and was stored across multiple network locations. This often restricted data discovery and user access. 3 Significant operational optimization within a GIS environment can be gained when data is managed consistently within a formal structure. Thus, determining a management strategy for raster data sources specific to business requirements should aim to satisfy both short and long term company objectives. Recent hardware infrastructure upgrades at Talisman Energy were aimed at increasing capacity while improving data access and dissemination methods. Successful system infrastructure upgrades required data migration to ensure raster data is compatible with upgraded software and hardware. Specifically, raster data software compatibility required the adoption of a database-related raster management technology using a mosaic data model prior to the deployment of web-based services. The overall goal of this project is to provide a synopsis of the benefits and components of the design and implementation of a medium- to large-scale database-related raster management strategy and system. As this report shows, implementation of this mosaic data model structure and technology does provide a framework within which to implement scalable raster management strategies that address current and long-term business objectives. 1.2 Short Term Data Management System Objectives Before the integration of operations of Talisman Energy and Repsol began, an assessment of spatial raster data at Talisman Energy concluded that data had been stored in many folder locations and in many instances access to data was restricted. Raster source data is typically provided by vendors with unique file delivery methods, a situation that leads to proliferation of a disjointed data archive. A key goal of Talisman Energy’s revised spatial raster management strategy was to ensure that all corporately accessible raster data, regardless of extent, be managed and distributed as a single unit. This would lead to a data management strategy that reduces 4 maintenance costs while avoiding data duplication and reducing overhead relating to data management. As a further benefit, the need to configure the data to an internal and centralized location would meet business requirements where there is a growing need for web- and server- based applications. Thus, the short term objectives of this project that address immediate needs within the Talisman Energy organization were:  Successfully implement a database-related raster management technology.  Accommodate web-based raster data dissemination requirements.  Establish raster data management structure and guidelines.  Consolidate all of Talisman Energy’s North American raster data in a single location.  Accommodate server-based raster dissemination requirements.  Confirm that workflow improvements related to these technology upgrades satisfy objectives. 1.3 Long Term Data Management Objectives In March 2015, Repsol completed the acquisition of Talisman Energy to become one of the largest oil and gas exploration, production and downstream companies worldwide. Repsol places an emphasis on technological innovation to improve efficiencies towards a sustainable energy mode and uses GIS technology to support operations. The global presence of Talisman Energy is complimentary to Repsol’s other international operations such that the integrated company now has a presence in over 40 countries (Figure 2). 5 Figure 2 Global Repsol Operations. Repsol’s new assets after the acquisition of Talisman Energy To begin integration of the companies, a new organizational structure was implemented with defined objectives including the alignment of respective assets portfolios into a one- company model. Other key aims include the alignment of structure and operation of corporate functions while also focusing on long-term technological process and business sustainability (Repsol 2015). In preparation for GIS integration between legacy Talisman and Repsol systems, attention was placed on ensuring data was prepared for company integration activities. Thus, to align with key organizational goals specific to the Talisman Energy/Repsol company integration and following best practices for enterprise GIS implementations, it was necessary to consolidate and document all raster data while also creating a functional raster catalog that was accessible across the integrated company. Thus, the long term objectives for the integrated raster data management system are: 6  Consolidate and catalog remaining international Talisman Energy raster data sources in preparation for Talisman/Repsol GIS integration.  Consider and adapt to emerging raster technologies in order to optimize data access, use, and storage. 1.4 Data Scope of the Project Project management practices direct that it is important to define the project scope before implementation takes place (Rifaie et al. 2008). Therefore, the scope of the design, implementation, and workflow assessments in this project are focused on enterprise raster data architecture for the North American division of Talisman Energy where a wide range of raster data types, including imagery, elevation and thematic raster data with varying spatial-temporal characteristics, are used. The key objective of implementing an enterprise spatial raster management strategy is to ensure that all corporately accessible data, regardless of extent, are managed and distributed as a single unit The system design and implementation outlined in this document incorporated all spatial raster datasets that were deemed to be used company-wide. In other words, the data should and would be accessed by multiple users at multiple times. Data that did not meet these requirements (e.g., is limited in spatial extent (regional) and is acquired by a business unit for a specific project and was, therefore, not to be used by others in the organization) are not included in the scope of this study. Similarly, data that was maintained (stored) in the regional offices would also not be included. Additionally, the scope of data software included accommodates the ArcGIS platform primarily with consideration of other raster data applications used within the organization. 7 1.5 Structure of this Document Having outlined the context for this project, the remainder of this document describes project components before assessing workflow and data management efficiencies. Chapter 2 provides a summary of raster technology advancements, data management practices, and examples of other raster data management systems. In Chapter 3, high-level design details are documented to assist the reader in understanding the role of raster data and technology within the Talisman Energy organization. Detailed technical components associated with raster data storage and dissemination using the mosaic data model are outlined as low-level design components in Chapter 4. Chapter 5 provides a post-implementation technology assessment of improvements in raster management processes through the database-related raster technology. This assessment summarizes the suitability of the mosaic dataset to support organizational business drivers and workflows, and outlines the project benefits and complexities specific to both the short and long term raster management objectives outlined in Chapter 1. Lessons learned, future work and concluding comments are contained in Chapter 6. 8 Background Chapter 2 This chapter includes an overview of topics that provide context to this project. An overview of raster technology innovations highlights the need for improved raster data management strategies. Technological advancements in remote sensing technology support a wide range of industry applications. Finally, a brief overview of enterprise GIS practices provides the context in which this work was completed. 2.1 Advancements in Raster Data Technology Recent advancements in sensor technology are changing the way remotely sensed data is collected. Technological advancements are expected to provide inexpensive and abundant satellite data marketed to agriculture, finance, and natural resource industries (Quan, 2014). Use of unmanned aerial vehicles (UAV’s) and satellite-based earth observation technologies is expanding. According to Quan, low orbit micro- and nano-satellites can be built and launched into space relatively inexpensively (between US$300,000 and $4 million), and more than 700 new satellites are expected to launch between 2015 and 2020. As a result, remotely sensed data, including imagery and elevation sources, are being generated with increased quality, extent, and frequency. As remote sensing data volumes and coverage are expected increase, so is the inventory of georeferenced scanned maps and the volume of thematic raster data that can be generated from analysis of raster imagery. It will become crucial for organizations to manage and catalog spatial raster data so that they are able to accommodate overlapping maps and images with unique temporal and spatial properties. Advancements in raster technologies are far-reaching and wide-spread within Talisman Energy. Therefore, topics relating to the advancement of raster web 9 technology, processing improvements, and data management practices were considered relevant topics worth researching as part of the design process. 2.1.1. Raster data services Advancements in web mapping technology have played a significant role in improving raster data dissemination and management practices in the spatial data industry. Web innovations over the last decade have resulted in exciting new applications. The evolutions of Web 2.0 mapping applications on the web, such as Google Maps, MapQuest, and Bing Maps, have delivered innovative and rich user experiences, leveraging raster data such as imagery as a backdrop (Li et al. 2009). On the web, vector map data can be provided in a raster format as a means to protect back-end spatial vector data from being acquired and used by competitors (Mao et al. 2013). Service Orientated Architecture (SOA) is a concept that considers data architecture best practices when disseminating data and other software. A major SOA theme is the ability to accommodate data interoperability for enterprise applications within a rapidly changing environment. Spatial data is used in many industries and it is common for large organizations to internally develop and manage spatial data services using SOA to streamline access and management of GIS data sources including raster data. Concepts of SOA relating to GIS categorize functions as data services, processing services, and catalog services (Saleh et al. 2012). The ability to publish raster web services that are compatible with other software packages including web application programing interfaces (API) provides a mechanism to configure GIS SOA. 10 2.1.2. Raster compression Server and web mapping technology is becoming increasingly prevalent in the GIS industry. Commercial navigation systems provide high performance maps leveraging a variety of data types. Raster data, including imagery and elevation data, can be deployed in a web map to provide users with location context. However, vendor sourced raster data can often have an overabundance of detail without providing meaningful information to a user. In an effort to reduce storage and network transmission requirements, raster compression methods are being constantly evaluated with technological innovations in mind (Fourest, 2012). Generally, raster compression methods are categorized as either lossy or lossless compression (Mao et al. 2013). Lossy raster compression is able to reduce file size by permanently eliminating redundant data within an image so that the amount of information is retained while the file size is reduced. Lossless compression should be considered when analysis needs to be performed on data to generate derived products because none of the original data values have been modified. Raster data compression can be an important factor in reducing disk storage and improving network performance. The ability to compress raster data provides opportunities to optimize source data so that transfer over disk and network is improved. 2.1.3. Esri’s mosaic data model The Esri mosaic data model allows organizations to model raster data specific to meet user requirements with its ability to provide raster data across multiple platforms including desktop, web and mobile applications. Data model functionality includes the ability to generate dynamic information while handling large raster data sources. Mosaic dataset rendering and 11 process functions are operations applied to source data so that data is processed and disseminated to users on demand making it easy to generate multiple products from a single raster source. The ability to maintain metadata, reductions in data redundancies, and its ability to handle large data sources with unique types are some of the key abilities of mosaic data model. Data management and dissemination enhancements reduce imagery processing times and enable users to handle multi-temporal overlapping data sources to meet user requirements (Childs, 2010). Figure 3 provides a visual overview showing the relationship between file-based raster data, mosaic datasets and the ArcMap software interface. . Figure 3 The Mosaic Data Model. The mosaic data model can dynamically represent overlapping or edge joined raster data while preserving the integrity of source raster data Source: Childs, 2010. A key advantage of the mosaic data model compared with raster data stored as separate files on disk is its added ability to link footprint and attribute information to raster data as an attribute table with geometry outlining the extent of source data. ArcCatalog provides the ability to deploy mosaic based web services with ease, and Desktop ArcGIS users are able to access these raster data services using the ArcGIS platform as well as 3 rd party platforms. 3 12 2.1.3.1. Overview files The mosaic data model provides capabilities to improve raster data rendering through the use of overview files and pyramids. Overview files are associated with mosaic datasets and can be constructed and configured in order to produce lower resolution copies of data that improve the performance of data sources when viewed over large geographic extents or transmitted over enterprise networks. 2.1.3.2. Mosaic dataset table components Mosaic datasets have a core dependency to parent geodatabase tables that are hidden to desktop users. These linked tables allow data footprints, processing logs, item caches, and other properties to be managed within a geodatabase. Table 7 lists these associated tables. These features allow functionality such as the ability to render images at reduced resolution instantly for rapid visualization. Processing tools built into the ArcGIS Desktop framework link to mandatory and optional tables providing additional levels of metadata that allow users and data administrators to access archived processing logs and mosaic properties. Table 1 Mosaic dataset tables. Source: Esri Table Name Description AMD_<NAME>_ANA This table stores the analysis results generated by the Analyze Mosaic Dataset geoprocessing tool. AMD_<NAME>_ART This table stores a history of all the raster types used in this mosaic dataset. AMD_<NAME>_BND The boundary for a mosaic dataset is a feature class. This table defines the boundary (extent) of the mosaic dataset. AMD_<NAME>_CAT This is the raster catalog behind the mosaic dataset. It stores raster datasets that participate in mosaicking. AMD_<NAME>_CCA This table stores information about color correction. AMD_<NAME>_CHE This table stores a managed item cache generated by the Build Item Cache geoprocessing tool or Cached Raster function in ArcGIS for Desktop. 13 Table Name Description AMD_<NAME>_CSL This table stores cell size levels of a mosaic dataset. The table is created when you specify or calculate the cell size for the mosaic dataset. Information in the table is updated when calculating cell size levels or defining overviews and is used while generating overviews. AMD_<NAME>_LOG This table stores errors, warnings, and messages that are generated by various operations performed on a mosaic dataset. AMD_<NAME>_OVR By default, the overview rasters that are created for mosaic datasets are stored in this table. If you change the storage location for the overviews, which can be done using the Define Overviews tool, this table will not be populated. Instead, a pointer to the overview location will be stored in the urihash field of the AMD_<NAME>_CAT table. AMD_<NAME>_SML This table defines the seamline shapes that are used for mosaicking when the MosaicMethod used is seamline. AMD_<NAME>_STR This table stores stereo IDs of the pairs that participate in a stereo image. AMD_<NAME>_STS This table maintains a one-to-many relationship between each analyzed mosaic dataset item and the set of corresponding analysis results. 2.1.3.3. Raster functions The mosaic data model provides the ability for users to dynamically query and process raster data through the use of raster functions, enhancing data modeling, mapping, and GIS analysis workflows. Raster functions provide the ability to embed raster processing functionality such as “hillshade” or “slope” into mosaic datasets. Raster functions and geoprocessing tools differ in that raster functions store processing steps while geoprocessing tools create and save outputs. ArcGIS provides multiple out of the box raster functions that can be accessed and applied to multiple raster types and subtypes. Such built-in functions are powerful when working with large data sets because calculation processes are only applied to individual pixels as they are rendered for display. Combining raster functions with a single source mosaic dataset allows users 14 and data administrators to significantly reduce processing times and avoid the creation of intermediate data sources. In instances where out of the box functionality does not meet user requirements, it is possible to insert custom functions and to build raster function chains. Raster function chains can be configured using predefined inputs as variables and can accommodate sequential raster functions into a single template. For example, combining azimuth and altitude processes with hillshade visualizations into a function chain can provide visualization improvements in both flat and sloped areas. Figure 4 illustrates raster function chains. Figure 4 Mosaic Dataset Raster Functions. Source: Sweet and Lucotch 2011 Once configured, function chain templates can be added to mosaic datasets so that complex raster processes can be generated on-the-fly as users render pixels. Function chains also 15 store descriptive properties so that metadata information can be tagged within the template for further optimization of requested processing. ArcGIS raster functions and function chains can be set up on both mosaic datasets and on image services. Because raster functions process a limited amount of pixels in memory on-the- fly, they provide the impressive ability to combine multiple functions into chains that can be dynamically applied on large and robust data sources. However, despite their versatility, raster functions are limited when a workflow needs to account for values beyond the processing window. As well, raster functions cannot be used in Model Builder or ArcPy scripts. 2.1.3.4. Image analysis interface Raster data workflow enhancements gained using on-the-fly process functions can make use of an interface specific to raster data in ArcMap, the Image Analysis window. While many tools to process raster data in ArcGIS can be accessed as normal ArcGIS geoprocessing tools, the Image Analysis window provides a user friendly method to dynamically alter and process raster data (Figure 5). The ability to dynamically process and visualize mosaic raster data sources using this window provides benefits when preparing maps and data for analysis and also in the development of web-based images services consumed by other applications. 16 Figure 5 Image Analysis Window. The image analysis window in ArcMap provides the ability to adjust raster display and processing properties 2.2 Enterprise Spatial Data Management The objective of enterprise data management (EDM) in general is to bring order and structure to data while aligning with the organizational technologies and structures (Simon 2014). The need to remedy data fragmentation within an organization has become increasingly relevant as technological advancements contribute to the opportunities provided by additional data management resource allocations. Fundamentally, the enterprise data management planning process must be strategic and controlled, leading to a structure that is centralized and standardized (Peters 2008). In an era of technological advancements, many organizations are recognizing the importance of constructing a thoughtfully developed EDM road map that is adapted towards data and software within an organization. An enterprise spatial data management system is one component of an enterprise GIS. The components comprising an enterprise GIS are integrated across an organization, allowing many diverse users to access, visualize, analyze, and interpret various kinds of spatial products, including raster data (Croswell 2009). This framework ties closely with this project’s objectives. 17 In fact, enterprise spatial data operations at Talisman Energy have been enabled previously to manage corporate vector datasets, but these systems have paid little consideration to the management and use of multi-temporal and multi-themed raster data collections. Such data management requires an entirely different data structure. 2.3 Foundations for Enterprise Raster Data Management Systems While no publications regarding the generic implementation of enterprise raster management systems were found, examples of cases that leverage project management, enterprise data management and emerging raster technologies can contribute to understating real world raster management solutions. The concept of database enabled raster management systems can be better appreciated with an understanding of its origins, from tiled databases to web- enabled data management systems to enable data dissemination. 2.3.1. Emergence of tiled database raster management systems Technology for managing raster image databases is not new and advancements have been driven by growing demand for easy access to large archives of raster data from a diverse range of remote sensing sources. Emerging in the late 1990s, the software RasDaMan (Raster Data Manager) helped to pioneer relational DBMS to optimize the storage and retrieval of raster data (Baumann et al. 1999). This system was innovative in that it provided database support to manage large volumes of spatio-temporal raster data. This technology is relevant to both commercial and academic interests and is still being used today to drive raster data innovation and optimization (Gutierrez and Baumann 2008). Storage management using this technology breaks images into smaller “tiles” that can then be spatially indexed and managed within a relational database. The RasDaMan API initially used RasQL and Raster Library (RasLib) to manage and query arrays of multidimensional 18 discrete data. According to Baumann, tiling strategies are key in order for query functions to be optimized within relational DBMSs. Improvements in the performance of computation of queries in raster-related databases are achieved through advanced data aggregation. The result is a reduced CPU cost required for the computation of queries. RasDaMan paved the way for the next generation of management and analysis of large multidimensional raster data sources. However, although it is a system intended for use on intranets, RasDaMan technology does not provide a graphical user interface to allow easy access to data on the web. 2.3.2. Example of a web-enabled database raster management systems The TERNO initiative supports four terrestrial earth observatories dispersed across Germany. Spatial data collected by these observatories, including large and frequently updated raster data sources, required dissemination to both distributed scientists and the general public. Consequently, a web-enabled spatial data portal, TEODOOR (TErno Online Data repOsitORry), was developed to manage data access by users (Kunkel et al. 2013). Implemented with a standardized structure, TEODOOR is able to manage, distribute and publish increasing amounts of terrestrial observation data. It provides a relevant example of large-scale raster technologies used to process and manage large amounts of data using web capabilities. Independent TEODOOR data structures connect to data through a central “umbrella” that provides users the ability to visualize and query data through a web interface (Figure 6). 19 Figure 6 TEODOOR System user data access methods. Source: Kunkel et al. 2013 TEODOOR uses the Open Geospatial Consortium (OGC) Sensor Observation Service (SOS) standard that was developed for web service use cases where sensor data requires interoperable functionality. OGC SOS has become a standard management interface for many types of fixed and moving sensors used in radar, weather and other remote sensing technologies. A long-term goal of this system is to record and monitor environmental change at regional levels so that scientists and researchers can share findings and develop strategies for adapting to environmental changes. This system is important in its ability to support research about climate and land use change using long-term data sets that cover a broad range of topics. Under this system, individual observatories and institutions are responsible to manage and 20 implement a local data structure using tools that allow the automation of a consistent, shared data management structure. The scope of this technology is significantly larger in scale compared to what is required at Talisman Energy but it cultivates the important concept of the importance of data management systems planning. Raster data management systems exist and can be used to manage and disseminate big data while providing for distributed ownership, access restrictions and standardized, shared interfaces. 2.4 Summary Investigations into emerging raster technology and associated enterprise GIS practices have provided improved understanding of real-world raster data dissemination practices. The examples outlined demonstrated both basic and advanced cases of database raster management systems. It has been shown that Esri’s mosaic data model provides opportunities to deploy and disseminate raster data to an existing ArcGIS software user base. Having laid the foundation to understand what is needed and possible in a raster data management system, the next two chapters discuss the design components of the system implemented for Talisman Energy 21 High Level Design Components Chapter 3 The development of an enterprise raster data management strategy required consideration of multiple components with varying levels of details. This high level design summary provides information about raster data and software requirements at Talisman Energy that were considered in the design of the data management architecture. A summary of existing hardware and software components and raster data acquisition and dissemination strategies provides a basic framework for understanding business drivers associated with this project implementation. 3.1 Software Requirements The ArcGIS platform has long been used as a spatial software tool at Talisman Energy. ArcGIS for Desktop is used by employees with multiple disciplinary backgrounds and GIS experience that ranges from beginner to advanced levels. The existing license agreements, software and large user base played a significant role in the adoption of the final database raster management technology (Table 2). Prior to the development of the mosaic data model solution, ArcGIS for Server had already been configured and provided the ability to centrally host spatial data web services. This software was flexible to deploy within the organization and was supported by internal IT infrastructure resources. Esri’s mosaic dataset and ArcGIS Image Extension for Server were therefore selected to provide the technology for enterprise raster management solutions within the organization. Several aspects of the ArcGIS platform are relevant for the system design. These are discussed below. 22 Table 2 Software used to access process and disseminate spatial raster data at Talisman Energy Major Spatial Raster Software Components Name Description ArcGIS Desktop Map and process spatial data to visualize relationships, patterns and trends ArcGIS for Server Web GIS implementation will allow desktop, web and mobile based devices to access raster spatial information ArcGIS Image Extension for Server Manage and share large collections of raster data including imagery and elevation data. Imagery can be accessed by desktop and web sources with capabilities that include dynamic mosaicking and analysis processes 3.1.1. Geodatabase model Storing spatial data in a database model benefits organizations because of its ability to improve data administration, management of data access, and integration with other organizational databases (Zeiler 2010). This flexible model has been structured using unique data configurations that provide developers with the ability to design geographic databases to match logical data models. Multiple geodatabase models are available with unique specifications and storage capacity differences. When looking to implement an Esri mosaic data model it is important to consider what type of geodatabase will be used since all (file, enterprise and personal geodatabases) support raster data. The file geodatabase model is a hybrid between the personal and enterprise geodatabase models and has been selected to manage Talisman Energy enterprise raster data. GIS IT directives have defined file geodatabase to store mosaic data model and possess the following storage raster characteristics:  Size limit of 1 TB exceeds general raster data requirements at Talisman Energy.  Geodatabase is able to support both single users and small workgroups. 23  Raster data storage can be managed on a file system and does not need to be directly loaded into the geodatabase reducing data duplication.  Incremental raster data updates are supported.  Support of subtypes, domains and replicating processes align with existing enterprise data management practices at Talisman Energy.  Configuration keywords are supported allowing data storage optimizations specific to particular data types which in turn optimize storage performance and efficiency. Configuration keywords can be used to increase the maximum file size of the file geodatabase up to 256 TB. 3.2 Hardware Architecture Requirements and Overview Desktop hardware at Talisman Energy with install instances of ArcGIS Desktop meet or exceed minimum hardware requirements outlined by Esri. The HP Z820 workstation has been used to migrate and process raster data from source to the geodatabase mosaic data model. Desktop GIS hardware used in this project far exceeds minimum hardware requirements. These are summarized in Table 3. Table 3 Talisman Energy’s ArcGIS 10.2 Desktop Hardware Specifications Minimum requirements Existing hardware CPU Speed 2.2 GHz, HTT or Multi-core recommended Intel 2.7 GHz Memory/Ram 2GB 64Gb Display/ Screen Resolution 24-bit, 1024x 768 (96 DPI) 32-bit, 1024x 768 Video/Graphics 64 MB RAM (256 recommended 256 Mb RAM Disk space 2.4 GB, 500 MB swap space minimum 500Gb 24 3.3 Raster Data Types and Subtypes Business requirements specific to existing data should be considered in the development of a raster data architecture and data standards. Raster data types, subtypes, sources and formats used within an organization are important components to investigate in developing a raster data management strategy. The availability of raster data types specific to geographic regions plays a major role in developing an integrated data management structure to store raw and processed data. The range of corporate raster data sources used contributes to the development of a raster data structure template and provides guidance on the scope and extent of data to be integrated. At Talisman Energy, there are three key types of raster data used: imagery, elevation data and thematic data. The following sections describe each of these types. 3.3.1. Imagery Imagery data typically consists of continuous aerial or satellite images. Many applications exist that depend on the use of raster datasets that leverage imagery as a core application component (Giovalli and Lemoine 2013). Legacy Talisman had collections of image data that include both satellite and air-borne remote sensing technologies. High-resolution satellites and airborne sensors have the ability to capture a broad range of imagery types that vary in quality and functionality, representing various bands on the electromagnetic spectrum. A wide variety of vendors, data sources and imagery capture technologies were considered in the design of the enterprise raster management model and system. Satellite, aerial and UAV imagery products can contain proprietary information or data that is often limited by licensing agreements. Access to this information had to be considered in the design of the system. 25 3.3.2. Elevation data The shape of the land surface is frequently represented as digital elevation models (DEMs) where cell values represent the elevation values at cell locations over the continuous terrain surface. Elevation raster products can be derived using various techniques including photogrammetry, LiDAR, and land surveying. Often, DEMs are divided into subtypes. Digital surface models (DSM) represent the earth’s surface and all objects on it, including such objects as trees and buildings (Wężyk et al. 2015). At Talisman Energy, DSM rasters are sometimes referred to as “Full Feature” DEMs and are often collected from the first return of LiDAR. Digital terrain models (DTM) are DEMs that represent bare ground without including other objects such as vegetation and infrastructure. At Talisman Energy, DTMs are sometimes referred to as “Bare Earth” DEM models. GIS functions that can be applied to raster elevation data are very different from those used on imagery data because cell values are true elevation values rather than reflectance (or “color”) values as in imagery. Raster elevation data is an important data source for multiple business functions in both exploration and production at Talisman Energy. 3.3.3. Thematic data Cell values in thematic raster data are integer values that correspond to a limited set of unique features or characteristics. These are often generated using image classification tools that provide the ability to classify continuous image data into groups based on similar reflectance values (Godfrey and Eveleth 2015 ). Classified thematic data are also available through government entities such as the Province of Alberta, Canada and may include data about soils, land use and landscape risk factors such as landslide potential. 26 Digital conversion of hardcopy maps by scanning is also common practice. The resulting images can be stored in raster format in which the pixel values represent the colors on the original maps. Many potentially useful historical maps can be digitized in this way. Scanned maps in raster format provide a way to archive and catalog hardcopy maps into a consolidated location where it can be accessed by other users. Thematic maps are used for a broad range of oil and gas industry applications. Thus, the integration of thematic raster data within the raster data management system is needed to accommodate current and future requirements. 3.4 Data Acquisition and Dissemination Frequently, spatial data needed for specific oil and gas exploration and development purposes must be purchased from commercial vendors. Spatial data requirements range from simple imagery for visual display to complex survey grade elevation data used for engineering operations. The spatial data industry has countless commercial data providers. The main consideration for businesses looking to acquire spatial data is to ensure that data products provide value as a decision-making tool. Prior to commencing any data acquisition-storage- dissemination work, business requirements must be gathered and documented by GIS IT and business stakeholders. Enterprises should consider vendor management when looking to acquire raster spatial data. End-user license agreements (EULA) between licensors and organizations define contractual obligations and are often required when purchasing from commercial data sources. Data sharing and publication should adhere to the EULA. Therefore, it is important to update metadata relating to EULA items. 27 If the business has already acquired the data, then a review of that data should be completed by the GIS team to determine if it is suitable for adding to the corporate raster repository. GIS support will follow established best practices in procuring the most appropriate data to meet business demands. Table 4 outlines common considerations that can lead to the appropriate selection of vendors, products and expected deliverables. Table 4 Common business considerations relating to spatial raster data at Talisman Energy Raster Data Considerations Cost What are your budget limits? Can you afford the data you want? Is there an alternative within your budget? Availability Does the data already exist? How often is the data updated? Will you receive updates as individual tiles or a single update with complete coverage? Can you receive this data in a timely manner? Licenses Can you share or distribute this data? Can you use this data in multiple projects? What can you do with the information or data derived from the original data? Can you serve this to the public using the Internet? Resolution Will the available level of detail provide the required information? Temporal Resolution Does data provide up-to-date detail to capture features of interest? Is more recent imagery available? Can cost be saved by purchasing archived images? Extent How many raster tiles are required to cover the extent of the project? Storage What database or file formats will be used? How large is each file? How much total disk space is needed? Accuracy What is the level of accuracy stated by the data vendor? Will the stated data resolution provide you with the required spatial accuracy? How will the data be verified and validated? Accessibility and pricing Is the data accessible or will it be accessible on a network? Will the data vendor charge fees for data access or download? Who will have access to the data? How will you control access and sales? Coordinate Reference System (CRS) Ensure that data is provided in a CRS that matches with project specifications. Data can be projected into other CRS’s but the performance benefits should justify data duplication. 28 Once satisfactory source data has been acquired, it must be stored in prescribed locations. If the data does not require access from multiple users at one time, then a suitable storage location needs to be determined with the business users. If raster data access is required by multiple users at one time, then storage and dissemination strategies should be followed. 3.4.1. Data dissemination protocol In the case of the Talisman Energy system, the dissemination of corporate raster data to the users includes the following components:  ArcGIS Layer files (on SharePoint site and in shared General folder)  Image services (published on SharePoint site)  Communication sent to users (newsletter) ArcGIS layer files (.lyr) are particularly useful in the mosaic data model since they can reference mosaic datasets through a network file location or internally based web portals. Layer files using raster data sources can be constructed using a combination of mosaic datasets and image services. 3.4.2. Web server clients Internal GIS server infrastructure supports the capability to incorporate the mosaic data model and provide internally hosted image services to desktop and web interfaces. A combination of internal and external data sources is hosted within the Talisman Energy organization with web GIS clients in Houston and Pittsburgh office locations. Figure 7 provides a schematic to summarize existing web service hosts and internal data clients. 29 Figure 7 Web-based spatial raster data servers and clients within Talisman Energy. 3.4.1. Web server configuration Mosaic datasets that need to be accessed by web and 3 rd party applications, such as Google Earth and Javascript Web API’s, qualify to be hosted as web-enabled Image Services. An Image Service is a web-enabled raster dataset that can be consumed in multiple applications. Within ArcGIS for Server, an image service is a type of web service that is generated from raster data. Services are visualized, queried and analyzed in a web-based environment. Service based raster data management workflows are intended for individuals familiar with data management standards and ArcGIS for Server technology. The standard Talisman Energy protocol of data dissemination uses a three-tiered environment. Data is initially loaded and configured in the development (DEV) environment. Once deemed satisfactory by the raster lead, it is handed off to the data administrator for 30 promotion to the test (TEST) environment. The raster administrator acts as a gatekeeper to ensure the service is configured properly. Once running in TEST, the business client is asked to conduct user acceptance testing. Upon acceptance, the service is promoted to the Production environment (PROD) by the data administrator. Finally, the raster data lead disseminates the data to the business team. Each staging environment has unique operation objectives that are outlined in Table 5. Table 5 Talisman Energy multi-tier server environment for raster data services Server Environment Description PROD Environment The PROD environment refers to the production environment for image services. Services in this environment are published by IT resources. The general GIS user has read-only access to the PROD environment. TEST Environment The TEST environment is a carbon copy of the PROD environment. This environment is where an image/web service can be published by PTS resources. In the event that the PROD environment becomes unstable TEST can be leveraged. The TEST environment is also to be used for User Acceptance Testing, DEV Environment This server environment accessed by power users within internal teams to develop and publish spatial services for web consumption. The DEV environment is to be used for publishing, and non-publishing processes should be avoided. GIS users in the business teams do not have access to the DEV Environment. Once development is complete in DEV, the dataset (mosaic and image service) can be promoted to the TEST environment. 3.5 Summary In order to adequately develop a strategy for raster storage and dissemination, it was crucial to consider Talsiman’s raster requirements with respect to data scope, resource allocation considerations, and existing practices. The next chapter further explores design requirements at the low-level of data management. 31 Low-Level Design Components Chapter 4 Raster data acquisition and storage workflows determine how raster layers are developed and web services are provided to clients within the Talisman Energy organization. While high-level design components address data, software and hardware considerations, low-level design components focus on the raster data management. This chapter explores these low-level design components as they impact the design of the raster data management system. 4.1 Source Data Management and Processing In a traditional environment, users are able to directly access source raster data files for processing and visualization. Enterprise data management practices bring order and structure to file based raster data components. The proper storage of source data in a dedicated storage location ensures metadata, such as capture date and coordinate reference system, can be cross- referenced in subsequent processes. When source data is properly managed, it can reduce data duplication and streamline workflow processes. 4.1.1. Source data structure and naming conventions Talisman Energy source data is managed and administrated by a raster data administrator familiar with enterprise data management practices. Following implementation of this data management system, raster data is now stored in a prescribed file-based location following a standardized naming convention and folder structure. This naming convention was developed to meet business specific access considerations. The folder structure considers geographic location, projection and coordinate reference system, and raster data types to ensure that data managers are able to organize data using a consistent and structured environment and casual users can 32 access data using easy to deploy layer files and/or image services. Table 6 outlines this folder hierarchy. Table 6 Source data folder hierarchy Folder Level Descriptions Level Category Description 1 Raster data This folder level is where all source raster data folders are contained 2 Country/ Region Folder are named using the country or geographic region. In instances where raster data extent spans multiple countries it is possible to create an additional region folder specific to continent. 3 Type This folder level is broken down into Imagery, Elevation, Thematic and Sub-Surface. The imagery and elevation folders are self-explanatory. Thematic raster products include discrete data such as survey files, topographic maps, and land use datasets. The subsurface folder is a location where raster data specific to subsurface users will be stored. 4 Subtype This folder level is categorized by the raster product subtype. 5 Projection Folder names describe the source data projection properties. Folders that contain a “Source” postfix indicate data is in a raw and unprocessed format. Folders that do not contain a “Source” postfix indicate that source raster data has been processed into a projection that is different from the source raster. Source data folder hierarchy 6 Date and Vendor In instances where a capture date is available, folders are named using an YMD format. Vendor name is post fixed after the date. 7 Vendor delivery structure In many instances there will be metadata and associated files that are delivered alongside a remote sensing product. The level 7 folder structure will reflect the original folder structure of a data delivery so that data has a format consistent with what a vendor provides. Data administrators allow for folder structure flexibility at this level in order to accommodate variations in vendor delivery methods. Adopting a file structure and naming convention has provided order and logic to the organization of raster data. Defined folder levels serve as a solid base within which to organize 33 newly added raster data with inherent consideration to metadata. Figure 8 provides a visual overview of the entity relationships and folder structures used to manage source raster data. 4.2 Mosaic Dataset Management and Processing Once the raster source data is stored in a file-based structure, it can be added to a new or existing mosaic dataset. A mosaic dataset provides the ability to store, manage, view, and query large and small collections of raster and image data. It is a data model within a geodatabase used to manage a collection of raster datasets (images) stored as a catalog and viewed as a mosaic dataset. Mosaic datasets have advanced raster querying capabilities and processing functions and can also be used as a source for serving web-based image services that can be consumed by other applications including the web. 4.2.1. Mosaic data structure and naming conventions Mosaic datasets, residing within file geodatabase, are within a file based environment. Storage of geodatabase and mosaic datasets employ a file based structure consistent with the source data structure (Figure 9). The mosaic dataset employs a simplified version of the defined source data file structure. The folder hierarchy and data management strategy continues to consider geographic region, raster data types and subtypes. 34 Figure 8 Source Data Structure and Hierarchy. Folder levels consider vendor metadata. 35 Figure 9 Mosaic Dataset Folder Hierarchy. 4.2.2. Creating a source mosaic dataset Once the mosaic dataset structure was created, source raster data could be added into the mosaic dataset. During this process it was necessary to remember the following items:  The CRS of a source mosaic should match source data. If needed, CRS transformations can be configured on subsequent referenced mosaics.  Source data is not being duplicated and copied into the file geodatabase but instead a reference to source data is created.  Mosaic dataset name limit is 22 characters. Mosaic datasets data is stored and accessed on the Talisman Energy network using “Universal Naming Convention” (UNC) to identify the location where source raster data that is shared with mosaic datasets and associated services are stored. Local network conventions were 36 avoided when adding data so that mosaic datasets were able to consistently reference sourced data without a reference to a local mapped drive location that has potential to be inconsistent between users. 4.2.3. Mosaic data design template and schema Developing a mosaic dataset template that considers key raster attributes helps to ensure that mosaic datasets are modelled with consistency. Source raster properties that have been embedded into the file based structure of source data will be leveraged and implemented into mosaic attribute database schema. To assist with consistency, data management, and query functions on enterprise mosaic datasets, a mosaic template was implemented. The template is a standard set of attribute fields that is applied when building the corporate mosaic dataset. The field names implemented within the mosaic dataset template consider metadata embedded into the source data structure. Implementing this template and attribute schema ensures that the core components are attributed in all mosaic datasets, thus providing consistency and structure. Mosaic dataset footprint boundaries are populated with consistent data attributes specific to raster metadata that is summarized in Table 7. Table 7 Mosaic dataset attribute template for enterprise raster data sources at Talisman Energy Field Name Description Required Data Type Domains COUNTRY Country of dataset, if data is not specific to country is can be considered "Regional" or "Global" Yes Text No DATATYPE equivalent to level 3 of source data structure Yes Text Yes SUBTYPE equivalent to level 4 of source data structure Yes Text Yes 37 RESOLUTION_M Resolution of dataset in meters No Double No DATE Capture date of image Yes Date No VENDOR Vendor of raster product Yes Text No SENSORNAME Sensor name (if applicable) No Text No LICENCE License agreement document link No Blob No Developing a standard mosaic template containing attribute fields provided a means to document and enforce the population of raster data metadata. This information is valuable to users and can be used identify and query enterprise raster data sources. The field names implemented within the mosaic dataset template consider metadata embedded into the source data structure. Implementing this template and attribute schema ensures that the core components are attributed in all mosaic datasets providing consistency and structure. Mosaic datasets have attribute tables associated with them that allow information about the dataset to be stored with them and made accessible to users. 4.2.4. Referenced and derived mosaics The creation of referenced mosaic datasets allows data administrators and users to create a copy of a mosaic dataset that remains connected to its source and that continues to reflect changes applied to a view of mosaic that is also connected to the source. Referenced mosaic datasets support enterprise management best practices by avoiding duplication of data. This functionality is particularly useful when producing elevation products such as slope and hillshade rasters. Referenced mosaics are created as a backup copy of the original mosaic and act as a storage location to test and develop raster function chain processes. In the Talisman Energy system, referenced mosaics are contained in file geodatabases with the same structure as source 38 mosaics except that they have an “R” prefix. For example, source mosaics for LiDAR are stored in “LiDAR.gdb” referenced mosaics are stored in “R_LiDAR.gdb”. This is best practice because it is possible to corrupt a source mosaic, for example, when editing a raster function chain item. Derived mosaic datasets allow multiple mosaic datasets to be combined into a single source. This provides the ability to mashup data from multiple mosaic datasets to meet user custom user’s requirements without duplication of source data. Derived mosaic datasets can also reduce the amount of image services and management resources dedicated to raster sources through consolidation of multiple services. 4.3 Data Updates and Maintenance Mosaic datasets provide the flexibility to maintain data sources residing in the source data structure. Raster data can be added to existing mosaic datasets using multiple methods to ingest data. It is possible to add data synchronized with metadata that automatically populates attributes such as capture date and sensor information. The ability to join additional attribution to a mosaic dataset through tables means that data managers can join updated information to raster data sources without having to update the data themselves. Configuration allows for mosaic datasets to continuously audit directories so that additional data can be easily integrated into enterprise data repositories. The ability to easily add data and associated dependencies in a mosaic dataset provides potential to streamline update processes to accommodate new data sources. This one-way operation preserves source data while providing updated information to the user. 4.4 Project Workflow Summary This project was developed and executed over an 11-month period. Working in collaboration with a group of colleagues, I completed a series of steps leading to the 39 implementation of the raster data management system. Project timelines and dependencies are outlined in Table 8. Step 2 required completion of the previously initiated major upgrade to the GIS hardware and software infrastructure by the GIS IT group. Table 8 Project Milestones Project Milestones Step Thesis Deliverable Timeline Dependencies 1 Raster Management Training Completed November 2014 None 2 Implementation of Development (DEV) Server by GIS IT Completed February 2015 None 3 Source Data Organizational Structure and Processing Completed Due July 2015 None 4 Mosaic Dataset Data Structure and Processing Completed Due July 2015 Step 3 5 Dissemination Structure Completed September 2015 Step 4 6 Implementation of Production Services Completed September 2015 Step 5 The individual steps that were completed leading to the implementation of the final raster data management strategy at Talisman Energy are summarized in Table 9. 40 Table 9 Implementation Steps Completed Step Description 1 Determination of the spatial raster data usage requirements It was necessary to achieve agreement on the data usage requirements for Talisman Energy corporate data. These requirements were documented and are reflected in Section 3.3 above. 2 Data audit For each data source to be ingested into the new system, the protocols and considerations outlined in Section 3.4 were documented. 3 Source data storage After agreeing on the data to be included in the system, the source data structure as shown in Figure 9 was designed and the migration process into the source data structure began. 4 Creation of mosaic datasets Included the following steps: • Create source mosaic dataset • Create overview files in central location • Create a master mosaic dataset (if combining 2 source mosaic datasets) • Create referenced mosaic datasets (ensures source mosaic datasets are not corrupted) • Create derived mosaic datasets • Create raster functions as required 5 Creation of image services Included the following steps: • Create imagery services • Create tiled (cached) services if appropriate • Creation of raster functions as required • Promotion of the data through our standard DEV, TEST and PROD environments • Documentation of dependencies 6 Dissemination to the user(s) Dissemination of corporate raster data to the users entails the following components (Section 3.4.1) • ArcGIS Layer files (on SharePoint site and in shared General folder) • Image services (published on SharePoint site) • Communication sent to users (newsletter) 7 Data maintenance This stage entailed the following processes (discussed in Section 4.3): • Additions of new datasets • Merging in of updates or replacements to existing datasets 4.5 Summary As a result of this implementation process, a generalized workflow for the raster data life cycle at Talisman Energy can be visualized as shown in Figure 10. 41 Figure 10 Raster Data Life-Cycle at Talisman Energy These workflows are reviewed and updated regularly as needed to accurately describe best practices and duties and to reflect technology advancements. Mosaic datasets are modeled using a method that is specific to business demands while reducing overhead and the need to duplicate data. Strategic management of source data and subsequent mosaic datasets support all stages of the raster data life cycles at Talisman Energy. 42 Technology Assessment Chapter 5 The enterprise raster data management system for Talisman Energy, designed using the raster mosaic data model functionality and design principles described above, was successfully implemented during 2015. This chapter discusses how well this implementation meets the raster data business requirements of both desktop and web data consumers and summarizes key benefits specific to the Talisman Energy organization. 5.1 Raster Workflow Assessments This section provides several examples to demonstrate how data management system requirements to improve data management and workflow processes relating to the common raster types have been achieved. Associated benefits are categorized based on specific raster data sources and dissemination methods. As can be seen below, the mosaic data model has been able to successfully accommodate raster data from multiple sources and in formats that vary in both extent and quality. 5.1.1. Elevation and LiDAR workflow enhancements As explained earlier, raster source data is typically provided by vendors with unique file delivery methods. It is common for elevation data to be delivered in a multi-file tiled format. However, different vendors use different tile systems. Using the implemented mosaic data model, multiple vendor specific raster delivery structures have been successfully ingested, providing significant workflow and performance improvements. Once elevation data is added to the mosaic data model, users have the ability to dynamically and efficiently prepare and process data regardless of how data is prepared by a vendor. Streamlined data management and workflow practices associated with mosaic raster management technology and data has enabled 43 data managers to reduce physical disk space and associated workforce resources, contributing to the overall productivity of the GIS community at Talisman Energy. Common elevation related workflows leveraging the mosaic data model provide significant benefits when compared with traditional raster management practices and technology. Legacy workflows require significant processing and data storage resources when compared with the mosaic data model. Additionally, mosaic dataset based workflows have improved upon legacy workflows by reducing processing times using mosaic functionality. As an example of these improvements, Table 10 summarizes elevation workflow improvements by comparing the disk space and processing time requirements of traditional file based workflows with those of the newly implemented mosaic raster data workflows. Table 10 Comparison of traditional and mosaic dataset elevation workflow. File Based Workflow Workflow Step Permanent Disk Space Geoprocessing Time Extract and store source data 400 MB NA Merge tiles 148 tiles into new DEM raster 587MB 56.7 seconds Generate slope from new raster 587 MB 13.7 seconds Generate hillshade from new raster 146.87 MB 42.5 seconds Generate pyramids on each raster source 284 MB (Merged: 94 MB Slope:157 MB Hillshade: 33 MB) 17.9 seconds Total 1417 MB 4 geo-processes (130.8 seconds) 44 Table 10 Comparison of traditional and mosaic dataset elevation workflow (continued). Mosaic Based Workflow Workflow Step Permanent Disk Space Geoprocessing Time Extract and store source data 400 MB NA Add rasters to mosaic 1.77 MB 27.3 seconds Generate statistics 1.81 MB 35 seconds Generate overviews 54 MB 13.5 seconds Apply slope function to DEM 0 MB No geoprocessing Apply hillshade function to DEM 0 MB No geoprocessing Total 458 MB 3 geo-processes (75.8 seconds) 5.1.2. Thematic workflow enhancements: Alberta Wet Area Mapping Prior to the implementation in the mosaic data model, problematic data rendering issues had been identified with the “AlbertaWet Area Mapping” (WAM) thematic data source provided by the Provincial Government of Alberta. This data set is an elevation derived product and was released to the general public over a prolonged-time period. Source data tiles possess unique attribute characteristics. The source raster data type is 8-bit unsigned meaning that it carries enough digits to record values from 0 to 255, an unnecessary inefficiency since the data has only 5 unique values. When displayed as raw data, tiles overlap and no data frames overlap data on other frames. Figure 11 illustrates these rendering issues. 45 Figure 11 Raw Thematic Data. Default raw source data does not render properly and lacks metadata. By default, values range from 0-255 in ArcMap software. When implemented in the mosaic data model, tile extent and attributes enhance user capabilities using through query and rending enhancements. The mosaic data model functionality provides the ability to remove areas that should not be rendered through the configuration of footprints when appropriate using the “Define Mosaic Dataset NoData” tool. This provides business users with an improved ability to view data and query metadata specific to each tile of the classified dataset (see Figure 12). 46 Figure 12 Processed Thematic Data. The mosaic dataset appends metadata to tile footprints and removes invalid values. By default, valid numbers (0-4) and descriptions are displayed in ArcMap The ability for the mosaic data model to directly import data specific footprints, boundaries and attribute information has provided a significant reduction in data management overhead while accommodating enterprise metadata standards. This examples demonstrates the ability of the mosaic data model to provide an effective method to manage and deploy archival and current thematic raster data. Table 11 compares the disk space and processing times required to undertake a typical thematic raster workflow. 47 Table 11 Thematic raster processing workflow comparison File-based workflow Workflow Step Permanent Disk Space Geoprocessing Extract and store source data 9.15 GB NA Build Pyramids 2.38 GB 43 min 9 seconds Build Statistics 2.99 MB 4 min 44 seconds Mosaic to new raster for each zone (21 zones) 23.1 GB 9 hours 50 minutes Total 35.9 GB 23 geo-processes (10 hours 38 minutes) Source Rasters: 760. Additional Folders Created: 21 Mosaic Dataset Workflow Workflow Step Permanent Disk Space Geoprocessing Time Extract and store source data 9.15 GB NA Add rasters to mosaic 0 1 hour 5 minutes Import Mosaic Geometery (Footprint and Boundary) 1MB 44 seconds Insert attribute table function into mosaic 0 NA Generate overviews 5.28 GB 1hr 33 minutes Insert attribute table function into mosaic 0 NA Total 15.3 GB 3 geo-processes (1 hour 58 minutes) Source Rasters: 760. Additional Folders Created: 1 5.2 Data Management and Interface Appraisal Mosaic dataset interface within ArcGIS has provided flexible implementation to accommodate internal enterprise raster management strategy guidelines leveraging the mosaic dataset raster model. It is critical to properly configure mosaic dataset properties. ArcCatalog provides the ability to modify mosaic dataset properties The proper configuration of mosaic dataset properties allow data administrators to properly design raster data and through the configuration of data types, resampling, and display properties. Source data is managed through windows based file structure. The primary method to manage data using the mosaic data model is through the use of ArcCatalog and ArcMap software 48 packages. ArcCatalog serves as a primary method to configure process and disseminate mosaic data raster sources. ArcCatalog can be accessed as a window in ArcMap or as a standalone application and provides the ability to access and manage geodatabases that reference or contain spatial raster data. Internal documentation guidelines specific to the development and maintenance of raster use on ArcCatalog for file management capabilities. Software interface compliments data management guidelines for raster source and mosaic data components (Figure 13). Figure 13 Mosaic Dataset Interface. Defined data structure for mosaic dataset items compliments software GUI's and enterprise management practices. 5.2.1. Web service management and dissemination The primary data type used by web mapping applications is vector data and because of resource limitations raster data is located on the same server as enterprise and project based 49 vector web services. The mosaic data model provides the ability to easily deploy web-based raster services for use in both desktop and web-based environments using “ArcGIS image Extension for Server”. Raster sourced web services can be deployed as image services in catalog view by highlighting a mosaic dataset and selecting server parameters to configure the type of web service that is published. Web-enabled raster service publication and data access interfaces provide a solid interface to manage and access data. Web hosted elevation image services have been developed for all North American corporate raster data sources and can be accessed by connecting to internally hosted GIS servers Below is an showing North American Elevation Image Services completed during this project (Figure 14). Figure 14 Image Service Deployment and Access. Summary of consolidated elevation Image Services. Deployment (left) and data access of services (right) using ArcCatalog 5.2.2. Data administration Mosaic datasets support a broad range of raster data types including imagery, DEM and thematic raster types. Successful data processing of large elevation, imagery and thematic mosaic datasets demonstrate the successful ability to acquire catalog and mange raster data at regional and global extents with the ability to disseminate data to multiple corporate applications. Raster 50 source data hierarchy design has been able to successfully accommodate defined raster types and subtypes with regional and global extents with examples provided in Table 12. Table 12 Examples of stored source raster data using a defined structure Source Data Structure Folder Levels and Example DataSet Level 1 Level 2 Level 3 Level 4 Level 5 Level 6 Level 7 GEBCO raster_data Global Elevation Hybrid GCSWGS84_Source 2015_GEBCO RN- 3109_143024… WAM raster_data Canada Thematic WAM NAD83UTMZ11_Source 2012_ESRD WetAreaMapping It is important to consider that mosaic disk space requirements can increase when data manipulation and database fragmentation is applied to a geodatabase. Personal and file geodatabase models contain binary files that become fragmented when data is added, removed or edited. Data management workflows can be applied to potentially reduce size and improve performance by defragmenting binary files within a geodatabase. Improvements on legacy spatial raster data administration practices have resulted in an environment with better organization, management, and control. Enterprise data administration is especially important because of its relationship with system performance and scalability. A history of what has been processed applied to the mosaic dataset is recorded in a table that is contained within file geodatabase. The ability to generate a referenced mosaic data sources provides data management and overhead benefits that accommodate enterprise business requirements because of a referenced mosaics ability to leverage previously construed and processed data sources with the added ability to adjust and test function chains without risking the corruption of either source data or source mosaic data sets. Referenced mosaic source are directly connected to source mosaic data sources allowing data administrators to further reduce overhead when additional data is processed and added to source data sources. 51 5.3 Hardware and Data Performance Assessment The ability to save copies of processed raster products using the mosaic datasets provides added value when requirements are present that require data to be transferred externally. This is also beneficial in the even that a particular software package is not compatible with data dissemination methods outline in section 4.3. Quick access to processed and source mosaic data sources and a key function for enterprise users and multiple methods are available for users to access and export raster data sources needed outside the mosaic data model. Download functionality provides robust capabilities to provide copies of source data to disconnected applications. Proliferation of source data is not encouraged but multiple export functionalities provides great opportunities for applications to have a copy of data sources provided for local consumption and helps address applications that are not compatible with mosaic datasets 5.3.1. Display Performance The mosaic data model accommodates enterprise data management practices while providing users with consistent display in both web and desktop interfaces. To get a better understanding of raster performance an assessment was to raster rendering capabilities relate to data dissemination methods using a map performance. And diagnosis tool name “mxdperfstat” has been used to diagnose raster display performance within the Talisman Energy GIS environment. Map document performance statistics calculations provided display performance statistics on source data, mosaic datasets and internal/external image services using ArcGIS Desktop software. Raster display performance comparing raw source data with mosaic datasets residing in a geodatabase are comparable. Processing factors, such as pyramids, statistics and overviews have minor impacts on display performance. Display performance of mosaic data sources is 52 determined to be suitable with accelerated mosaic data sources enabled displaying fastest by fractions of a second. Raster data display performance in web mapping applications provides adequate performance for Calgary and Houston based web mapping applications. 5.4 Achievement of Short Term Data Management Objectives This section reviews how successfully this mosaic dataset functionality and implementation supports the specific short term objectives identified in Chapter 1. 5.4.1. Successful implementation of a database related raster management technology The innovative data structure of the ArcGIS mosaic data model provides the ability to publish raster data and services quickly and efficiently while incorporating best practices of enterprise data management on raster data. Mosaic datasets footprint and attribute information allow users and administrators to populate and define metadata, attributes, and geographic scales where raster source will be activated. Services utilizing the mosaic data model have been adopted by business are particularly well received in web mapping applications. 5.4.2. Established raster data management structure and guidelines Determining the level of success after a GIS enterprise implementation can be expensive, complex and time consuming because benefits are widely spread across a broad range of operational functions in an environment company initiatives are controlled beyond the control of GIS management (Croswell, 2009). Within an enterprise environment the mosaic data model provides many advantages when compared with legacy enterprise raster management strategies and software. The mosaic data model and ArcGIS software has enhanced the ability of Talisman Energy GIS raster managers to configure data sources as unique files or folder directories (workspaces). 53 Consistent data management structure and guidelines have been be implemented, and the standards are flexible enough to be adapted to variety of business requirements. Implementing raster data management strategy and architecture has helped to reduce poor data management practices by building a foundation to support the ArcGIS platform and third party mapping software’s that are able to leverage emerging technologies such as web services. A reduction of data duplication across multiple network locations and has resulted in efficient data management overhead. Source data is stored being managed following defined structures and management procedures so that data is stored with consistency while capturing metadata components. The developed enterprise data management structure uses Talisman Energy specific raster organization components and provides flexibility to support file delivery structures adopted by data vendors. Enterprise data architecture can provide many variables in software, hardware and resources and presents major challenges to justify data management structure. A limitation of this research process is that data types and file structures have been developed specific to Talisman Energy, and, therefore, data architecture is organization specific and will not meet the needs for all enterprise organizations. 5.4.3. Accommodate server based raster dissemination requirements Enterprise based dissemination methods of spatial data provides a method for users to quickly access information from a centrally managed location for web service compatible software and Application Programing Interfaces (API). Registry/catalog services manage and maintain web service information. A combination of these service types is designed to help address inherent data management and interoperability drawbacks associated with GIS technology. The ability to save copies of processed raster products using the mosaic datasets 54 provides added value when requirements are present that require data to be transferred externally. This is also beneficial in the even that a particular software package is not compatible with data dissemination methods. The development of theses enterprise raster management guidelines paralleled ongoing corporate hardware and software upgrades. Implementing the mosaic data set model along with the ArcGIS Image Extension has provided a flexible method to serve imagery to enterprise clients with added functionality not available using legacy technology. Database driven raster data model, mosaic datasets, have provided added functionality to catalog, query and dynamically process spatial raster data following EDM practices with the ability to offer SOA to users through image and web services. Mosaic datasets and ArcGIS for Server software support both short and long-term enterprise data management to provide access to single source data across multiple applications including ArcMap, ArcGlobe, Google Earth and Petrel software. In a cost efficient enterprise environment, the mosaic data models ability to manage and host raster image services. The mosaic data model provides the ability to support a combination of both internal and external image sources within a single image service. This supported functionality provides data administrators the ability to create hybrid mash-ups that include both internal and external data raster combined a single mosaic dataset. The model provides a solid means to provide propriety raster data sources that would otherwise not be accessible through public and private raster data providers. Raster data dissemination sources including mosaic datasets, layer files, and web service are managed and deployed using ArcCatalog interface. 5.4.4. Consolidate all of Talisman Energy’s North American raster data in a single location Management structure leveraging Esri’s mosaic dataset geodatabase model has provided raster data with a single location to act as a center point for corporate raster data at Talisman 55 Energy while considering enterprise data practices. The value of internal image service hosting versus external service providers could provide cost savings and business value to organizations with large repositories of raster data and GIS users. Elevation subtypes for Radar, LiDAR, and hybrid data sources within Canada has been consolidated and processed as mosaic datasets providing the ability to visualize, manage and query raster types and subtypes through both map and catalog interfaces. Image services are accessed using a server that hosts both vector and raster services (Figure 15). Figure 15 Extent of Consolidated DEM Data. Corperate DEM data cataloged by data subtype (left) and data catalog view of mosaic data sources (right). A catalog of North American imagery data can now be leveraged by Repsol to assist with GIS integration process and provide increased awareness of available raster data sources within the Talisman Energy organization. Elevation, imagery, and thematic image services have been configured and deployed to both desktop and web business users. 56 5.4.5. Confirm that workflow improvements related to these technology upgrades satisfy objectives A common workflow is to download, process and provide project-based teams with elevation data. GIS users often need LiDAR data prepared for both mapping and visual analysis and elevation sourced workflows are common. Industry raster data vendors typically provide elevation data divided into geographic tiles and this data needs to be processed in before it can be consumed by users. Mosaic dataset technology has the potential to provide significant reductions of intermediate data sources. Implementation of the mosaic dataset model has provided considerable benefits so that elevation data can be prepared, processed and deployed with efficiency and consistency. Elevation data with higher precision can result in increased disk space utilization and acquisition costs making it less feasible to capture information over a larger extent, but the mosaic dataset provides functionality to efficiently deploy products supporting high precision and large extents. Thematic rasters such as the Alberta Wet Area Mapping data are used to support decision making so that business units can avoid areas where development could have environmental impacts. Mosaic dataset raster representations of WAM source data have been implemented without the need to process and manage additional derived data sources. Both thematic and elevation based mosaic data model workflows have been benchmarked and compared with legacy workflows. Associated workflow improvements provide a significant reduction in the amount of intermediate data sources generated from mapping and analysis functions. Data visualization and rendering improvements using ArcMap software raster interfaces have provided immediate workflow improvements. 57 5.5 Achievement of Long Term Data Management Objectives This section reviews how successfully this mosaic dataset functionality and implementation supports the specific long term objectives identified in Chapter 1. 5.5.1. Prepare for Data Consolidation for Repsol Integration The inventory of raster data sources with both global and regional extents will be leveraged for GIS integration functions between Talisman and Repsol. A large volume of existing raster data no longer pertains to business objectives at Talisman and Repsol due to asset sales and transfers of ownership. Consolidation of global raster sources into the mosaic data model is an ongoing process building on established data management practices to ensure consistency. The raster management structure serves current business needs with an easy ability to integrate with Repsol operations and technology. 5.5.2. Adapt to emerging technologies and business objectives The software upgrade and implementation of the mosaic raster data model put Talisman Energy in a solid position to support advanced raster management and dissemination methods. The model described in this document has been designed and developed to accommodate multiple raster data types supporting regional and global scales with flexibility to adapt to constantly evolving business drivers. According to Esri, general support will be offered for Image Extension technology until July 2019 at which time it will be retired. 5.6 Summary New infrastructure and software leveraging the mosaic data model improves upon legacy raster management processes because of its ability to efficiently process and deploy data to users across multiple platforms. Data sources at regional and global extents have successfully been 58 processed and deployed to clients on an as needed basis using the mosaic data model and therefore meet short term objectives and provides significant benefits on traditional raster consumption methods on the ArcGIS Platform GIS is a very effective technology with disadvantages relating to agility and software compatibility. Organizations that leverage raster imagery should consider the benefits of utilizing a technology that is capable of displaying the spatiotemporal enabled raster data so that development can be recorded at increased frequencies. Data management and format upgrades relating to raster data’s sources have been completed as a result of infrastructure upgrades to GIS infrastructure within the Talisman Energy organization. As a result, the data archive is now designed to accommodate enterprise data management practices by avoiding duplication of source data and reducing data management overhead Traditional file based workflows require significant increases in both processing resource and data management overhead when compared with workflows using the mosaic data model. Raster functions and function chains align with both industry and Talisman Energy enterprise data management practices by reducing data storage and processing times while using central data sources and reducing the proliferation of intermediate data sources required for mapping and spatial analysis. Upon processing and creation of mosaic datasets, it is possible to visualize and query data sources. As gold standard data is added to the raster folder directory, it is possible to exploit source rasters associated with a mosaic through ArcCatalog. Mosaic properties and geoprocessing tools can be configured in catalog view. The mosaic data model’s embedded ability to create vector based footprints and source dataset extents can be used to visualize, query, and dynamically process a configured mosaic dataset. 59 Data management and format upgrades relating to raster data sources have been completed as a result of infrastructure upgrades to GIS infrastructure within the Talisman Energy organization. As a result, raster data structures can be designed to accommodate enterprise data management practices by avoiding duplication of source data and reducing data management overhead. 60 Conclusion Chapter 6 As the previous chapter demonstrated, the design envisioned at the start of this project has been successfully implemented. Many benefits have been cataloged and demonstrated. However, there are always improvements and enhancements that could be undertaken. This chapter summarizes these closing thoughts. The design model in this document has been developed to accommodate multiple raster data types supporting regional and global scales with flexibility to adapt to constantly evolving business drivers. Attempting to implement a consistent data structure with standards flexible enough to be adapted to variety of considerations is an ongoing task. The raster data model migration project outlined in this paper was difficult for a single person to complete since it required considerable coordination and communication with many colleagues. 6.1 Future Tasks to be Completed and Proposed Enhancements Of course, a project such as this can never be completed. There are many additional tasks pending and potential future enhancements to consider. These include:  Increased consolidation of data.  Adopt temporal attribute functionality to allow users to access a chronology of overlapping images.  Build more mosaics that leverage legacy imagery sources with overlapping footprints so that temporal changes can be monitored with increased granularity.  Automate workflow to ensure that enterprise metadata entry and management can be streamlined with increased consistency. 61  In instances where relationships are in place with data vendors, Talisman Energy should consider approaching vendors to accommodate data delivery formats and methods ensuring easy integration with Talisman Energy raster management practices.  Enhance individual users’ raster related processing and storage workflows through the development and dissemination of workflows using the enterprise raster management system.  If raster usage in the web map and on the server increases, there is potential to justify migration of data dedicated server.  The amount of public and private image services available for consumption continues to increase and data administrators should consider the creation of mosaic datasets that leverage suitable external services in order to reduce load on local server resources.  Low resolution data sources such as SRTM, GMTED can be used to provide low- resolution views in derived mosaics and replace the need to display large extents of high resolution elevation data sources (LiDAR, Hillshades, and DEM).  Software development kits (SDKs) provide the ability to code custom applications and functions and should be investigated when looking for workflow and performance improvements.  Connectivity to ENVI software might be considered for those few specialized users who need to access raster sources directly. 62 6.2 Final Comments Esri mosaic data model technology is able to catalog a large range of raster data sources while providing the ability to quickly process and visualize raster data on desktop, web and third party applications. Talisman Energy had chosen to use Esri’s technology and data models because of an existing user base and software license agreement within the organization. Mosaic datasets with image services provide flexibility to quickly deploy raster datasets to meet business requirements. Talisman Energy GIS users are now provided access to a larger repository of raster data. Overall the project has added value in terms of efficiency, flexibility and cost effectiveness as well as providing a basis on which to meet the future challenges that can be expected. 63 REFERENCES Baumann, Peter, Andreas Dehmel, Paula Furtado, Roland Ritsch, and Norbert Widmann. 1999. “Spatio-Temporal Retrieval with RasDaMan.” Proceedings of the 25th International Conference on Very Large Data Bases, Edinburgh, Scotland, September 7-10. Childs, Colin. 2010. “On-the-Fly Processing and Dynamic Raster Mosaicking Mosaic Datasets Resolve Many Traditional Raster Management Issues.” ArcUser Summer 2010. Accessed August 02, 2015. http://www.esri.com/news/arcuser/0610/files/mosaicdataset.pdf. Croswell, Peter L. 2009. The GIS Management Handbook: Concepts, Practices, and Tools for Planning, Implementing, and Managing Geographic Information System Projects and Programs. Frankfort, Kentucky: Kessey Dewitt Publications. Fourest, Sébastien. 2012. Satellite Imagery: From Acquisition Principles to Processing of Optical Images for Observing the Earth. Toulouse, France: Cépaduès Éditions. Giovalli, Martina, and Guido Lemoine. 2013. “Geo-Correction of High-Resolution Imagery Using Fast Template Matching on a GPU in Emergency Mapping Contexts.” Remote Sensing 5 (9): 4488-502. Godfrey, Bruce, and Hayley Eveleth. 2015. “An Adaptable Approach for Generating Vector Features from Scanned Historical Thematic Maps Using Image Enhancement and Remote Sensing Techniques in a Geographic Information System.” Journal of Map & Geography Libraries 11 (1): 18-36. Gutierrez, Angelica Garcia, and Peter Baumann. 2008. “Computing Aggregate Queries in Raster Image Databases Using Pre-Aggregated Data.” Lecture Notes in Engineering and Computer Science 2173 (1): 201-6. Kunkel, Ralf, Jürgen Sorg, Robert Eckardt, Olaf Kolditz, Karsten Rink, and Harry Vereecken. 2013. “TEODOOR: A Distributed Geodata Infrastructure for Terrestrial Observation Data.” Environmental Earth Sciences 69 (2): 507-21. Li, Deren, Jie Shan, and Jianya Gong. 2009. Geospatial Technology for Earth Observation. New York: Springer. Mao, Qingzhou, Zhou Baoding, Zou Qin, and Qingquan Li. 2013. “Efficient and Lossless Compression of Raster Maps.” Signal, Image and Video Processing 9 (1): 133-145. Meehan, William, Robert G. Brook, and Jessica Wyland. 2012. “GIS in Energy and Utilities.” In Springer Handbook of Geographic Information. by David M. Danko and Kresse Wolfgang, 887-910. Berlin: Springer Berlin Heidelberg. 64 Peters, Dave. 2008. Building a GIS: System Architecture Design Strategies for Managers. Redlands, California: ESRI Press. Quan, Kristene. 2014. “Eyes in the Sky, On the Cheap.” Canadian Business 87 15/16: 12-13. Repsol. 2015. “Repsol announces a new organizational structure following the integration of Talisman Energy.” Press Release, May 08. Accessed August 02, 2015. http://www.repsol.com/es_en/corporacion/prensa/notas-de-prensa/ultimas- notas/08052015-repsol-pone-en-marcha-su-nueva-organizacion-tras-integrar- talisman.aspx. Rifaie, Mohammad, Erwin Blas, Abdel Muhsen, Terrance Mok, Keivan Kianmehr. Reda Alhajj, and Mick Ridley. 2008. “Data Warehouse Architecture for GIS Applications.” Paper presented at 10th International Conference on Information Integration and Web-based Applications & Services (iiWAS), Linz, Austria, December 24-26. Saleh, Mortaza, Tahere Yaghoobi, and Ahmad Faraahi. 2012. “Suitability of Service Oriented Architecture for Solving GIS Problems.” International Journal of Advanced Information Technology 2 2: 1-11. Simon, Alan. 2014. Modern Enterprise Business Intelligence and Data Management: A Roadmap for IT Directors, Managers, and Architects. San Diego: Morgan Kaufmann. Sweet, Michael. and John Lucotch. 2011. “Services Meet Management and Analysis Needs: Furnishing Raster Data for Projects Large and Small.” ArcUser Spring 2011: 36-39. Wężyk, Piotr, Marta Szostak, Wojciech Krzaklewski, Marek Pająk, Marcin Pierzchalski, Piotr Szwed, Paweł Hawryło, and Michał Ratajczak. 2015. “Landscape Monitoring of Post- Industrial Areas Using LiDAR and GIS Technology.” Geodesy and Cartography 64 (1): 125-37. Zeigler, Michael, 2010. Modeling Our World: The ESRI Guide to Geodatabase Concepts. Redlands, California: ESRI Press.

Abstract (if available)

Abstract Spatial data is used in many industries and it is common for large organizations to internally manage and employ spatial data to assist with operations. The intent of this thesis is to identify and accommodate existing technical architecture strategies, user requirements, and platform software in order to design an enterprise data management solution specific to raster spatial data used within an international oil and gas exploration organization. Company specific data management objectives are identified and assessed in terms of their viability with respect to data acquisition, management, and dissemination. Raster data model improvements were considered necessary to provide web and third party applications with centrally hosted data. Raster technology innovations are examined within the context of enterprise data management practices, providing use case examples that leverage database raster management technology. High and low-level components for the design of the enterprise data management solution are described in detail. Components specific to the mosaic data model and its associated data dependencies are detailed as low-level design components. Subsequent to the implementation of the system, a technology assessment was undertaken to identify the raster workflow benefits associated with raster data model enhancements. Critical analysis of both the strengths and weaknesses of the system with respect to current and future business operations is provided. In summary, the original objectives are revisited to assess their achievement and future considerations for the continuing management of the system are investigated.

Linked assets

University of Southern California Dissertations and Theses

Conceptually similar

PDF

GIS data curation and Web map application for La Brea Tar Pits fossil occurrences in Los Angeles, California

PDF

The role of GIS in asset management: County of Kauai Department of Parks and Recreation a need for an asset management program

PDF

Integrating spatial visualization to improve public health understanding and communication

PDF

Testing LANDIS-II to stochastically model spatially abstract vegetation trends in the contiguous United States

PDF

Social media canvassing using Twitter and Web GIS to aid in solving crime

PDF

A Web GIS application for airport pavement management

PDF

Finding environmental opportunities for early sea crossings: an agent-based model of Middle to Late Pleistocene Mediterranean coastal migration

PDF

Modeling patient access to point-of-care diagnostic resources in a healthcare small-world network in rural Isaan, Thailand

PDF

Using pattern oriented modeling to design and validate spatial models: a case study in agent-based modeling

PDF

Wake County District Overlay: an online electoral data visualization application

PDF

Building a spatial database for agricultural record keeping and management on a regenerative farm

PDF

Address points and a maser address file: improving efficiency in the city of Chino

PDF

Applying GIS to landscape irrigation systems: a case study of the Music Academy of the West campus in Montecito, CA

PDF

Radio frequency identification queuing & geo-location (RAQGEO): a spatial solution to inventory management at XYZ Logistics, Inc.

PDF

Development of a mobile GIS high-water mark data collection application for the Mississippi River Basin

PDF

GIS-aided production of equipment locator maps for Metro rail maintenance and support

PDF

Deriving traverse paths for scientific fieldwork with multicriteria evaluation and path modeling in a geographic information system

PDF

Cartography for visualizing anthropogenic threats: a semiotic approach to communicating threat information in 3-D spatial models

PDF

A user study of GIS infused genealogy with dynamic thematic representation and spatiotemporal control

PDF

Distribution and correlates of feral cat trapping permits in Los Angeles, California

Asset Metadata

Creator Scobie, David Robert (author)

Core Title Design and implementation of an enterprise spatial raster management system

School College of Letters, Arts and Sciences

Degree Master of Science

Degree Program Geographic Information Science

Publication Date 04/18/2018

Defense Date 01/19/2016

Publisher University of Southern California (original), University of Southern California. Libraries (digital)

Tag enterprise,mosaic data model,OAI-PMH Harvest,raster,satellite,spatial data management,UAV

Format application/pdf (imt)

Language English

Contributor Electronically uploaded by the author (provenance)

Advisor Kemp, Karen (committee chair), Chiang, Yao-Yi (committee member), Swift, Jennifer (committee member)

Creator Email scobie.rob@gmail.com,scobie@usc.edu

Permanent Link (DOI) https://doi.org/10.25549/usctheses-c40-231566

Unique identifier UC11278340

Identifier etd-ScobieDavi-4276.pdf (filename),usctheses-c40-231566 (legacy record id)

Legacy Identifier etd-ScobieDavi-4276.pdf

Dmrecord 231566

Document Type Thesis

Format application/pdf (imt)

Rights Scobie, David Robert

Type texts

Source University of Southern California (contributing entity), University of Southern California Dissertations and Theses (collection)

Access Conditions The author retains rights to his/her dissertation, thesis or other graduate work according to U.S. copyright law. Electronic access is being provided by the USC Libraries in agreement with the a...

Repository Name University of Southern California Digital Library

Repository Location USC Digital Library, University of Southern California, University Park Campus MC 2810, 3434 South Grand Avenue, 2nd Floor, Los Angeles, California 90089-2810, USA