Skip to main content
U.S. flag

An official website of the United States government

A guide to creating an effective big data management framework

September 26, 2023

Many agencies and organizations, such as the U.S. Geological Survey, handle massive geospatial datasets and their auxiliary data and are thus faced with challenges in storing data and ingesting it, transferring it between internal programs, and egressing it to external entities. As a result, these agencies and organizations may inadvertently devote unnecessary time and money to convey data without existing or outdated standards. This research aims to evaluate the components of data conveyance systems, such as transfer methods, tracking, and automation, to guide their improved performance. Specifically, organizations face the challenges of slow dispatch time and manual intervention when conveying data into, within, and from their systems. Conveyance often requires skilled workers when the system depends on physical media such as hard drives, particularly when terabyte transfers are required. In addition, incomplete or inconsistent metadata may necessitate manual intervention, process changes, or both. A proposed solution is organization-wide guidance for efficient data conveyance. That guidance involves systems analysis to outline a data management framework, which may include understanding the minimum requirements of data manifests, specification of transport mechanisms, and improving automation capabilities.

Publication Year 2023
Title A guide to creating an effective big data management framework
DOI 10.1186/s40537-023-00801-9
Authors Samantha Arundel, Kevin G McKeehan, Bryan B Campbell, Andrew N. Bulen, Philip T. Thiem
Publication Type Article
Publication Subtype Journal Article
Series Title Journal of Big Data
Index ID 70248950
Record Source USGS Publications Warehouse
USGS Organization NGTOC Rolla; Center for Geospatial Information Science (CEGIS)