ScienceBase Geospatial Services
Instructions for creating geospatial web services in ScienceBase.
- Shapefiles
- Rasters
- Data Symbology
- ArcGIS Mapping Services
- Service Definition Best Practices
- Access Data from a Service Definition
- ArcGIS GeoDatabase
- Connect to ScienceBase Geospatial Services
- AWS S3 Publishing for Cloud-Optimized File Formats
Map Service Integration with USGS ArcGIS Online (AGOL)
- Publish a service definition (.sd) file from ScienceBase to USGS AGOL
- Create a service definition (.sd) file for USGS AGOL in ArcPro
Please note: hosting GIS mapping services is resource intensive for the ScienceBase system, so please only create these services if users are expected to interact with the data through the service. To provide a simple preview image of data, a static image (in either .jpg or .png format) can be uploaded instead.
Shapefiles
A shapefile comprises of a set of files that can be uploaded as individual files or as a single zipped file. ScienceBase automatically generates WMS (web mapping service) and WFS (web feature service) for unzipped shapefiles. Note: an uploaded shapefile in .zip format can be unzipped during the file upload process. If a map display and GIS services are not needed, the uploaded shapefiles can remain zipped.
A projection (.prj) file must be included in the shapefile bundle to render and create GIS services. The WMS and WFS service URLs are displayed in the "Spatial Services" section under the preview map on a ScienceBase item (note: an item needs to have public view settings for these GIS Services to be accessible). The default symbology is a solid blue outline. WGS84 Web Mercator projection is recommended if services will be combined with publicly available basemaps (e.g., from ArcGIS Online or Google).
Rasters
Any raster format can be uploaded to ScienceBase; however, only GeoTIFF (.tif) files can be parsed by ScienceBase and used to create GIS services. The automatically generated WMS and WCS URLs are displayed in the "Spatial Services" section under the preview map on a ScienceBase item. Note: an item needs to have public view settings for these GIS Services to be accessible. A blue color ramp is the default symbology. WGS84 Web Mercator projection is recommended if services will be combined with publicly available basemaps (e.g., from ArcGIS Online or Google).
Data Symbology
For both shapefiles and rasters, ScienceBase will default to blue symbology. To customize your symbology, you can use one of the following methods:
- Upload an ESRI service definition (.sd) file to ScienceBase. The process is explained below in the 'ArcGIS Mapping Services' section.
- Add open source symbology to an uploaded shapefile or raster. For example, Styled Layer Descriptor (SLD) files can be included with shapefile or GeoTIFF data to add custom symbology rules to a WMS display. SLD (.sld) files contain content in XML format. The specification for the SLD format is available at https://www.ogc.org/standards/sld. Note: an SLD file needs to be named with the same file prefix as the related shapefile or GeoTIFF for ScienceBase to recognize it as a symbology layer.
Shapefile:
- Example instructions on how to create a .sld file for a shapefile in QGIS: https://docs.geoserver.org/stable/en/user/styling/qgis/index.html
Raster (two options):
- Example instructions on how to create a .sld file for a raster in QGIS: https://docs.geoserver.org/stable/en/user/styling/qgis/index.html
- Add a colormap to a GeoTIFF file bundle (for single-band rasters that are stored as 8 or 16-bit unsigned integer). If the GeoTIFF contains a colormap, ScienceBase will render the raster accordingly. To create a colormap in ArcMap: https://desktop.arcgis.com/en/arcmap/latest/manage-data/raster-and-images/creating-a-color-map-clr-file.htm
ArcGIS Mapping Services
A service definition (.sd) file must be uploaded as an extension to generate an ArcGIS REST service. Service definition file uploads automatically create ArcGIS REST services for items that are publicly readable (as set in the item’s permissions). For items that are not publicly readable, you must take an extra step to create your ArcGIS service. In the "Item Actions" section at the bottom of the page, go to the "Manage ArcGIS Service Definition" section and click on "Create ArcGIS Services". Please note: the ArcGIS service will be publicly accessible, even if the ScienceBase item permissions are restricted.
After the service has been created, the service URL will be displayed under the preview map in the upper right corner of the item. The "Manage ArcGIS Service Definition" section will also display an option to view the service endpoint:
To create a service definition file using ArcPro, please see the instructions and best practices in ESRI ArcPro documentation, e.g., https://pro.arcgis.com/en/pro-app/3.1/help/sharing/overview/introduction-to-saving-service-definitions.htm.
Service Definition Best Practices
General rules for creating a usable service definition (.sd) file and ArcGIS REST service:
- ScienceBase file size limits: single .sd file uploads should not exceed 1-2GB. ScienceBase can process uploaded .sd files if the unzipped content does not exceed 4GB.
- If you need to make a change to an existing ArcGIS REST Service, please delete the old .sd extension first and then add in the new one.
- ArcGIS REST service URLs are persistent, even if uploaded .sd files are replaced. For example, the service URL https://gis.usgs.gov/sciencebase3/rest/services/Catalog/620408c1d34e622189de5ad6/MapServer contains the ScienceBase item ID, which is a unique persistent identifier, and isn’t dependent on an individual file ID. A benefit of persistent service URLs is that if a URL is coded into an application that uses or displays the service, the data can be updated in ScienceBase without necessitating changes to the downstream application.
Tips for ESRI service definition files that maximize REST service performance:
- Please note: creating a .sd file will copy the ENTIRE dataset from your ArcPro project, not just what is seen in the current data frame extent. This has the potential to create very large, unmanageable files.
- In the process of creating an .sd file, make an attempt to fix "Warnings" and "Informational Messages" in the "Analyze" step to further improve performance.
- Make sure all datasets and data frames use the same coordinate system. Failure to do so can result in a service slowed by on-the-fly projection.
- If the service will be combined with publicly available basemaps (e.g., from ArcGIS Online), it is strongly recommended to use a projection that matches the standard coordinate system of these basemaps: Projected Coordinate Systems > World > WGS 1984 Web Mercator (auxiliary sphere). Failure to do so can result in a service slowed by on-the-fly projection.
- When authoring your map document, set large complex datasets to only display when zoomed-in.
- Avoid using highly complex, large datasets that contain too many vertices. Examples of datasets that can cause a slow service include raster datasets that have been converted to vector (the original raster dataset should be used instead) and complex multipolygons with many vertices.
- Use the "Simplify" tool to help overcome the problem of a slow service with too many vertices: ArcToolbox > Cartography Tools > Generalization > Simplify Line/Polygon. Use the "point remove" method and experiment to find the right balance of speedy performance versus accurate representation of the data.
- Click File > Map Document Properties and fill in the form to the best of your abilities. This will be used as metadata for your service.
- Make sure to generate pyramids for all raster datasets.
- Large raster datasets should use compression. We recommend converting all rasters to .tif format using LZW or LZ77 compression. Failure to do so can lead to unfavorable file size, slower performance, and problems in the unpackaging/unzipping process.
Access data from a service definition (.sd) file
A service definition file is a zipped bundle that contains the service properties, capabilities, and the original GIS data. To access the data on a local system, download the .sd file from ScienceBase and use unzipping software to unzip the file.
ArcGIS GeoDatabase
A GeoDatabase should be zipped prior to uploading to ScienceBase. During the upload process, please do not select the unzip option. Geodatabases cannot be parsed or displayed in a map by ScienceBase. Users can download the zipped GeoDatabase file from ScienceBase and then unzip the file for use on local systems.
Connect to ScienceBase Geospatial Services
For an ArcGIS REST service, open to the REST service URL in a web browser to view the connection options:
For a Web Map Service (WMS), use an application that can parse and display WMS services:
For a Web Feature Service (WFS) or Coverage Service (WCS), use an application, such as ArcGIS Pro or ArcGIS Online, that can parse and display WFS and WCS services.
AWS S3 Publishing: Supporting Direct File Access from Storage for Cloud-Optimized File Formats
Please note: this feature is intended for finalized and approved USGS data products that have been structured for use with this type of access protocol.
ScienceBase has a feature that allows an authorized user to publish files to a publicly readable ScienceBase AWS S3 bucket. When performed, the files are moved to a dedicated ScienceBase bucket where they are directly readable from object storage at a Uniform Resource Identifier (URI). The ability to access a file in this way (i.e., direct read from cloud storage) has multiple benefits for data stored in cloud optimized formats. The first use case that this workflow supports is for Cloud Optimized GeoTIFFs (COG) files.
Raster files are often large in size, for example, when the data cover a large spatial extent or are captured at a high spatial resolution. A COG file provides the same data and functionality as a traditional GeoTIFF, but includes additional information in the file to support faster rendering of the dataset (pyramids) and subsetting over http protocol. Publishing raster files in a COG format and storing them in a publicly readable location allows users to access the data (the full dataset or geographically bounded subsets), view it dynamically in a map viewer, and pull raw values directly into a programmatic workflow (e.g., in Python or R). This enables more flexible code-based workflows and, in many cases, eliminates the need for downstream users to download entire files.
To make a file available to cloud workflows, navigate to the ScienceBase File Manager, which can be accessed from the "Item Actions" section on ScienceBase items:
Click the "Actions Menu" dropdown arrow next to the cloud optimized file and select "Publish." This action will add a copy of the file to the public ScienceBase S3 bucket and will add the public S3 URI to the item JSON, which can then be shared with users.
Map Service Integration with USGS ArcGIS Online (AGOL)
The ScienceBase team has a newly developed workflow to pull an existing ESRI service definition (.sd) file from on-premise storage and make a small modification to the file to work with the USGS ArcGIS Online (AGOL) instance. An automated process can then move the file to ScienceBase cloud storage, push the file to AGOL to start an ESRI map service, and update the ScienceBase item JSON with the new map service URL.
This will result in an updated map service URL displayed on the item and in the item's JSON, but otherwise the item will remain unchanged.
Please note:
- Data should be reviewed and approved for public sharing (as USGS data release or other finalized product) and cannot contain any personally identifiable information (PII), controlled unclassified information (CUI), or sensitive data.
- The ESRI services in AGOL are fully public if created in this route from ScienceBase. The ScienceBase connection to AGOL has been granted elevated permissions to support public service generation for defined cases as outlined above. No additional steps are required from the author to make services public.
- Services created in this way are not visible to a user when personally logged into AGOL; they can only be managed through ScienceBase. Starting and stopping the service, or removing the .sd file, are all performed from the item page in ScienceBase.
- Service definition (.sd) files must be properly formatted for AGOL from a licensed USGS instance of ESRI ArcMap or ArcPro. See instructions below for creating an .sd file in ArcMap and ArcPro.
Publish a service definition (.sd) file from ScienceBase to USGS AGOL
Upload the properly formatted .sd file using the ScienceBase File Manager (not the traditional 'Manage Item' > 'Edit Item' method). The File Manager can be accessed from the "Item Actions" section at the bottom of the page.
After the .sd file is uploaded to ScienceBase cloud storage, publish the file to the publicly accessible ScienceBase S3 bucket:
Users will then see the option to publish the .sd file to the USGS AGOL space:
Users will receive an email notification to let them know when the service has been created (note: there may be a delay before the email is received).
How to create a service definition (.sd) file for USGS AGOL in ArcPro
For the complete set of instructions, see the ESRI documentation: https://pro.arcgis.com/en/pro-app/latest/help/sharing/overview/save-a-service-definition-for-a-map-service.htm.
Start by working within local GIS software to create the dataset.
Note: USGS users must be logged in to ArcGIS Online (AGOL) within ArcPro before creating the .sd file. If an .sd file is created while the user is logged out, the service will not work in AGOL.
Log in with the 'USGS PIV Login' option:
Right click on the name of the map in the contents pane, then select "Save As Offline Service Definition":
Select 'Analyze' to identify errors or warnings:
Once there are no errors or warnings, select 'Save'.
