Evaluating a new open-source, standards-based framework for web portal development in the geosciences
Web portals are one of the principal ways geospatial information can be communicated to the public. A few prominent USGS examples are the Geo Data Portal (http://cida.usgs.gov/gdp/ [URL is accessible with Google Chrome]), EarthExplorer (http://earthexplorer.usgs.gov/), the former Derived Downscaled Climate Projection Portal, the Alaska Portal Map (http://alaska.usgs.gov/portal/), the Coastal Change Hazards Portal (http://marine.usgs.gov/coastalchangehazardsportal/), and The National Map (http://nationalmap.gov/). Currently, web portals are developed at relatively high effort and cost, with web developers working with highly skilled data specialists on custom solutions that meet user needs. To address this issue, the Australian National Government funded the development of an open-source framework for building web portals called TerriaJS, which began in early 2015 (TerriaJS, 2015). TerriaJS takes advantages of capabilities in modern browsers to deliver a browser-only solution that consumes web map services from Esri and Open Geospatial Consortium (OGC), the most commonly used web map service (WMS) standards employed at the USGS and throughout the geoscience community. Because TerriaJS runs completely within the web browser, it is possible to generate custom portals by using simple configuration files that can be located anywhere on the internet. This means that basic portals based on WMSs can be created by non-JavaScript developers such as scientists, environmental managers, and emergency response support personnel. It also means they can be constructed rapidly, in hours or days instead of weeks or months. Finally, TerriaJS could also reduce the development cost of more sophisticated portals by providing a broad framework that covers a wide range of common portal mapping needs.
While TerriaJS appeared promising, we needed to more closely examine its capabilities, potential, and risks to fully understand its value. What are the generic portal needs that can be addressed by the existing framework? Does the framework architecture allow expansion to address more sophisticated needs beyond basic web service mapping? Does the framework have sufficient documentation and interaction or encouragement from the lead developers to actually function as a community-driven open-source project, or are enhancements only reasonably created by core developers? The project team planned to address these questions by taking a deep dive into TerriaJS; adding specific enhancements needed to support access to meteorologic, oceanographic, and hydrologic model data; and using the framework to create several web portals using a combination of developer resources from the USGS Office of Water Information and the Australian CSIRO/Data61 team.
Principal Investigator : Richard P Signell
Cooperator/Partner : Christopher Barker, Patricia (Soupy) A Dalyander, Cameron Hunt, Richard Knudsen, Kevin Ring, Jordan I Walker
Accomplishments
The project team achieved the main objective of assessing the role of TerriaJS in the USGS suite of available tools for creating web services. Using a combination of USGS and Data61 developer resources, the team made critical enhancements for dealing with meteorologic, oceanographic, and hydrologic model data; tested deployment in the USGS computational environment; and developed several demonstrations using TerriaJS, which allowed them to assess performance and ease of installation and use.
The code enhancements enabled by the CDI project were merged back into the master branch on GitHub (TerriaJS, 2015). These enhancements were controls that allow the user to select layers, change color ranges, and select styles from ncWMS and ncWMS2 services integrated into THREDDS Data Servers (fig. 11). These enhancements allow effective exploration of model data via WMS in TerriaJS, and thanks to this project, this capability is now available not only to the USGS, but to everyone.
A configuration was set up to explore multiple time-dependent datasets from geospatial standards, purported strengths of TerriaJS. The project team created a bird migration example using Cesium Markup Language (CZML), a JavaScript Object Notation (JSON) that allows representation of geometric information in space and time, to display the bird migration data, and using WMS from a National Oceanic and Atmospheric Administration (NOAA) THREDDS server to display underlying monthly temperature data (fig. 12). This example portal is available at http://gamone.whoi.edu/terriajs/ (content no longer available) and https://cdn.rawgit.com/USGS-CMG/terriajs-dive/master/examples/bird_migr…. This example serves to show the power and flexibility of TerriaJS in handling time-dependent data, as well as ease of configuration by end users.
The USGS system for automatic deployment of services controlled by continuous integration on a code repository was also tested. The system performed as advertised; code deployed on a staging branch on Bitbucket, available at https://my.usgs.gov/bitbucket/projects/CDI/repos/terriamap/ (content no longer available) automatically was deployed on the USGS endpoint.
Lessons Learned
Developing enhancements for TerriaJS was indeed challenging. USGS developer Jordan Walker found the documentation scarce, the code complex, and even building the software tricky due to numerous changing dependencies. Some of the challenges were caused by the fact that TerriaJS switched to a new user interface (UI) just after this CDI project was initiated. Most enhancements, therefore, ended up being developed by Kevin Ring, the lead developer for TerriaJS. As documentation and experience with the new UI grow, the situation with developing enhancements is expected to improve. A positive outcome, however, is that the new UI plays nicely on mobile devices, which is essential for uptake by the USGS community.
When enhancements were made to support the ncWMS and ncWMS2 services, the project team discovered that displaying information from unstructured grid (for example, triangular grid) models in TerriaJS was extremely slow and significantly slower than the Godiva3 ncWMS2 client. They discovered this because of a combination of factors: (1) slow delivery in general by the Java-based ncWMS2 service, (2) requesting tiles in projected coordinates rather than the native geographic coordinates, and (3) requesting many more tiles than Godiva3 for the same spatial extent. This discovery has triggered new, ongoing work to understand how to make delivery of information from unstructured grid models more efficient.
Although automatic deployment to the USGS infrastructure was successful, many of the external datasets were not displayed because of the USGS requirement of only accessing data from HTTPS. This project’s use cases, designed to show integration of data from heterogeneous data services from the broad geospatial community, therefore, did not work because few external data providers supported HTTPS. It may be possible to work around this in the future by using a proxy. An additional weakness of the USGS TerriaJS endpoint is that it is only accessible via the internal USGS network. To overcome these shortcomings, the team set up a publicly-accessible TerriaJS endpoint on a USGS/WHOI cooperative computer that doesn’t have the HTTPS restrictions. This endpoint can be accessed at http://gamone.whoi.edu/terriajs (content no longer available).
The project team had hoped to bring in new USGS data from Dr. Patricia (Soupy) Dalyander to demonstrate integration of Docker in the TerriaJS platform, but security concerns with Docker prevented deployment of the THREDDS Data Server Docker container. The hope is that these security concerns are addressed by USGS in the future, as Docker makes deploying and maintaining services much easier, isolates server processes from each other, and makes services on specific machines more robust. The team would like to test deployment of the THREDDS Docker container on the USGS Cloud Hosting Solution in the near future.
Visual Representation
Figure 11. Screen capture showing surface currents from the IOOS forecast model for New England, which uses a native triangular grid, but is regridded at varying resolution by ncWMS2. Shown also are the user controls for selecting the layer, the elevation, and the color range. The ability to access ncWMS2 endpoints, as well as control the color range, elevation, and style, were all CDI enhancements to TerriaJS. These enhancements have been merged into the master branch of TerriaJS.
Figure 12. Screen capture showing bird migration from a Cesium Markup Language (CZML) data source superimposed on monthly temperature data from WMS (via a NOAA THREDDS server). Shown also are the user controls added in this project for selecting the style and color range. Note that both the CZML file and the JSON configuration for this demo are on GitHub at https://github.com/USGS-CMG/terriajs-dive, and referenced directly in the URL of the portal. This means that users can create custom portals on their own, without any interaction from the provider of the TerriaJS endpoint, which just serves to deliver the TerriaJS code to the browser. TerriaJS runs completely within the browser.
Note: This description is from the Community for Data Integration 2016 Annual Report.
- Source: USGS Sciencebase (id: 56d87a7de4b015c306f6cfcf)
Web portals are one of the principal ways geospatial information can be communicated to the public. A few prominent USGS examples are the Geo Data Portal (http://cida.usgs.gov/gdp/ [URL is accessible with Google Chrome]), EarthExplorer (http://earthexplorer.usgs.gov/), the former Derived Downscaled Climate Projection Portal, the Alaska Portal Map (http://alaska.usgs.gov/portal/), the Coastal Change Hazards Portal (http://marine.usgs.gov/coastalchangehazardsportal/), and The National Map (http://nationalmap.gov/). Currently, web portals are developed at relatively high effort and cost, with web developers working with highly skilled data specialists on custom solutions that meet user needs. To address this issue, the Australian National Government funded the development of an open-source framework for building web portals called TerriaJS, which began in early 2015 (TerriaJS, 2015). TerriaJS takes advantages of capabilities in modern browsers to deliver a browser-only solution that consumes web map services from Esri and Open Geospatial Consortium (OGC), the most commonly used web map service (WMS) standards employed at the USGS and throughout the geoscience community. Because TerriaJS runs completely within the web browser, it is possible to generate custom portals by using simple configuration files that can be located anywhere on the internet. This means that basic portals based on WMSs can be created by non-JavaScript developers such as scientists, environmental managers, and emergency response support personnel. It also means they can be constructed rapidly, in hours or days instead of weeks or months. Finally, TerriaJS could also reduce the development cost of more sophisticated portals by providing a broad framework that covers a wide range of common portal mapping needs.
While TerriaJS appeared promising, we needed to more closely examine its capabilities, potential, and risks to fully understand its value. What are the generic portal needs that can be addressed by the existing framework? Does the framework architecture allow expansion to address more sophisticated needs beyond basic web service mapping? Does the framework have sufficient documentation and interaction or encouragement from the lead developers to actually function as a community-driven open-source project, or are enhancements only reasonably created by core developers? The project team planned to address these questions by taking a deep dive into TerriaJS; adding specific enhancements needed to support access to meteorologic, oceanographic, and hydrologic model data; and using the framework to create several web portals using a combination of developer resources from the USGS Office of Water Information and the Australian CSIRO/Data61 team.
Principal Investigator : Richard P Signell
Cooperator/Partner : Christopher Barker, Patricia (Soupy) A Dalyander, Cameron Hunt, Richard Knudsen, Kevin Ring, Jordan I Walker
Accomplishments
The project team achieved the main objective of assessing the role of TerriaJS in the USGS suite of available tools for creating web services. Using a combination of USGS and Data61 developer resources, the team made critical enhancements for dealing with meteorologic, oceanographic, and hydrologic model data; tested deployment in the USGS computational environment; and developed several demonstrations using TerriaJS, which allowed them to assess performance and ease of installation and use.
The code enhancements enabled by the CDI project were merged back into the master branch on GitHub (TerriaJS, 2015). These enhancements were controls that allow the user to select layers, change color ranges, and select styles from ncWMS and ncWMS2 services integrated into THREDDS Data Servers (fig. 11). These enhancements allow effective exploration of model data via WMS in TerriaJS, and thanks to this project, this capability is now available not only to the USGS, but to everyone.
A configuration was set up to explore multiple time-dependent datasets from geospatial standards, purported strengths of TerriaJS. The project team created a bird migration example using Cesium Markup Language (CZML), a JavaScript Object Notation (JSON) that allows representation of geometric information in space and time, to display the bird migration data, and using WMS from a National Oceanic and Atmospheric Administration (NOAA) THREDDS server to display underlying monthly temperature data (fig. 12). This example portal is available at http://gamone.whoi.edu/terriajs/ (content no longer available) and https://cdn.rawgit.com/USGS-CMG/terriajs-dive/master/examples/bird_migr…. This example serves to show the power and flexibility of TerriaJS in handling time-dependent data, as well as ease of configuration by end users.
The USGS system for automatic deployment of services controlled by continuous integration on a code repository was also tested. The system performed as advertised; code deployed on a staging branch on Bitbucket, available at https://my.usgs.gov/bitbucket/projects/CDI/repos/terriamap/ (content no longer available) automatically was deployed on the USGS endpoint.
Lessons Learned
Developing enhancements for TerriaJS was indeed challenging. USGS developer Jordan Walker found the documentation scarce, the code complex, and even building the software tricky due to numerous changing dependencies. Some of the challenges were caused by the fact that TerriaJS switched to a new user interface (UI) just after this CDI project was initiated. Most enhancements, therefore, ended up being developed by Kevin Ring, the lead developer for TerriaJS. As documentation and experience with the new UI grow, the situation with developing enhancements is expected to improve. A positive outcome, however, is that the new UI plays nicely on mobile devices, which is essential for uptake by the USGS community.
When enhancements were made to support the ncWMS and ncWMS2 services, the project team discovered that displaying information from unstructured grid (for example, triangular grid) models in TerriaJS was extremely slow and significantly slower than the Godiva3 ncWMS2 client. They discovered this because of a combination of factors: (1) slow delivery in general by the Java-based ncWMS2 service, (2) requesting tiles in projected coordinates rather than the native geographic coordinates, and (3) requesting many more tiles than Godiva3 for the same spatial extent. This discovery has triggered new, ongoing work to understand how to make delivery of information from unstructured grid models more efficient.
Although automatic deployment to the USGS infrastructure was successful, many of the external datasets were not displayed because of the USGS requirement of only accessing data from HTTPS. This project’s use cases, designed to show integration of data from heterogeneous data services from the broad geospatial community, therefore, did not work because few external data providers supported HTTPS. It may be possible to work around this in the future by using a proxy. An additional weakness of the USGS TerriaJS endpoint is that it is only accessible via the internal USGS network. To overcome these shortcomings, the team set up a publicly-accessible TerriaJS endpoint on a USGS/WHOI cooperative computer that doesn’t have the HTTPS restrictions. This endpoint can be accessed at http://gamone.whoi.edu/terriajs (content no longer available).
The project team had hoped to bring in new USGS data from Dr. Patricia (Soupy) Dalyander to demonstrate integration of Docker in the TerriaJS platform, but security concerns with Docker prevented deployment of the THREDDS Data Server Docker container. The hope is that these security concerns are addressed by USGS in the future, as Docker makes deploying and maintaining services much easier, isolates server processes from each other, and makes services on specific machines more robust. The team would like to test deployment of the THREDDS Docker container on the USGS Cloud Hosting Solution in the near future.
Visual Representation
Figure 11. Screen capture showing surface currents from the IOOS forecast model for New England, which uses a native triangular grid, but is regridded at varying resolution by ncWMS2. Shown also are the user controls for selecting the layer, the elevation, and the color range. The ability to access ncWMS2 endpoints, as well as control the color range, elevation, and style, were all CDI enhancements to TerriaJS. These enhancements have been merged into the master branch of TerriaJS.
Figure 12. Screen capture showing bird migration from a Cesium Markup Language (CZML) data source superimposed on monthly temperature data from WMS (via a NOAA THREDDS server). Shown also are the user controls added in this project for selecting the style and color range. Note that both the CZML file and the JSON configuration for this demo are on GitHub at https://github.com/USGS-CMG/terriajs-dive, and referenced directly in the URL of the portal. This means that users can create custom portals on their own, without any interaction from the provider of the TerriaJS endpoint, which just serves to deliver the TerriaJS code to the browser. TerriaJS runs completely within the browser.
Note: This description is from the Community for Data Integration 2016 Annual Report.
- Source: USGS Sciencebase (id: 56d87a7de4b015c306f6cfcf)