Lesson 14a - Bulk Download of US Topo Maps

Download Video
Right-click and save to download

Detailed Description

Lesson 14a- Bulk Download of US Topo Maps. In this lesson you will learn how to use the internal USGS network for bulk download of US Topo Maps. Access the USGS “I-SPI” site. Set up a query by county, state bounding box, or cell IDs. Download multiple US Topo maps to your local drive
 

Details

Date Taken:

Length: 00:13:47

Location Taken: Augusta, ME, US

Transcript

This is lesson 14a, bulk download of US Topo Maps

US Topo maps are a new digital topographic map started in 2010 and available by 7.5-minute map quadrangle, like the classic paper USGS topo map. Lesson 8 provides details on the US Topo maps.  This lesson will demonstrate how USGS employees can get bulk access to US Topo maps through an internal query process.

In this lesson you will learn how to use the internal USGS network for bulk download of US Topo Maps. Access the USGS “I-SPI” site. Set up a query by county, state bounding box, or cell IDs. Download multiple US topo maps to your local drive

This map shows US Topo map availability across the country.  The different shades of green correspond to map publication dates between 2013 and 2016.

Over 55,000 maps cover the continental United States.  Alaska, once complete, will add another 11,200 maps.  Maps are also available for Hawaii, Puerto Rico, the U.S. Virgin Islands, and are planned for the Pacific territories.  Maps are targeted to be updated every three years, with the exception of Alaska.

US Topo maps are typically accessed through public  USGS sites such as The National Map Download Client or the USGS Store.  However, USGS employees and contractors have additional internal access to these maps.  If you have a need to acquire large amounts of maps, such as for an entire state or to meet a requirement spread among several states, it is possible to download even thousands of maps.

All USGS employees have access to the “Interactive Status of Product Inventories” commonly called I-SPI.  I-SPI can only be accessed when logged on to the USGS network.  Once logged on, open an internet browser and go to the I-SPI URL shown on screen

The I-SPI main page shows many types of mapping products available to review.  However, to download US Topo maps, we are interested in the Graphics theme.

This page has access to very detailed information about US Topo map availability and production.  But for the bulk download of these maps, you only need to be concerned with the “GDA Search (internal)” selection here.

The Geospatial Data Architecture (or GDA) query opens a new page in your browser.  This form is where you will specify which maps to download.  But before entering any details, first click on the “Help” tab at the upper right.

Once you are on the help page, scroll to the bottom of the page.  Click on the “US Topo and Historical Map Download Steps” link. 

This page provides details on several different ways to select maps – by listing map cell ID numbers, by defining a bounding box, or by selecting a state or county.  Each method describes how to populate the different fields in the GDA Query form.

Note that this the page opened as a new page in the browser.  This makes it easier to fill in the GDA Query form while being able to refer to the help page.

Click on the GDA Query and Download Application page and then click on the “Text Query” tab to show the form to fill out.  Then go back to the US Topo and Historical Map download Steps page

In this example, we will use the instructions on downloading maps for a state to generate a GDA query.

In this example, we are looking at the instructions to enter the criteria to download all the US Topo maps in a state.

There are only a few fields to fill out in the GDA Query form to request all the maps in any state.

The main consideration to note is there are different entries to make in a couple fields depending on whether you want the US Topo or the historical topo maps.  These entries are shown in the red text here

Now we will go to the GDA Query form and select all the US Topo maps for a sample state download – we will choose California

The Help documentation clearly notes any fields to complete to set up the query.

To recap, changes are made to the Name, State, Best Available, and Grid Size fields.

To set up the query and download for all new US Topo maps in California, set Name to “%tnm geopdf” then for the State field, select “California” from the list of names.  Set the Best Available  field to “YES” and verify that the Grid Size is 7.5 by 7.5 grid.  This is currently the only grid size available for US Topo maps.

Then click on the Go button at the middle right of the form.

It may take a few minutes for the query to be completed.  Results will automatically appear on the same page.

The query has now selected the most recent US Topo maps for all of California.  The query results are presented in several different formats down the page.  The one we will use to download all of California is located halfway down the page, and is conveniently labeled “Batch Download.” 

The Batch Download list is showing the first 50 maps retrieved.  In the case of California, there are 2,883 maps total

Instructions for setting up the download are at bottom of the Batch Download section.  The process is straightforward and allows you to quickly retrieve a download file.  This download file will begin downloading once it is activated.  The files will be placed in the directory where the batch file resides.  Since a batch download may transfer thousands of files, adequate disk space is important.  The batch download list provides an estimated size of the data to download.  In the case of US Topo maps, the size of the compressed files is approximately equivalent to the uncompressed size.

Once you ask the system to generate the Batch Download file, the file is quickly created. 

You will be used whether you want to run or save the file.  Save the file and then move it to the hard drive and directory to contain the entire download.

The batch download file is primarily a listing of the internal USGS addresses of all the US Topo files requested in the I-SPI query.

The batch download file initiates the “Wget” transfer process and then contains two lines of text for each US Topo map file to be downloaded.  Depending on the size of your query, the batch download file may contain thousands of entries.

Once the batch download file is in the right storage location, simply click on the filename in Windows Explorer to start the process

The small “wget” window will open and files will scroll by as they download.  Speed will vary by file size and internet speed but tests for this video were about 25 seconds per file.  Extrapolated for California that would be about 20 hours.

If you look in the directory containing the batch download file, you can see the US Topo files being stored as they download.

The files will accumulate here until the wget script is completed.

Along with the downloaded maps and the batch download file, there will be another file created.  This will be a text log file with the same name as the batch download file.

This file will record when each topo maps is downloaded and the log data can be helpful in recovering any data not initially downloaded.

If all of the maps in the Batch Download file are not initially downloaded, then the batch file will need to be modified and run again

To see how well the batch download worked, you can look in the folder holding all the downloaded files, or you can look at the log file.

Scroll down the folder showing the maps that were downloaded   

In this image, we can see a number of file names with a size of zero bytes 

Sometimes, files fail to download.  The Batch Download file will scroll through all the files in it and download those it can and skip those it cannot.  In this example, 129 maps in a row failed to download.

You can also see what files downloaded and which ones failed to download by looking at the batch download log file.

Remember, this file has the same name as the batch download file but is listed as a text document.

Open the log file to view the file list.  In this example, you can see the files that failed to download marked with an error notice.  The map with the ID number of 7999743 was the first one in the batch to fail to download.   The files listed should match the results in your target download directory.

If all the files in your query did not download, you will know it from reviewing the target download folder files and the log file.

What is needed now is to rerun the batch download file to find and download those files that failed to come across the first time.

But you do not want to restart the batch download file as it currently is.  That would begin a new download of every map in your query – you only want the ones that were skipped.

To start this, look at your download results and note which files failed to download.   While this may be a lot of files, it likely will not be large relative to the total number in your original query.  Also, files that fail to download usually occur in bunches so you can simply record the first and last files in a group that fail to download.

To complete the download, you will need to edit the Batch Download file to remove the list of filenames that successfully downloaded.

This will leave the Batch Download file containing only those filenames which failed to download.  When the revised Batch Download file is run again, the missing files will be requested again.

Open the batch download in an editor.  To do this, use Windows Explorer and right click on the batch download file and select “Edit”.  In the editor, delete the entries for the files that downloaded.  Presuming most of the file downloaded correctly, this will mean removing most of the content, which is okay.

Once the file is thinned to contain only the missing files, save the batch download file and run it again.   

To recap, I-SPI is an internal USGS resource providing details on several national mapping products.

One product, the new US Topo maps, can be downloaded in bulk using a query within I-SPI

The simple query form can be filled out to request US Topo maps for several geographic units, including statewide.

The query form is used to generate an executable batch download file which is run to download the maps.

If all the maps in the query fail to download, the batch form can be modified and run again.

This concludes lesson 14a