Data.gov survey: Raw dataset modification dates

Description: 
This demonstration shows when the data.gov datasets were last updated, as determined by their HTTP header responses. How up-to-date are the datasets listed in data.gov? We start with a time line histogram for a moment in time.
Contributor:
Try the new version to compare crawls.
On the afternoon of 21 Sept, 2010, we crawled data.gov's links to data sources and recorded the modification dates reported by the source agency's HTTP servers. Of __ data.gov datasets, __ datasets returned HTTP last-modified dates. The list of data.gov datasets came from the 13 Sept 2010 version of their Dataset 92.

Process used

The following diagram shows the process used to obtain the last-modified dates of the data.gov datasets.

Use of provenance

The following diagram illustrates how the Proof Markup Language was used to record the HTTP headers obtained from the government agency servers. A portion of this data graph is selected by the SPARQL query listed above (the headers are excluded). The red links show the HTTP redirects and HTML anchor hrefs from the data.gov details page to the actual dataset URL on the hosting agency's domain. The green boxes represent URIs for the information obtained (HTTP headers), while the purple boxes associate the information received with its source and when it was received. For more discussion on how provenance is used in LOGD and csv2rdf4lod, see A look at how csv2rdf4lod incorporates provenance into its tabular conversions.

Uses Dataset: 
Uses Technology: 
Uses Technology: 
Uses Technology: 
Uses Technology: 
Uses Technology: 
Uses Technology: 
Uses Technology: 
Thumbnail: 
Your rating: None Average: 5 (1 vote)