Dynamic RSS generation using SPARQL and Yahoo Pipes

View Source Code of "Dynamic RSS generation using SPARQL and Yahoo Pipes"

<p>Following is a step-by-step explanation on the workflow of this demo.</p>
<div style="color: #2b3856; font-weight: bold;"><span style="background: none repeat scroll 0% 0% #2b3856; color: white; padding: 3px;">Step 1</span> 
locate RDF data (TWC LOGD in the News)</div>

<p>The RDF data used in this demo is collected by parsing RDFa metadata embeded on web pages on http://data-gov.tw.rpi.edu, and here is a <a href=" http://data-gov.tw.rpi.edu/wiki/In_the_News:_White_House_Cites_Rensselaer_Polytechnic_Institute_Web_Science_Research_on_One-Year_Anniversary_of_Data.gov">link to an example page</a>. The RDFa annotation of the example page can be extracted by any RDFa parser, and here is a <a href="http://data-gov.tw.rpi.edu/ws/lodcx.php?url=http%3A%2F%2Fdata-gov.tw.rpi.edu%2Fwiki%2FIn_the_News%3A_White_House_Cites_Rensselaer_Polytechnic_Institute_Web_Science_Research_on_One-Year_Anniversary_of_Data.gov&operation=test">link to the parse result</a>. </p>
<div style="color: #2b3856; font-weight: bold;"><span style="background: none repeat scroll 0% 0% #2b3856; color: white; padding: 3px;">Step 2</span> compose a SPARQL query to prepare the content of RSS</div>
<p>The following <a href="http://logd.tw.rpi.edu/demo/feed/twc-logd-in-the-news.rss.sparql">SPARQL query</a> (using SELECT construct) is used in this demo. The variable names are carefully selected to match the terms used in RSS. Therefore, users should keep using the same variable names(including title, image, description, link, image, and pubDate) when they want to compose RSS content on the other RDF data.</p>
<pre>PREFIX rss: 
PREFIX dcterms: 
PREFIX foaf: 

SELECT *
WHERE{
 GRAPH ?g {
 _:x a 
    ;foaf:name ?title
    ;foaf:depiction ?image
    ;dcterms:description  ?description
    ;dcterms:source ?link
    ;dcterms:created ?pubDate
 .
}}
</pre><p>
<br />This SPARQL also have some limitations as users cannot control the tile and description of the RSS channel. Although "CONSTRUCT" primitive looks promising in solving this issue, it also has limitations. First, SPARQL results uses XSD:dateTime to encode date (e.g. "2010-05-21T00:00:00") while RSS 2.0 requires data formated in RFC822 (e.g."<a id="l22" name="l22"><code class="a">Thu, 20 May 2010 17:00:00 -0700"</code></a>). Second, the output RDF/XML syntax used "rdf:Resource" as element tag, while RSS 2.0 demands specific XML format (they are looking for "<channel>" and "<item>" instead). Therefore, we stick to "SELECT" based data construction and generate RSS using a web service-- Yahoo! pipes in this demo.</p><div style="color: #2b3856; font-weight: bold;"><span style="background: none repeat scroll 0% 0% #2b3856; color: white; padding: 3px;">Step 3</span> 
query RDF data in a SPARQL endpoint using SparqlProxy and then get JSON back</div><p>The data in this demo are all loaded at a triple store with SPARQL endpoint  <a href="http://data-gov.tw.rpi.edu/ws/lodc.php">http://data-gov.tw.rpi.edu/ws/lodc.php</a>. For each page, it RDF metadata is stored in one named graph in the triple store. </p><p>The SparqlProxy query interface and choices of parameters can be viewed via <a href="http://logd.tw.rpi.edu/ws/sparqlproxy.php?query-option=text&query=PREFIX+rss%3A+%3Chttp%3A%2F%2Fpurl.org%2Frss%2F1.0%2F%3E%0D%0APREFIX+dcterms%3A+%3Chttp%3A%2F%2Fpurl.org%2Fdc%2Fterms%2F%3E%0D%0APREFIX+foaf%3A+%3Chttp%3A%2F%2Fxmlns.com%2Ffoaf%2F0.1%2F%3E%0D%0A%0D%0ASELECT+*%0D%0AWHERE{%0D%0A+GRAPH+%3Fg+{%0D%0A+_%3Ax+a+%3Chttp%3A%2F%2Fdata-gov.tw.rpi.edu%2Fwiki%2FCategory%3AIn_the_News%3E%0D%0A++++%3Bfoaf%3Aname+%3Ftitle%0D%0A++++%3Bfoaf%3Adepiction+%3Fimage%0D%0A++++%3Bdcterms%3Adescription++%3Fdescription%0D%0A++++%3Bdcterms%3Asource+%3Flink%0D%0A++++%3Bdcterms%3Acreated+%3FpubDate%0D%0A+.%0D%0A}}&service-uri=http%3A%2F%2Fdata-gov.tw.rpi.edu%2Fws%2Flodc.php&output=sparqljson&textoutput=on&callback=&tqx=&tp=&ui-option=query">this link</a>.</p><ul><li> check "Query URI" and paste Query URI after</li><li> check "sparqljosn"  as output format
* click "query" button</li></ul><p><br />The query results can be accessed by application via <a href="http://logd.tw.rpi.edu/ws/sparqlproxy.php?query-option=text&query=PREFIX+rss%3A+<http%3A%2F%2Fpurl.org%2Frss%2F1.0%2F>%0D%0APREFIX+dcterms%3A+<http%3A%2F%2Fpurl.org%2Fdc%2Fterms%2F>%0D%0APREFIX+foaf%3A+<http%3A%2F%2Fxmlns.com%2Ffoaf%2F0.1%2F>%0D%0A%0D%0ASELECT+*%0D%0AWHERE{%0D%0A+GRAPH+%3Fg+{%0D%0A+_%3Ax+a+<http%3A%2F%2Fdata-gov.tw.rpi.edu%2Fwiki%2FCategory%3AIn_the_News>%0D%0A++++%3Bfoaf%3Aname+%3Ftitle%0D%0A++++%3Bfoaf%3Adepiction+%3Fimage%0D%0A++++%3Bdcterms%3Adescription++%3Fdescription%0D%0A++++%3Bdcterms%3Asource+%3Flink%0D%0A++++%3Bdcterms%3Acreated+%3FpubDate%0D%0A+.%0D%0A}}&service-uri=http%3A%2F%2Fdata-gov.tw.rpi.edu%2Fws%2Flodc.php&output=sparqljson&callback=&tqx=&tp=">this link</a> and viewed in text format via <a href="http://logd.tw.rpi.edu/ws/sparqlproxy.php?query-option=text&query=PREFIX+rss%3A+%3Chttp%3A%2F%2Fpurl.org%2Frss%2F1.0%2F%3E%0D%0APREFIX+dcterms%3A+%3Chttp%3A%2F%2Fpurl.org%2Fdc%2Fterms%2F%3E%0D%0APREFIX+foaf%3A+%3Chttp%3A%2F%2Fxmlns.com%2Ffoaf%2F0.1%2F%3E%0D%0A%0D%0ASELECT+*%0D%0AWHERE{%0D%0A+GRAPH+%3Fg+{%0D%0A+_%3Ax+a+%3Chttp%3A%2F%2Fdata-gov.tw.rpi.edu%2Fwiki%2FCategory%3AIn_the_News%3E%0D%0A++++%3Bfoaf%3Aname+%3Ftitle%0D%0A++++%3Bfoaf%3Adepiction+%3Fimage%0D%0A++++%3Bdcterms%3Adescription++%3Fdescription%0D%0A++++%3Bdcterms%3Asource+%3Flink%0D%0A++++%3Bdcterms%3Acreated+%3FpubDate%0D%0A+.%0D%0A}}&service-uri=http%3A%2F%2Fdata-gov.tw.rpi.edu%2Fws%2Flodc.php&output=sparqljson&textoutput=on&callback=&tqx=&tp=">another link</a>. </p><p> </p><p><span style="background: none repeat scroll 0% 0% #2b3856; color: white; padding: 3px;">
<br/>
Step 4</span> 
build a Yahoo Pipes! web service that convert the SPARQL query results (in SPARQL/JOSN format) into an RSS2.0 data.</p><p>the code is available at <a href="http://pipes.yahoo.com/pipes/pipe.info?_id=5d8842906152a7d3152a128666970f13">http://pipes.yahoo.com/pipes/pipe.info?_id=5d8842906152a7d3152a128666970f13 </a>. Following are some notable points:</p>
<ul><li> you need a yahoo ID to access Yahoo Pipes!
</li><li>SPARQL/JOSN is used to provide access to data structure. Yahoo! Pipes has hard time deal with the XML version</li><li>representation of date is not compatible between RDF (ISO  and RSS (required RFC822)</li></ul><p><img src="http://logd.tw.rpi.edu/images/screenshot/demo-twc-logd-in-the-news-yahoo-pipes.png" alt="" width="746" height="640" /></p><p>Finally, the result RSS is generated, and Yahoo! also provided some human readable page for the RSS feed (wiht images). The RSS feed is also imported by TWC LOGD front page's RSS aggregator.</p><p><img src="http://logd.tw.rpi.edu/images/screenshot/demo-twc-logd-in-the-news.png" alt="" width="800" height="800" /></p>

To the extent possible under law, Tetherless World Constellation has waived all copyright and related or neighboring rights to TWC LOGD. TWC LOGD is an educational project on open government data using Semantic Web technologies. Datasets hosted on this site are converted from a number of data sources such as data.gov. All data created by us are open for reuse, and usage of data created and managed by other sources should follow their own licenses.

The data contained on this site is automatically repopulated from US government or other open data sites, and any personal data in our linked-data versions is coming from those sources. If your information is removed from the government sources, it will be automatically removed from ours on the next update.

Linking Open Government Data

Menu

User login

Navigation

Share

View Source Code of "Dynamic RSS generation using SPARQL and Yahoo Pipes"