# xqia.txt - UTF-8 # XQIA - XQuery in Action # # (c) 2015 Alexander Holupirek, Christian Grün # DBIS Group, U Konstanz # XML Technologies, Winter Term 2015 ################################################################################ ### Scope After being introduced to 'XML Basics' in the first lecture and having a glance at 'XPath' and the foundation of 'XQuery' in lecture two and three, we want to give you a 'hands-on' experience entitled 'XQuery in Action'. As XQuery processor we will use BaseX in its stand-alone GUI variant. ################################################################################ ################################################################################ ### BaseX XQuery processor - Java 7 is required. - Download XPath processor ( BaseX.jar ) $ wget http://files.basex.org/releases/BaseX.jar # Create database files in current directory. $ touch .basexhome # Create database from input file $ java -cp BaseX.jar org.basex.BaseX -c "create database foo foo.xml.gz" # Print some info about newly created database $ java -cp BaseX.jar org.basex.BaseX -c "open foo; info database; info index;" # Start BaseX GUI to formulate and evaluate XQuery interactively $ java -jar BaseX ################################################################################ ### XML Data We will work with snippets from Discogs, short for discographies, a website and database of information about audio recordings. Original data can be downloaded at: -------------------------------------------------------------------------------- http://www.discogs.com/data/ Here you will find monthly dumps of Discogs Release, Artist, Label, and Master Release data. The data is in XML format and formatted according to the API spec: http://www.discogs.com/developers/ This data is made available under the CC0 No Rights Reserved license: http://creativecommons.org/about/cc0 -------------------------------------------------------------------------------- Snippets we are working with are contained in the xml directory 145K MD5 (xml/Nirvana.xml) = 5fa1c3103fab433492aa2c841b20da62 9.9M MD5 (xml/ACDC.xml) = cb50164fb116462c2590158a02b8779f 20M MD5 (xml/BobDylan.xml) = efd0697cf049e002e0dbd6cd07c6d821 49M MD5 (xml/Beatles.xml) = dc4ecc43d39063c942b1330a6f153b98 ################################################################################ ################################################################################ ### Exploring the data with XQuery Let's start by loading the Nirvana document into BaseXGUI [Screenshot: gfx/1.png] -- xquery ---------------------------------------------------------------------- doc('./xml/Nirvana.xml') -- result ---------------------------------------------------------------------- … the complete XML document … -------------------------------------------------------------------------------- • seems to be an important container. ================================================================================ Q: How many releases do we have for Nirvana? -- xquery ---------------------------------------------------------------------- count(doc('./xml/Nirvana.xml')//release) -- result ---------------------------------------------------------------------- 30 -------------------------------------------------------------------------------- ================================================================================ Q: How are they entitled? -- xquery ---------------------------------------------------------------------- doc('./xml/Nirvana.xml')//release/title -- 30 results ------------------------------------------------------------------ Nevermind Nevermind In Utero MTV Unplugged In New York MTV Unplugged In New York MTV Unplugged In New York Nevermind Unplugged In New York Nevermind Nevermind Bleach + Bonus Tracks Nevermind Bleach From The Muddy Banks Of The Wishkah In Utero From The Muddy Banks Of The Wishkah MTV Unplugged In New York From The Muddy Banks Of The Wishkah Bleach Bleach Nevermind In Utero Nevermind Bleach Bleach From The Muddy Banks Of The Wishkah Bleach MTV Unplugged In New York Nevermind Nevermind -------------------------------------------------------------------------------- • A lot of duplicates? Where do they come from? • Let's create something like a report about the releases. ================================================================================ Task: Nirvana release report. The report shall give a compact overview about the Nirvana releases and their differences. -------------------------------------------------------------------------------- Step 1: Basic report setup -- xquery ---------------------------------------------------------------------- let $db := doc('./xml/Nirvana.xml') (: our 'database', i.e., the xml file :) let $releases := $db//release (: release nodes :) return (: constructing a result XML structure :) element report { attribute number-of-releases { count($releases) } } -- result ---------------------------------------------------------------------- -------------------------------------------------------------------------------- • A first XQuery expression (not yet a FLWOR but still …) • XQuery comments • Two let clauses declare variables • Evaluation of an XPath step bound to a variable • XML construction -------------------------------------------------------------------------------- Step 2: Augment report Remark: [xml/release.xml] shows a single element (for orientation). -- xquery ---------------------------------------------------------------------- let $db := doc('./xml/Nirvana.xml') (: our 'database', i.e., the xml file :) let $releases := $db//release (: sequence of releases :) return (: constructing a result XML structure :) element report { attribute number-of-releases { count($releases) }, for $rel in $releases return element release { attribute title { $rel/title }, attribute label { $rel/labels/label/@name }, attribute released { $rel/released || ' (' || $rel/country || ')'} } } -- result ---------------------------------------------------------------------- -------------------------------------------------------------------------------- • Added FLWR expression in XML construction • Used '||' operator, a shortcut for fn:concat(…) -------------------------------------------------------------------------------- Step 3: Group releases for better overview -- xquery ---------------------------------------------------------------------- let $db := doc('./xml/Nirvana.xml') (: our 'database', ie the xml file :) let $releases := $db//release (: sequence of releases :) return (: constructing a result XML structure :) element report { attribute number-of-releases { count($releases) }, for $rel in $releases group by $title := $rel/title return element release-group { attribute title { $title }, attribute group-count { count($rel) }, (: the 'grouped-together' releases :) (: Now process each group :) for $r in $rel return element release { attribute label { $r/labels/label/@name }, attribute released { $r/released || ' (' || $r/country || ')'}, attribute format { $r//format//description } } } } -- result ---------------------------------------------------------------------- -------------------------------------------------------------------------------- • group-by operator -------------------------------------------------------------------------------- Step 4: As a last step, let's add the tracks the release report. -- xquery ---------------------------------------------------------------------- let $db := doc('./xml/Nirvana.xml') (: our 'database', ie the xml file :) let $releases := $db//release (: sequence of releases :) return (: constructing a result XML structure :) element report { attribute number-of-releases { count($releases) }, for $rel in $releases group by $title := $rel/title return element release-group { attribute title { $title }, attribute group-count { count($rel) }, (: the 'grouped-together' releases :) (: Now process each group :) for $r in $rel order by $r//released return element release { attribute label { $r/labels/label/@name }, attribute released { $r/released || ' (' || $r/country || ')'}, attribute format { $r//format//description }, attribute videos { count($r//video) }, element tracks { attribute count { count($r//track) }, for $t at $pos in $r//track return attribute { 't' || $pos } { substring($t//title, 1, 9) || '…' } } } } } -- result ---------------------------------------------------------------------- … -------------------------------------------------------------------------------- • FLWR at $pos construct • dynamic element/attribute name construction • order by construct ################################################################################ ### Website generation Now that we explored the data and have an overview, we want to get productive. The task will be to produce a webpage. We already produced content, namely the report in XML. Now we produce HTML from the data using XQuery. On-the-fly we learn about XQuery functions and code separation. ================================================================================ ### Task: Nirvana release report as webpage. Instead of an XML report, use XQuery to construct a webpage. The page shall display information to be found in the discogs XML data. There is a mockup (a screenshot of the expected result) and a HTML template. The mockup can be found in [./gfx/mockup.png] The HTML template is in [./html/mockup.html] The html directory contains some additional stuff: html ├── css ..................................... Cascading style sheets │   └── bootstrap.css ├── img ..................................... graphics, etc. │   ├── glyphicons-halflings-white.png │   └── glyphicons-halflings.png ├── (nirvana.html ........................... the file we shall create) └── mockup.html ............................. a static mockup of the page we want to create Bootstrap is a 'Sleek, intuitive, and powerful front-end framework for faster and easier web development.' It allows you to style webpages with ease. For some details have a look at: http://getbootstrap.com/ -------------------------------------------------------------------------------- Step 1: To actually produce an HTML file, we have to write a file to disk -- xquery ---------------------------------------------------------------------- (:~ : Constructs basic HTML webpage. : @returns HTML :) declare function local:html() as element(html) { Discogs XML Data

Hello, nothing to see, yet ;-)

}; (: --------- Main entry point (could be another query file ) ------- :) let $html := local:html() return file:write('./html/nirvana.html', $html) -- result ---------------------------------------------------------------------- A file has been created. html ├── nirvana.html ............................ the file we just created └── mockup.html Screenshot [./gfx/html-step1.png] -------------------------------------------------------------------------------- • XQuery function declaration • Comments to functions (XQDoc style, http://xqdoc.org/) • XQuery extensions (such as file operations) • BaseX (and other XQuery processors) come with a rich set of functions that extend the XQuery core language. • For BaseX documentation can be found at - http://docs.basex.org ................... overall documentation - http://docs.basex.org/wiki/XQuery ........ XQuery specific documentation - http://docs.basex.org/wiki/File_Module .. the file module we have just used - http://expath.org/spec/file ............. File module specification -------------------------------------------------------------------------------- Step 2: Feeding content into HTML template -- xquery ---------------------------------------------------------------------- (:~ : Constructs basic HTML webpage. : : @param $releases sequence of discogs release data container : @returns HTML :) declare function local:html( $releases as element(release)* ) as element(html) { Discogs XML Data

Nirvana @ Discogs ( { count($releases) } releases )

}; (: --------- Main entry point (could be another query file ) ------- :) let $db := doc('./xml/Nirvana.xml') (: our 'database', ie the xml file :) let $releases := $db//release (: sequence of releases :) let $html := local:html($releases) return file:write('./html/nirvana.html', $html) -- result ---------------------------------------------------------------------- Screenshot [./gfx/html-step2.png] -------------------------------------------------------------------------------- • typed function parameters, return type • { } XQuery evaluation in static content -------------------------------------------------------------------------------- Step 3: Produce 'correct' (acc. to mockup) headline, separate it to a function -- xquery ---------------------------------------------------------------------- (:~ : Constructs basic HTML webpage. : : @param $releases sequence of discogs release data container : @returns HTML :) declare function local:html( $releases as element(release)* ) as element(html) { Discogs XML Data
{ local:headline($releases), for $rel in $releases group by $title := $rel/title return (: release-group :) local:render-release-group( $title, $rel ) }
}; (:~ : Constructs headline part of release webpage. : [./gfx/html-step3.png] : : @param $releases sequence of discogs release data container : @return a sequence of
nodes constructing the headline :) declare function local:headline( $releases as element(release)* ) as element(div)+ { (

Discover Music (display Discogs XML data)

,
,

Nirvana @ Discogs ( { count($releases) } releases )

) }; (: --------- Main entry point (could be another query file ) ------- :) let $db := doc('./xml/Nirvana.xml') (: our 'database', ie the xml file :) let $releases := $db//release (: sequence of releases :) let $html := local:html($releases) return file:write('./html/nirvana.html', $html) -- result ---------------------------------------------------------------------- Screenshot [./gfx/html-step3.png] -------------------------------------------------------------------------------- • Returning a sequence of
nodes -------------------------------------------------------------------------------- Step 4: Render release-group -- xquery ---------------------------------------------------------------------- (:~ : Constructs basic HTML webpage. : : @param $releases sequence of discogs release data container : @returns HTML :) declare function local:html( $releases as element(release)* ) as element(html) { Discogs XML Data
{ local:headline($releases), for $rel in $releases group by $title := $rel/title return (: release-group :) local:render-release-group( $title, $rel ) }
}; (:~ : Constructs headline part of release webpage. : : @param $releases sequence of discogs release data container : @return a sequence of
nodes constructing the headline :) declare function local:headline( $releases as element(release)* ) as element(div)+ { (

Discover Music (display Discogs XML data)

,
,

Nirvana @ Discogs ( { count($releases) } releases )

) }; (:~ : Constructs release group rendering. : [./gfx/html-step4.png] : : @param $title of release group (grouping condition) : @param $grouped-releases the, by $title, grouped-together releases : @return a sequence of
nodes constructing a single release group :) declare function local:render-release-group( $title as xs:string, $grouped-releases as element(release)* ) as element(div)+ {

{ $title }  ( { count($grouped-releases) } releases )

}; (: --------- Main entry point (could be another query file ) ------- :) let $db := doc('./xml/Nirvana.xml') (: our 'database', ie the xml file :) let $releases := $db//release (: sequence of releases :) let $html := local:html($releases) return file:write('./html/nirvana.html', $html) -- result ---------------------------------------------------------------------- Screenshot [./gfx/html-step4.png] -------------------------------------------------------------------------------- • Nothing really new • Used entity (  ^=  ), look up in ./basex/etc/xhtml-lat1.ent -------------------------------------------------------------------------------- Step 5: Render individual release -- xquery ---------------------------------------------------------------------- (:~ : Constructs basic HTML webpage. : [./gfx/html-step2.png] : : @param $releases sequence of discogs release data container : @returns HTML :) declare function local:html( $releases as element(release)* ) as element(html) { Discogs XML Data
{ local:headline($releases), for $rel in $releases group by $title := $rel/title return (: release-group :) local:render-release-group( $title, $rel ) }
}; (:~ : Constructs headline part of release webpage. : [./gfx/html-step3.png] : : @param $releases sequence of discogs release data container : @return a sequence of
nodes constructing the headline :) declare function local:headline( $releases as element(release)* ) as element(div)+ { (

Discover Music (display Discogs XML data)

,
,

Nirvana @ Discogs ( { count($releases) } releases )

) }; (:~ : Constructs release group rendering. : [./gfx/html-step4.png] : : @param $title of release group (grouping condition) : @param $grouped-releases the, by $title, grouped-together releases : @return a sequence of
nodes constructing a single release group :) declare function local:render-release-group( $title as xs:string, $grouped-releases as element(release)* ) as element(div)+ { (

{ $title }  ( { count($grouped-releases) } releases )

, (: Process release group. :) for $release in $grouped-releases order by $release//released (: contains release date, eg. '2007' :) return local:render-release($release) ) }; (:~ : Renders a single release. : [./gfx/html-step5.png] : : @param $release a single release to be rendered : @return a sequence of
nodes constructing a single release :) declare function local:render-release( $release as element(release) ) as element(div)+ { ( 
Release Headline
, 
Cover area
Track Table
Metadata Table
Video Table
Labels area
) }; (: --------- Main entry point (could be another query file ) ------- :) let $db := doc('./xml/Nirvana.xml') (: our 'database', ie the xml file :) let $releases := $db//release (: sequence of releases :) let $html := local:html($releases) return file:write('./html/nirvana.html', $html) -- result ---------------------------------------------------------------------- Screenshot [./gfx/html-step5.png] -------------------------------------------------------------------------------- • A new function to render individual releases has been added with stubs for the sections of a release: a) Release Headline b) Cover area c) Track Table d) Metadata Table e) Video Table -------------------------------------------------------------------------------- Step 6a: Render sections of individual release - Release Headline -- xquery ---------------------------------------------------------------------- (:~ : Renders a single release. : [./gfx/html-step5.png] : : @param $release a single release to be rendered : @return a sequence of
nodes constructing a single release :) declare function local:render-release( $release as element(release) ) as element(div)+ { ( local:render-release-headline($release) , 
Cover area
Track Table
Metadata Table
Video Table
Labels area
) }; (:~ : Renders release headline. : [./gfx/html-step6a.png] : : @param $release the single release for which the headline should be rendered : @return a
node constructing the release headline :) declare function local:render-release-headline( $release as element(release) ) as element(div) {

{ $release/title/text() }  ( { let $vals := ($release/released, $release/country, $release//label/@name/data()) for $v at $pos in $vals return if ($v) then concat($v, if ($pos != count($vals)) then ',' else ()) else () } )

}; -- result ---------------------------------------------------------------------- Screenshot [./gfx/html-step6a.png] -------------------------------------------------------------------------------- -------------------------------------------------------------------------------- Step 6b: Render sections of individual release - Cover area -- xquery ---------------------------------------------------------------------- (:~ : Renders a single release. : [./gfx/html-step5.png] : : @param $release a single release to be rendered : @return a sequence of
nodes constructing a single release :) declare function local:render-release( $release as element(release) ) as element(div)+ { ( local:render-release-headline($release) ,
{ local:render-cover-area($release) } 
Track Table
Metadata Table
Video Table
Labels area
) }; ... (:~ : Renders cover area. : [./gfx/html-step6b.png] : : @param $release the single release for which the cover area should be rendered : @return a sequence of
nodes constructing the cover area :) declare function local:render-cover-area( $release as element(release) ) as element(div) {
{ for $img in $release/images/image return

}
}; -- result ---------------------------------------------------------------------- Screenshot [./gfx/html-step6b.png] -------------------------------------------------------------------------------- -------------------------------------------------------------------------------- Step 6c: Render sections of individual release - Track table -- xquery ---------------------------------------------------------------------- ... declare function local:render-release( $release as element(release) ) as element(div)+ { ( local:render-release-headline($release) ,
{ local:render-cover-area($release), local:render-track-area($release) } ... (:~ : Renders track area. : [./gfx/html-step6c.png] : : @param $release the single release for which the track table should be rendered : @return a sequence of
nodes constructing the track table :) declare function local:render-track-area( $release as element(release) ) as element(div) {
{ for $track in $release/tracklist/track return }
{ $track/position } { $track/title/data() }
}; -- result ---------------------------------------------------------------------- Screenshot [./gfx/html-step6c.png] -------------------------------------------------------------------------------- -------------------------------------------------------------------------------- Step 6d: Render sections of individual release - Metadata/Video table -- xquery ---------------------------------------------------------------------- ... (:~ : Renders a single release. : [./gfx/html-step5.png] : : @param $release a single release to be rendered : @return a sequence of
nodes constructing a single release :) declare function local:render-release( $release as element(release) ) as element(div)+ { ( local:render-release-headline($release) ,
{ local:render-cover-area($release), local:render-track-area($release), local:render-metadata-video-area($release) } 
Labels area
) }; ... (:~ : Renders metadata and video area. : [./gfx/html-step6d.png] : : @param $release the single release for which the metadata/video table should be rendered : @return a
node constructing the track table :) declare function local:render-metadata-video-area( $release as element(release) ) as element(div) {
Format { local:comma-separated-values($release//formats//description) }
Genre { local:comma-separated-values($release//genres/genre) }
Styles { local:comma-separated-values($release//styles/style) }
Notes { substring(local:comma-separated-values($release//notes), 1, 350) || '…' }
{ for $v in $release/videos//video return }
Video Duration
{ $v/title/data() } { $v/@duration/data() }
}; (:~ : Constructs a comma separated values string from a sequence of input nodes. : : @param $values to be concatenated : @return CSV string :) declare function local:comma-separated-values( $values as item()* ) as xs:string* { let $vals := $values for $v at $pos in $vals return if ($v) then concat($v, if ($pos != count($vals)) then ',' else ()) else () }; ... -- result ---------------------------------------------------------------------- Screenshot [./gfx/html-step6d.png] -------------------------------------------------------------------------------- -------------------------------------------------------------------------------- Step 6e: Render sections of individual release - Labels area -- xquery ---------------------------------------------------------------------- (:~ : Renders a single release. : [./gfx/html-step5.png] : : @param $release a single release to be rendered : @return a sequence of
nodes constructing a single release :) declare function local:render-release( $release as element(release) ) as element(div)+ { ( local:render-release-headline($release) ,
{ local:render-cover-area($release), local:render-track-area($release), local:render-metadata-video-area($release), local:render-labels-area($release) }
) }; ... (:~ : Renders labels area. : [./gfx/html-step6e.png] : : @param $release the single release for which the labels area should be rendered : @return a
node constructing the labels area :) declare function local:render-labels-area( $release as element(release) ) as element(div) {

{ $release/formats/format/@name/data() }

{ $release/formats/format/@text/data() }

{ $release/labels/label/@catno/data() }

{ $release/labels/label/@name/data() }

}; -- result ---------------------------------------------------------------------- Screenshot [./gfx/html-step6e.png] -------------------------------------------------------------------------------- -------------------------------------------------------------------------------- Complete XQuery code for website generation: -- xquery ---------------------------------------------------------------------- (:~ : XQuery code to render discogs release data (XML) as webpage (HTML). : : (c) 2013 DBIS group, U Konstanz, XML Technologies, Winter Term 2013/14 :) (:~ : Constructs basic HTML webpage. : [./gfx/html-step2.png] : : @param $releases sequence of discogs release data container : @returns HTML :) declare function local:html( $releases as element(release)* ) as element(html) { Discogs XML Data
{ local:headline($releases), for $rel in $releases group by $title := $rel/title return (: release-group :) local:render-release-group( $title, $rel ) }
}; (:~ : Constructs headline part of release webpage. : [./gfx/html-step3.png] : : @param $releases sequence of discogs release data container : @return a sequence of
nodes constructing the headline :) declare function local:headline( $releases as element(release)* ) as element(div)+ { (

Discover Music (display Discogs XML data)

,
,

Nirvana @ Discogs ( { count($releases) } releases )

) }; (:~ : Constructs release group rendering. : [./gfx/html-step4.png] : : @param $title of release group (grouping condition) : @param $grouped-releases the, by $title, grouped-together releases : @return a sequence of
nodes constructing a single release group :) declare function local:render-release-group( $title as xs:string, $grouped-releases as element(release)* ) as element(div)+ { (

{ $title }  ( { count($grouped-releases) } releases )

, (: Process release group. :) for $release in $grouped-releases order by $release//released (: contains release date, eg. '2007' :) return local:render-release($release) ) }; (:~ : Renders a single release. : [./gfx/html-step5.png] : : @param $release a single release to be rendered : @return a sequence of
nodes constructing a single release :) declare function local:render-release( $release as element(release) ) as element(div)+ { ( local:render-release-headline($release) ,
{ local:render-cover-area($release), local:render-track-area($release), local:render-metadata-video-area($release), local:render-labels-area($release) }
) }; (:~ : Renders release headline. : [./gfx/html-step6a.png] : : @param $release the single release for which the headline should be rendered : @return a
node constructing the release headline :) declare function local:render-release-headline( $release as element(release) ) as element(div) {

{ $release/title/text() }  ( { local:comma-separated-values(($release/released, $release/country, $release//label/@name/data())) } )

}; (:~ : Renders cover area. : [./gfx/html-step6b.png] : : @param $release the single release for which the cover area should be rendered : @return a
node constructing the cover area :) declare function local:render-cover-area( $release as element(release) ) as element(div) {
{ for $img in $release/images/image return

}
}; (:~ : Renders track area. : [./gfx/html-step6c.png] : : @param $release the single release for which the track table should be rendered : @return a
node constructing the track table :) declare function local:render-track-area( $release as element(release) ) as element(div) {
{ for $track in $release/tracklist/track return }
{ $track/position } { $track/title/data() }
}; (:~ : Renders metadata and video area. : [./gfx/html-step6d.png] : : @param $release the single release for which the metadata/video table should be rendered : @return a
node constructing the track table :) declare function local:render-metadata-video-area( $release as element(release) ) as element(div) {
Format { local:comma-separated-values($release//formats//description) }
Genre { local:comma-separated-values($release//genres/genre) }
Styles { local:comma-separated-values($release//styles/style) }
Notes { substring(local:comma-separated-values($release//notes), 1, 350) || '…' }
{ for $v in $release/videos//video return }
Video Duration
{ $v/title/data() } { $v/@duration/data() }
}; (:~ : Renders labels area. : [./gfx/html-step6e.png] : : @param $release the single release for which the labels area should be rendered : @return a
node constructing the labels area :) declare function local:render-labels-area( $release as element(release) ) as element(div) {

{ $release/formats/format/@name/data() }

{ $release/formats/format/@text/data() }

{ $release/labels/label/@catno/data() }

{ $release/labels/label/@name/data() }

}; (:~ : Constructs a comma separated values string from a sequence of input nodes. : : @param $values to be concatenated : @return CSV string :) declare function local:comma-separated-values( $values as item()* ) as xs:string* { let $vals := $values for $v at $pos in $vals return if ($v) then concat($v, if ($pos != count($vals)) then ',' else ()) else () }; (: --------- Main entry point (could be another query file ) ------- :) let $db := doc('./xml/Nirvana.xml') (: our 'database', ie the xml file :) let $releases := $db//release (: sequence of releases :) let $html := local:html($releases) return file:write('./html/nirvana.html', $html) -- result ---------------------------------------------------------------------- Screenshot [./gfx/html-step6e.png] -------------------------------------------------------------------------------- ================================================================================ ### Final remarks • XQuery as information processor (analyse, report, restructure, …) • XQuery as cross-media output generator Have fun using XML technologies, Alexander Holupirek, Christian Grün ================================================================================