john.snelson.org.uk

RDF for XML Enthusiasts

Sun, 10 Feb 2013 13:06:00 +0000

I’ve been learning RDF and SPARQL for a while now, and recently I had the chance to talk to a number of MarkLogic customers who are using RDF with or alongside MarkLogic. One of the results of that is this presentation, which I gave at XML Prague this weekend.

I need a place to host a photo of me

Thu, 17 May 2012 16:37:17 +0100

I need a place to host a photo of me

On determining itemtype-subtype() for recursive types

Fri, 22 Jul 2011 16:32:08 +0100

There were questions as to whether type checking using a lenient recursive type alias like I proposed in my previous post would terminate. I’ve been doing some research - so if like me you struggle to know the correct academic term to use to search for information, here is a handy guide:

Equirecursive types: The kind of type recursion we’re talking about for XSLT 3.0. More formally, a type system where A is considered the same type as B, given:

<xsl:type-alias name="A" as="map(xs:string, A)"/>
<xsl:type-alias name="B" as="map(xs:string, map(xs:string, B))"/>

In other words, a recursive type is considered equal to it’s unrolling by one level.

Isorecursive types: Recursive types that provide new type annotations. An unrolled isorecursive type is not considered equal to the original type. These are apparently considered simpler to reason about than equirecursive types.

Unification: An algorithm that allows you to check equality of two types (well, directed graphs actually). If implemented correctly, it handles graphs with cycles (recursive types), as well as graphs with variables (parametric types). Incidently the same algorithm an be used for both pattern matching and type inference.

My source material for the algorithm comes from the “Dragon Book”, section 6.5.5. The unification algorithm for ItemType equality would look something like this:

1) Express the types as directed graphs with type aliases expanded (I’ve numbered the nodes so I can refer to them later):

<!-- Slightly different defintion of B this time -->
<xsl:type-alias name="A" as="map(xs:string, A)"/>
<xsl:type-alias name="B" as="map(xs:string, map(xs:NMTOKEN, B))"/>

A:         map[1]-+
            /\    |
           /  ----+
xs:string[2]

B:         map[3]-----+
            /\        |
           /  \       |
xs:string[4]   map[5] |
               /\     |
              /  -----+
  xs:NMTOKEN[6]

2) The basic idea is to start with two nodes from the graphs, M and N, that you want to assert are equal. The algorithm keeps track of sets of nodes that must be equal for this assertion to be true, called equivalence classes. Equivalence classes are represented by a single node picked from that set.

3) find(n) returns the node representing the equivalence class for the node n. Each node starts off in it’s own equivalence class, and these classes get merged successively by the union(m,n) procedure.

4) The algorithm proceeds:

boolean unify(M, N) {
   S = find(M), T = find(N);
   if(S = T) return true;
   if(S and T have the same node label and the same number of children) {
     union(S, T);
     return every child s in S, child t in T satisfies unify(s, t)
   }
   return false;
}

5) The algorithm terminates because the number of equivalence classes is reduced by 1 for each recursion. Finally, we can extend the notion of unification to use equivalence classes based on itemtype-subtype() rules. We only need to make sure that every node in the equivalence class is a subtype of the representative node. By doing so, our unification algorithm verifies a subtype relationship rather than an equality relationship.

A working of the itemtype-subtype() algorithm for the types I named A and B above:

i) Equivalence classes: (map[1]), (xs:string[2]), (map[3]), (xs:string[4]), (map[5]), (xs:NMTOKEN[6])

unify(map[1], map[3])
   - S = map[1], T = map[3]
   - S and T are matching non-leaf nodes
   - union(S, T)
   - recursively check the children

ii) Equivalence classes: (map[1], map[3]), (xs:string[2]), (xs:string[4]), (map[5]), (xs:NMTOKEN[6])

unify(xs:string[2], xs:string[4])
   - S = xs:string[2], T = xs:string[4]
   - S is a subtype of T
   - union(S, T)
   - return true (no children to check)

iii) Equivalence classes: (map[1], map[3]), (xs:string[2], xs:string[4]), (map[5]), (xs:NMTOKEN[6])

unify(map[1], map[5])
   - S = map[1], T = map[5]
   - S and T are matching non-leaf nodes
   - union(S, T)
   - recursively check the children

iv) Equivalence classes: (map[1], map[3], map[5]), (xs:string[2], xs:string[4]), (xs:NMTOKEN[6])

unify(xs:string[2], xs:NMTOKEN[6])
   - S = xs:string[2], T = xs:NMTOKEN[6]
   - S is a subtype of T
   - union(S, T)
   - return true (no children to check)

v) Equivalence classes: (map[1], map[3], map[5]), (xs:string[2], xs:string[4], xs:NMTOKEN[6])

unify(map[1], map[3])
   - S = map[1], T = map[1]
   - S = T
   - return true

And we successfully determine that the recursive ItemType A is a subtype of the recursive ItemType B.

Proposal to add Type Aliases to XSLT 3.0

Thu, 21 Jul 2011 15:27:00 +0100

Type Alias Declaration

<!-- Category: declaration -->
<xsl:type-alias
   name = qname
   as = item-type
   visibility? = "public" | "private" />

[DEFINITION: A type alias declaration adds a binding to the static context, between an expanded-QName and an ItemType, called the aliased type. The aliased type is specified in the mandatory “as” attribute.]

[ERR TBD] A type alias name must be in a namespace, and must not be the same as the name of any type in the in-scope schema components, or any other type alias name.

Type aliases are imported and included from stylesheet modules, and may be exported by a package where they can be given visibilities of “public” and “private”. NOTE: This proposal does not provide a way for type aliases to be overridden.

[DEFINITION: A type alias may be used as a SequenceType by using a type alias reference, which is a SequenceType consisting only of a bare EQName (which is parsed as an ItemType by the AtomicOrUnionType grammar production in XPath 3.0) resolving to an expanded-QName equal to the type alias name.] A type alias reference behaves exactly as if the aliased type had been used in it’s place. A type alias is in scope for every stylesheet module in the stylesheet, and may be used anywhere an ItemType is expected. NOTE: Type aliases do not provide new type annotations with which values are annotated - in this way they do not define new types, and are true to the name “alias”.

NOTE: This does not permit the use of a type alias in a number of places where an atomic type name is valid, including a cast expression and as the TypeName in an element test. [DEFINITION: A type alias A trivially depends on a type alias B if A’s aliased type is a type alias reference to B, or to a type alias C that itself trivially depends on B.] [ERR TBD] It is an error for a type alias to trivially depend on itself.

NOTE: This proposal does not extend to defining parametrized type aliases, but does not preclude adding them in the future. Examples:

<xsl:type-alias name="my:complexNumber"
   as="map(xs:boolean, xs:double)"/>

A type alias for a complex number implemented as a map from booleans to doubles.

<xsl:type-alias name="my:predicate"
   as="function(item()) as xs:boolean"/>

A type alias for a function which can be used as a predicate.

<xsl:type-alias name="my:tree" as="map(xs:string,my:tree)"/>

A recursive type alias for an edge labelled tree structure implemented using maps.

Other Changes

Add a paragraph to the end of Section 22.1.3 of XSLT 3.0:

<new>
If a new type is added to the type system, implementations may denote that type in the SequenceType syntax using a bare EQName in a (non-null) implementation specific namespace - which is parsed as an ItemType by the AtomicOrUnionType grammar production in XPath 3.0.

</new>

Ant Classpath Woes

Wed, 16 Mar 2011 12:06:00 +0000

When building the specification documents for the W3C XQuery and XSLT working groups, I’m constantly finding I have to battle against Java’s over-engineered dependency injection based XSLT APIs.

I could rant about how an XSLT 1.0 processor won’t do when I’ve got XSLT 2.0 stylesheets - indeed generally speaking I choose my libraries carefully because I want to use exactly those ones, not some others that happens to be hanging around.

Anyhow, this isn’t meant to be that rant. This is just to document the solution to my problems. It seems that the shell script that launches ant likes to add it’s dependencies to the classpath, despite my wishes to the contrary. The way to force it to my will is to use the CLASSPATH_OVERRIDE environment variable, thus: CLASSPATH_OVERRIDE=1 CLASSPATH=../../lib/xercesImpl.jar:../../lib/saxon9.jar:../../lib/saxon.jar ant

Sweet joy!

Australian Humour

Thu, 03 Feb 2011 14:24:04 +0000

A real email my sister received from Townsville Freecycle! Sent: Wednesday, 2 February 2011 12:39 PM
To: townsville_freecycle@yahoogroups.com
Subject: [townsville_freecycle] OFFER - Tropical Cyclone YASI (Category 5) - Townsville

No longer wanted, to large to fit anywhere. To be taken away ASAP please. Thanks in advance.

Ice Crystals in Wytham WoodsPosted from: Oxfordshire County...

Tue, 07 Dec 2010 14:51:52 +0000

Ice Crystals in Wytham Woods

Posted from: Oxfordshire County OX2, UK

Interesting use case for the XQuery 1.1 switch expression

Mon, 06 Sep 2010 15:03:00 +0100

XQuery 1.1 introduces a new switch expression, which was intended for matching an expression against a number of different possible values.

http://www.w3.org/TR/xquery-11/#id-switch

However, it can also be used instead of doing a series of nested if expressions, like this: switch(true())
case $a > 0 return “positive”
case $a < 0 return “negative”
default return “zero”

I think that could become a common usage pattern.

Transferring websites to my iPhone with QR codes

Mon, 17 May 2010 15:03:31 +0100

I often want to transfer a website I’ve been reading from my desktop to my iPhone so I can read it elsewhere. The solution I’ve come up with is the following javascript bookmark to generate a QR code for the current website I’m browsing: javascript:var%20url%20=%20”http://chart.apis.google.com/chart?cht=qr&chs=300x300&chl=”%20+%20encodeURIComponent(window.location);window.location%20=%20url;

Saving that to a bookmark on your browser’s toolbar, and you can easily create a QR code for any website. You can then use iPhone apps like ZBar to read the QR code direct from your monitor and open the webpage.

Definitely the best toad-in-the-hole I’ve ever cooked.

Wed, 14 Apr 2010 19:23:21 +0100

Definitely the best toad-in-the-hole I’ve ever cooked.

Grandpa, Emily and Jez

Mon, 15 Mar 2010 19:53:22 +0000

Grandpa, Emily and Jez

Proposal to add Function Annotations to XQuery 1.1 and XPath 2.1

Mon, 01 Mar 2010 13:05:04 +0000

This post looks at a way to unify recent syntactic category extensions to the function declaration syntax in XQuery (1.1, Scripting Extensions, Update Facility etc.), by providing a generic function annotation facility. This is designed to give implementations a way to extend XQuery in similar ways to those the XQuery Working Group have been pursuing, and would also provide a modest level of reflective capability to the language.

In choosing a syntax for function annotations, it was my aim that it should work with function declarations in the prolog, inline functions, and function item SequenceTypes. In addition for symmetry, it might also be beneficial to let the same syntax work with prolog variable declarations - especially considering XQSX’s ”assignable” qualifier.

Finding a syntax that works where needed proved difficult, and I could describe many more syntaxes that won’t work than ones that would. In the end, I have taken inspiration from Java’s annotations which have a syntax that starts with an “@” [1]. Of course, “@” is already used in XQuery, so I have used the “%” character.

Annotations

  FunctionDecl ::= "declare" FunctionAnnotation* "function" QName
    "(" ParamList? ")" ("as" SequenceType)? (EnclosedExpr | "external")

  FunctionAnnotation ::= "updating" | Annotation

  Annotation ::= "%" QName ("(" Literal ("," Literal)* ")")?

Annotations are pairs of QName and an <annotation value> that are statically associated with a function or FunctionTest. The <annotation value> is a non-empty XDM sequence constructed entirely of literal values.

The prefix of the QName in the Annotation is resolved using the statically known namespaces. An unprefixed annotation name is considered to be in the default function namespace.

If the annotation is followed by a parenthesised list of Literals, those literals make up the <annotation value>, otherwise the <annotation value> is a single xs:boolean item with the value “true”.

XQuery 1.1 defines a number of built-in annotations in the “fn” namespace, including “fn:private”, “fn:public”, “fn:nondeterministic”, and ”fn:deterministic”. Users SHOULD avoid using annotations in the “fn” namespace, since further use of this namespace is reserved for future language extensions.

An XQuery user MAY use arbitrary annotations of their own design. An implementation MAY choose to support a number of further annotations whose effect is beyond the scope of this specification.

NOTE: For backwards compatibility with XQuery Update 1.0, the bare keyword “updating” can be used as syntactic sugar for the annotation “%updating”.

Examples:

  declare %private %updating function local:bar()
  {
    put(<foo/>)
  };

A private updating function.

  declare %private %sequential %my:recursion-depth(10) function local:foo($a)
  {
    $a + 1, local:foo(tail($a))
  };

A private sequential function, using a custom annotation to limit recursion depth to 10.

  declare %rest:get("blog/") function local:get-blog($httpRequest)
  {
    <html>
      <head><title>My Blog</title></head>
      <body><h1>My Blog</h1></body>
    </html>
  };

A custom annotation associating the local:get-blog#1 function with a GET http request with a relative URI of “blog/”.

An external function with a custom annotation resolving the implementation to a Java static method.

Inline Function Annotations

  InlineFunction ::= Annotation* "function" "(" ParamList? ")"
    ("as" SequenceType)? EnclosedExpr

An inline function can also be preceded by a list of zero or more Annotations.

Examples:

  %nondeterministic function($a, $b) { 20 }[189]

A non-deterministic InlineFunction.

  %my:inline function($a) { $a + 1 }

An InlineFunction with a custom annotation directing the implementation to inline the function wherever possible.

FunctionTest Annotation Assertions

  FunctionTest ::= AnyFunctionTest | TypedFunctionTest
 
  AnyFunctionTest ::= Annotation* "function" "(" "*" ")"
 
  TypedFunctionTest ::= Annotation* "function" "(" (SequenceType
    ("," SequenceType)*)? ")" "as" SequenceType

Annotations associated with a FunctionTest are called <annotation assertions>. Annotation assertions can be used to further restrict the set of function items matched by the FunctionTest.

XQuery 1.1 defines a number of built-in <annotation assertions> in the “fn” namespace, including:

%fn:nondeterministic - matches function items with a %fn:deterministic or %fn:nondeterministic annotation.
%fn:deterministic - matches function items with a %fn:deterministic annotation. If no %fn:nondeterministic <annotation assertion> is present, this is the default matching behaviour.
%fn:updating - matches function items with a %fn:simple or %fn:updating annotation.
%fn:sequential - matches function items with a %fn:simple or %fn:sequential annotation.
%fn:simple - matches function items with a %fn:simple annotation. If no %fn:updating or %fn:sequential <annotation assertion> is present, this is the default matching behaviour.

Users SHOULD avoid using <annotation assertions> in the “fn” namespace, since further use of this namespace is reserved for future language extensions.

An XQuery user MAY use arbitrary <annotation assertions> of their own design. It is implementation defined how the effect of these assertions is determined. An implementation MAY choose to support a number of further <annotation assertion> whose effect is beyond the scope of this specification.

Examples:

Allow the function item in $a to have the %fn:nondeterministic annotation.

  declare function fn:map(
    $f as %simple function(item()) as item()*,
    $seq as item()*
  ) as item()*
  {
    ...
  }

Assert that the $f argument of fn:map has the %fn:simple annotation.

  declare function local:sort(
    $compare as %my:transitive function(item(), item())as xs:boolean,
    $seq as item()*
  ) as item()*
  {
    ...
  }

Use the custom <annotation assertion> %my:transitive to check that the $compare argument of local:sort has the %my:transitive annotation.

Querying Annotation Names

  fn:annotation-names($f as function(*)) as xs:QName*

Summary: Returns the set of annotation names associated with the function item $f as a sequence of xs:QName items.

Rules: The set of annotation names are returned in an implementation defined but stable order. The set will include the names of built-in annotations “fn:name” and “fn:arity”.

Examples:

  every $name in (xs:QName("fn:name"), xs:QName("fn:arity"),
    xs:QName("fn:simple"), xs:QName("fn:deterministic"))
  satisfies fn:annotation-names(fn:concat#2) = $name

Returns xs:boolean(“true”).

Querying Annotation Values

  fn:annotation($f as function(*), $name as xs:QName)
    as xs:anyAtomicType*

Summary: Returns the <annotation value> for the annotation named $name associated with the function item $f.

Rules: If $f has no annotation with name $name associated with it, the empty sequence is returned.

Examples:

Returns xs:QName(“fn:concat”).

  annotation(fn:concat#2, xs:QName("fn:arity"))

Returns 2.

  declare function fn:map(
    $f as function(item()) as item()*,
    $seq as item()*
  ) as item()*
  {
    if(fn:annotation($f, xs:QName("fn:sequential")))
      then my:sequential-map($f, $seq)
      else my:parallel-map($f, $seq)
  }

Call a parallel algorithm for fn:map() if the function item $f does not have the annotation %fn:sequential.

  declare function fn:ref($seq as item()*) as function() as item()*
  {
    %fn:ref function() { $seq }
  };
 
  declare function fn:deref($f as function() as item()*) as item()*
  {
    if(fn:annotation($f, xs:QName("fn:ref"))) then $f()
    else fn:error(xs:QName("err:XPTY0004"), "$f is not a sequence reference")
  };

Check for the %fn:ref annotation on the function item $f to make sure that it was returned as the result of a call to fn:ref().

Annotations for Global Variables

  VarDecl ::= "declare" Annotation* "variable" "$" VarName TypeDeclaration?
    ((":=" VarValue) | ("external" (":=" VarDefaultValue)?))

It’s harder to know how to query this kind of annotation, or whether querying it is even desirable. However I think it would be nice to allow the same annotation syntax in this case.

Examples:

  declare %assignable variable $bar as xs:decimal := 5;

Declare an assignable global variable in scripting.

  declare %my:volatile variable $time as xs:time external;

A custom annotation labelling the $time variable as volatile.

Adding Recursive Inline Functions to XQuery 1.1 and XPath 2.1

Fri, 05 Feb 2010 20:24:00 +0000

It was recently pointed out on the W3C Bugzilla that XQuery 1.1 can not define recursive inline functions. This post looks at possibilities for solving that problem.

Examining the Options

Lets consider a recursive function, and a pair of mutually recursive functions. Defined using named functions, they look like this:

declare function sum($a)
{
  if(empty($a)) then 0
  else $a[1] + sum(subsequence($a, 2))
};

declare function even($a)
{
  if($a == 0) then true()
  else odd(abs($a) - 1)
};

declare function odd($a)
{
  if($a == 0) then false()
  else even(abs($a) - 1)
};

1. Allow an inline function to specify a name

The first problem you hit here is that the obvious syntax is ambiguous:

function eq($a) { $a }
function eq ($a)

So lets say that we go with using the two tokens “local function” to start the named inline function - then our example functions look like this:

local function sum($a)
{
  if(empty($a)) then 0
  else $a[1] + sum(subsequence($a, 2))
}

local function even($a)
{
  if($a == 0) then true()
  else if(abs($a) - 1 == 0) then false()
  else even(abs($a) - 2)
}

Notice that odd() has been inlined inside even(), which shows the basic problem for mutually recursive functions using this technique. At least one of the functions no longer exists, meaning you can’t idependantly call it, and you can’t choose to factor that code out into a function. Note also that in more complex mutual recursion scenarios it’s not clear that this rewrite is even possible.

2. Provide a “special” variable for referencing the function

function($a)
{
  if(empty($a)) then 0
  else $a[1] + $self(subsequence($a, 2))
}

function($a)
{
  if($a == 0) then true()
  else if(abs($a) - 1 == 0) then false()
  else $self(abs($a) - 2)
}

The mutually recursive functions here suffer from the same problem as with solution 1. In addition, the technique relies on introducing a magic variable, which is both backwards incompatible and ineligant to my way of thinking.

3. Recursive let

letrec $sum := function($a)
{
  if(empty($a)) then 0
  else $a[1] + sum(subsequence($a, 2))
}
return $sum

letrec $even := function($a)
{
  if($a == 0) then true()
  else $odd(abs($a) - 1)
},
$odd := function($a)
{
  if($a == 0) then false()
  else $even(abs($a) - 1)
}
return ($even, $odd)

This is a well-known solution in the functional programming world, and provides a good solution to the mutually recursive function problem. However people have pointed out that it also allows possibly undesirable assignments like this:

letrec $a := 1 + $a
return $a

4. Local functions

local $sum := function($a)
{
  if(empty($a)) then 0
  else $a[1] + $sum(subsequence($a, 2))
}
return $sum

local $even := function($a)
{
  if($a == 0) then true()
  else $odd(abs($a) - 1)
},
$odd := function($a)
{
  if($a == 0) then false()
  else $even(abs($a) - 1)
}
return ($even, $odd)

Local function expressions work in a similar fashion to solution 3, but are restricted to only define function items. All variables declared this way are in scope for all the inline function expressions, and for the return expression. This solution has none of the drawbacks of the previous three.

Proposal I propose that we add a local function expression to XQuery 1.1 and XPath 2.1 by inserting a new section 3.16 (XQuery 1.1) or 3.12 (XPath 2.1) as follows:

3.16 Local Function Expressions

LocalFunctionExpr ::= "local" LocalFunctionBinding ("," LocalFunctionBinding)* "return" ExprSingle LocalFunctionBinding ::= LocalFunctionVar ":=" InlineFunction LocalFunctionVar ::= "$" VarName

Local function expressions bind InlineFunction expressions to variables, allowing the definition of recursive inline functions. Each variable defined by the local function expression is in scope for all InlineFunction expressions and for the return expression.

Evaluation of a local function expression is equivalent to the following:

Every LocalFunctionVar is bound to a hypothetical null value.
The InlineFunction expression from each LocalFunctionBinding is evaluated in turn, and its result is assigned to the LocalFunctionVar.
The return expression is evaluated and it’s result returned.

NOTE: The static type of every LocalFunctionVar can easily be inferred from the InlineFunction it is bound to.

Fast XML Pull Parser 0.3 released

Fri, 21 Mar 2008 07:18:00 +0000

I’ve been doing quite a bit of work on Faxpp recently. My enthusiasm had kind of ground to a halt for a while after I realised the full complexity of implementing entities, but then I decided I just needed to knuckle down and get it finished. The fruit of my labours can now be downloaded from Sourceforge.

I think I’ve got a robust framework for resolving and parsing internal and external entities - and I’ve learnt things about XML that I’m not sure many people in the world know:

Parameter entities (“%entity;”) can appear almost anywhere in an external subset (DTD), but their replacement value is substituted with an extra leading and trailing space if the reference isn’t in a literal value.
Character references in entity values are expanded when the entity declaration is parsed, but general entity references are not resolved until the entity value is substituted for a reference.
An XML 1.0 DTD referenced by an XML 1.1 document will be parsed as though it were XML 1.1.
At least two thirds of the code in an XML parser is there to support functionality that 90% of XML documents never use.

I can also lay claim to actually understanding what notations are, although I don’t think I’ll ever find a use for them.

I’m calling this release a beta, because I know there’s still a bit of work left to be done. Top of the list is implementing default attribute values, then maybe I’ll get to work on shrinking the parser - since the DTD parsing code has made it much larger than I want it to be.

XQilla in the News

Wed, 19 Mar 2008 06:19:00 +0000

Oracle officially announced the XQilla license change today. It feels like this has been a long time coming - I was involved in pushing for the original Pathanproject to be open sourced in 2003 when I worked at Decisionsoft. Later when I worked for Sleepycat, I was involved in pushing for a liberally licensed release of the XQuery implementation and improvements to Pathan which became XQilla some 3 years later.

It’s great to see something that I’ve worked on for the last 7 years start to get the exposure I always thought it deserved. XQuery has huge potential to change the way that people use their data, and it’s close relationship to the web means now might be the right time, and XQilla might be in the right place.

Thanks has to go to Mike Olson who put in the lion’s share of the work needed to make this happen. Hopefully his efforts will make it easier for even more Oracle code to reach it’s potential by being released as open source.

Parsing JSON into XQuery

Fri, 15 Feb 2008 01:21:00 +0000

Doug Crockford stirred things up at XML 2007 with his comments on JSON and XML. He was wrong - mainly because he’s only looking at structured data transfer, rather than anything else that XML is very good at like documents and semi-structured data. However he got me thinking about how easy it would be to process JSON in XQuery, which as Data Direct has shown is very good at manipulating all sorts of data formats.

I headed over to json.org to take a look at the work that had already been done on converting JSON to and from XML. Then I googled, and read around. I even had an email conversation with Dimitre Novatchev of FXSL fame, who’s written a JSON parser entirely in XSLT!

All of the designs I looked at had at least one of the following problems:

They could only convert a subset of JSON - things like map keys that weren’t valid NCNames would cause problems.
They didn’t specify a 1-1 mapping - in other cases map keys were munged to be valid NCNames using a function with no inverse. This would mean that I wouldn’t be able to convert back to JSON from it’s XML representation.
They lost the JSON type information, like whether a value was null or an empty string - which is also a way in which the mapping is not 1-1.

So given nothing fitted I chose to come up with my own mapping from JSON to XML - which is, after all, the JSON way. So here it is, in all it’s simplicity:

JSON	type(JSON)	toXML(JSON)
JSON	N/A	<json type=”type(JSON)”>toXML(JSON)</json>
{ “key1”: value1, “key2”: value2 }	object	<pair name=”key1” type=”type(value1)”>toXML(value1)</pair> <pair name=”key2” type=”type(value2)”>toXML(value2)</pair>
[ value1, value2 ]	array	<item type=”type(value1)”>toXML(value1)</item> <item type=”type(value2)”>toXML(value2)</item>
“value”	string	value
number	number	number
true / false	boolean	true / false
null	null	empty

The table defines two abstract functions “type” and “toXML”, which are recursively defined on the structure of the input JSON. The extension functions to parse and serialize JSON are called xqilla:parse-json() and xqilla:serialize-json(), and will available in the next release of XQilla.

xqilla:parse-json($xml as xs:string?) as element()?
xqilla:serialize-json($json-xml as element()?) as xs:string?

The translation produces a simple generic XML document - as an example, here’s a query to parse a sample of JSON (swiped from wikipedia):

xqilla:parse-json('{
     "firstName": "John",
     "lastName": "Smith",
     "address": {
         "streetAddress": "21 2nd Street",
         "city": "New York",
         "state": "NY",
         "postalCode": 10021
     },
     "phoneNumbers": [
         "212 732-1234",
         "646 123-4567"
     ]
 }')

And here’s its translation, the result of the query:

<json type='object'>
  <pair name='firstName' type='string'>John</pair>
  <pair name='lastName' type='string'>Smith</pair>
  <pair name='address' type='object'>
    <pair name='streetAddress' type='string'>21 2nd Street</pair>
    <pair name='city' type='string'>New York</pair>
    <pair name='state' type='string'>NY</pair>
    <pair name='postalCode' type='number'>10021</pair>
  </pair>
  <pair name='phoneNumbers' type='array'>
    <item type='string'>212 732-1234</item>
    <item type='string'>646 123-4567</item>
  </pair>
</json>

One of the nice things about the XML format for JSON is that it’s easy to navigate. If I want to get the city from the JSON object above, the XQuery for it would be:

xqilla:parse-json("...")/pair[@name="address"]/pair[@name="city"]

Or if I want to get both the phone numbers I could use:

xqilla:parse-json("...")/pair[@name="phoneNumbers"]/item

Now I probably have to do a follow up post about all the cool things you can do in XQuery when you can parse JSON…

Latency vs Throughput

Fri, 25 Jan 2008 18:42:00 +0000

I was helping a Berkeley DB XML user recently who complained that his query took as long as ~55s to run. It turned out he was trying to run 40 concurrent queries, so I tried out the query on his data set using our concurrency testing framework. I was confused when it reported to me that I was getting ~2 ops/s, which is a completely different ballpark.

It took me a while, but eventually I figured it out: When the user was saying that his query took ~55s what he meant was that a thread’s view of how long a query takes can be as much as 55s. When I talked about getting ~2 ops/s, I was looking at the system as a whole. The problem was that he was talking about latency, and I was talking about throughput.

Typically the way that a server gets smaller latency is to implement a thread pool and work queue. Since DB XML is an embedded library and does not spawn it’s own threads, it is down to the application using DB XML to manage it’s threads well.

So I created a Java test program that used a thread pool to manage the queries being run through DB XML (in Java it’s really easy). I created a test function that could run the query with arguments of how many threads to create, how many queries to run, and how many queries to start per second.

I ran a number of tests - in all of the test runs I kept the number of queries run as constant, so that exactly the same amount of useful work was done in all cases. I measured “workload” time, which is the time from when the query is requested until it is completed - this is a measure of latency. I also measured the total time taken for all the queries, and the number of operation per second executed - this is a measure of throughput. Here are the actual figures I got:

Queries run	Queries per second	Threads	Latency (avg query time in seconds)	Throughput (queries per second)
80	no delay	40	32.15	1.70
80	no delay	10	23.34	1.96
80	no delay	4	22.00	1.89
80	10	4	18.51	1.88
80	5	4	13.87	1.93
80	2	4	0.90	1.97

The first thing I noticed is that as the number of threads spawned decreased, latency decreases and throughput increases - this peaked around 4-5 threads on my system. Throughput increased because each thread has a processing overhead associated with it, so less threads means less overhead. Average latency decreased because instead of 5 queries each taking 50s (as an example), the first query takes 10s, and the last takes 50s:

Average time for 5 queries in parallel: (50s + 50s + 50s + 50s + 50s) / 5 = 50s
Average time for 5 queries in series: (10s + 20s + 30s + 40s + 50s) / 5 = 30s

Surprise number one - adding more threads can actually increase your latency!

Next I tried to add a little bit of realism to the simulation - it is very rare that 80 queries need to be executed at exactly the same time, so I added a little bit of a delay between starting each query. As the queries per second decreased throughput was more or less constant, but latency decreased dramatically. This is simple to explain - the queries spent less time in the work queue, when they aren’t actually being executed:

Average time for 5 queries executed at the same time: (10s + 20s + 30s + 40s + 50s) / 5 = 30s
Average time for 5 queries executed with a 10s delay between each: (10s + 10s + 10s + 10s + 10s) / 5 = 10s

So surprise number two - a more realistic workload can result in smaller latency!

Why DB XML Doesn't Do Path Indexes

Fri, 18 Jan 2008 00:55:00 +0000

Every so often we have a Berkeley DB XML user ask us if we implement path indexes. Or maybe they assume that we do implement them and are confused about why their index isn’t working. It turns out that a lot of these users have come to us after using eXist, which for a long time has only been able to specify path indexes.

A path index is specified by describing the path to the nodes that you want to index - often with a subset of XPath. So you might want to index all “firstname” elements, but only if they are children of an “author” element - which you might write like this:

//author/firstname

The good thing about path indexes is that they can be very small targeted indexes, which only record exactly the information that you want.

The trouble with path indexes, I’ve always explained, is that it’s really hard to work out when you can use one in a query. You miss out on lots of potential optimisation opportunities because query analysis can’t prove that an index is applicable. It’s not surprising really - path indexes are a subset of materialised views which is known to be a very hard problem, in the general case.

So it’s not surprising to see that recent work on eXist has added a new type of index which works in exactly the same way as DB XML’s do. The author of the blog post even describes the problems that I have been telling our users about, and recommends that path indexes be used sparingly if at all.

It’s good to see other people coming to the same conclusion that I came to. DB XML may implement path indexes some day, but they won’t be anywhere near as useful as some users think they should be.