Showing posts with label XProc. Show all posts
Showing posts with label XProc. Show all posts

Saturday, January 22, 2011

XML Processing in Healthcare Applications

Meaningful Use certification requires the ability to create patient summaries in either C32 or CCR format. One of the most frequently asked questions on the HL7 Structured Document mailing list is related to the processing of the CDA XML schema with data binding tools such as JAXB or Castor. Initially, people are not able to generate Java classes with JAXB at all. After some changes to the schema, JAXB finally works and creates hundreds of classes which are not very easy to work with and maintain. Then someone suggests using the Model-Driven Health Tools (MDHT) CDA tools which are Java-based. You face additional headache if you're not developing on the Java platform.

In a paper presented at the Balisage 2009 conference, a team of engineers who implemented the "Laika" C32 compliance testing tool described the issues with the CDA and C32 XML structure:

  • Repeated use of overly abstract data structures: The HL7 CDA defines a number of very generic objects that are used to represent information in a given document. Differing information, such as medications and conditions, are represented using the same XML elements with very subtle changes in their nesting and attributes. This makes a CDA document difficult to process.

  • Underspecified implementation, including lack of a normative schema: While there is an XML schema for the HL7 CDA, a final schema does not exist for the HITSP C32 or other CDA-based documents due to their use of attributes for selecting templates. Thus, defining schemas for these documents is impossible. As a result, CDA-based constructs such as HITSP C32 cannot be automatically validated by XML parsers; standard object mapping tools, such as XML Beans or JAXB, cannot be used.

  • Ambiguous data types: Data can be represented in multiple ways in a CDA document. Consumers of CDA documents must, therefore, write software that handles any of the numerous permutations of these data types. This leads to bloated software, or more likely, software that does not implement the full specification and experiences interoperability problems when it receives data in an unexpected format.

  • Steep and long learning curve: Mastery of the CDA and its many specifications and constructs takes an experienced software engineer many months to achieve. Once learned, it is very cumbersome to employ in robust software applications and services. These difficulties drive up the cost and time to develop and maintain health care software, thus reducing the pace of innovation.

In a previous post entitled "The Future of Healthcare Data Exchange Standards", I suggested some ideas on how to develop standard XML schemas that support the software development process as opposed to hindering it. Since we're not there yet, in this post I will suggest some ideas on dealing with the complexity of the CDA schema and C32 generation process.

The key is to leverage the power of XML related technologies such as XPath2, XSLT2, XQuery, XProc, ISO Schematron, and even XML Schema 1.1 (for assertions or business rules constraints) to simplify the task. First, generate a simple and perhaps flat XML representation (let's call it simpleC32) of the patient summary from your domain objects or database (through a data transfer object or DTO for example). That simpleC32 contains all the content that is needed to populate the C32 templates and generate a valid C32 document. You can create your own XML schema for your simpleC32 and use it for validation and data binding.

Once you have a valid simpleC32 document, you can use XSLT2 to transform the patient summary from your simpleC32 representation into a C32 document that can be validated against the NIST Meaningful Use C32 Validator. This is roughly the idea behind the GreenCDA project. Use that as an inspiration on how to create a simple representation of the C32. You can even use the GreenCDA XML schema as your simpleC32. But don't hesitate to create your own simpleC32 if the GreenCDA does not work for you, because the target is still the C32, and the idea here is to have an intermediary representation (an Adapter) to make your life easier. It is also an approach that allows you to isolate your domain model and prevent the complexity of the C32 data model from leaking into your domain layer (see my previous post on the concept of Anti-Corruption Layer in Domain Driven Design).

Why is this approach not used more often? Some developers who code with imperative programming languages (such as Java, C#, or JavaScript) are not comfortable with declarative programming using languages like XSLT2 and XQuery. I've recently seen a Java developer use JAXB to create hundreds of classes and thousands of hard to maintain lines of code for a simple transformation from the CDA to a different target XML schema.

The basic difference between declarative (and functional) programming languages and imperative languages is that the former specify the "what" (the intent) as opposed to the "how" (the algorithm). However, declarative programming with XSLT2 and XQuery can be mastered through training and practice: see my previous posts entitled: "In Defense of XSLT", "Why XProc Rocks", and Putting XQuery to Work in Healthcare".

While Java and C# are general purpose languages, processing languages like XSLT2, XQuery, and XProc are actually based on the XQuery 1.0 and XPath 2.0 Data Model (XDM) and specifically designed for the purpose of manipulating XML documents. This is particularly helpful when dealing with a complex and deep structure such as the HL7 CDA and other HL7 V3 messages. These XML-centric processing languages use XPath2 to navigate the XML tree. In general, consider using them in the following cases:

  • Applications that require dealing with a complex industry data exchange XML schema which is not easy to process with your databinding and other development tools. In that case, create an intermediary simpe XML representation and map it to the industry data exchange XML schema using XSLT2 or XQuery (XQuery is not just for querying native XML databases, it is also a powerful language for processing XML documents).

  • Applications that require translation from an XML schema to another target XML schema (for example a mapping from the HL7 CCD to the ASTM CCR or from the C32 to XHTML).

  • Applications that require translation from an XML representation to a non-XML representation and round-trip (for example HL7 v2.x to HL7 V3, C32 XML to JSON, or C32 to a non-XML serialization of RDF).

  • Consider using XProc if you need to chain multiple XML processing steps such as: query a data source with XQuery, expand XIncludes, validate against XML schema, validate against a schematron schema, transform with XSLT2, generate a PDF document with XSL FO, and so on.

The Universal Exchange Language proposed by the PCAST Report could be an opportunity to address the issues listed above.

Tuesday, May 19, 2009

Why XProc Rocks

These are exciting times to be a technologist in the news publishing business. The industry is going through fundamental changes driven by the current economic downturn. Innovation has become an imperative. Examples of recent innovations include news content APIs (newspaper as a Platform), specialized desktop news readers, the ability to publish to an increasing number of mobile devices (Kindle, iPhone, etc.), and migrating news content to the Semantic Web and Linked Open Data (LOD) cloud.

The result is that news publishing workflows are getting more complex. The following is an example of what can be expected from such a workflow:

  1. Retrieve content from a native XML database such as Exist using XQuery and from a MySQL database and combine the result as an XML document.
  2. Expand XIncludes and other content references in the XML.
  3. Apply various processing to the XML depending on the content type.
  4. Make a REST call to data.gov and recovery.gov to retrieve some government published data in XML for a graphical mashup visualization of the data.
  5. Transform the XML into XHTML+RDFa for consumption by Yahoo's SearchMonkey and the recently unveiled Google Rich Snippets.
  6. Transform the XML into RDF/XML and validate the result using the Jena command line ARP RDF parser. The RDF/XML output will provide an RDF dump for RDF crawlers and will be searched via a SPARQL endpoint.
  7. Transform the XML into an Amazon Kindle-friendly format such as Text.
  8. Publish the content as a PDF document with a print layout (e.g. header, footer, multi-column, pagination, etc.).
  9. Transform the XML into a NewsML document and validate the result against the NewsML XML Schema and an ISO Schematron schema.
  10. Generate an Atom feed containing NewsML elements and validate the result using NVDL (Namespace-based Validation Dispatching Language). NVDL is required here because the Atom feed will contain nodes that will be validated against the Atom RelaxNG schema as well as nodes that will be validated against the NewsML XML schema.

As you can see, this publishing workflow is XML document-centric. If you are a Java developer, your first instinct might be to automate all those processing steps with an Ant build file. There is now a better alternative and it is called XProc (an XML Pipeline Language) currently a W3C candidate recommendation. Here are some reasons why XProc is a superior alternative when dealing with an XML document-centric publishing workflow:

  • With XProc, XML documents are first-class citizens. Java objects are first-class citizens in Ant. In fact, using Ant to automate a complex XML document-centric publishing workflow can quickly lead to spaghetti code.
  • XProc allows you to add, delete, rename, replace, filter, split, wrap, unwrap, compare, insert, and conditionally process nodes in XML documents by addressing these nodes using XPath.
  • XProc comes with built-in steps for XML processing such as p:xquery, p:xinclude, p:xslt, p:validate-with-xml-schema, p:validate-with-relax-ng, p:validate-with-schematron, p:xsl-formatter, and p:http-request for making REST calls.
  • XProc is declarative, and programming language and platform neutral. For example, work is underway for an XQuery implementation for the Exist database. Note that step 1. above can be implemented with the SQL extension function provided by Exist for retrieving data from relational databases using JDBC.
  • XProc is extensible, so you can add custom steps. For example the XML Calabash XProc implementation provides extension steps such as cx:nvdl and cx:unzip.
  • XProc provides exception handlers which are essential for any complex workflow.
  • XProc pipelines are amenable to streaming.
  • XProc can simplify the design, documentation, and maintenance of very complex publishing workflows. Since XProc is in XML format, one can envision a visual designer for building XProc pipelines or a tool (such as Graphviz) to visualize XProc documents in the form of processing diagrams.

There are other pipeline technologies that can be useful as well, particularly if you're building mashup applications. Yahoo Pipes is a good choice for combining feeds from various sources, manipulating them as needed, and outputting RSS, JSON, KML, and other formats. And if you are into Semantic Web meshups, DERI Pipes provides support for RDF, OWL, and SPARQL queries. DERI Pipes works as a command line tool and can be hooked into an XProc pipeline with the XProc built-in p:exec step.

Friday, February 29, 2008

Declarative Programming Recipes

If you've been using an imperative programming language such as Java, C#, or JavaScript, then you should consider the benefits of declarative programming when designing your next application. In this blog, I will explore the benefits of the following technologies:

  • XML Schema
  • ISO Schematron
  • XForms/XBL (XML Binding Language)
  • XSLT 2.0
  • XQuery
  • Atom Syndication
  • AtomPub

The basic difference between declarative programming languages and imperative programming languages such as C# and Java is that the former specify the “what” (the intent) as opposed to the “how” (the algorithm). The following are some reasons to consider this new paradigm:

  • Declarative programming languages are accessible to many non-programmers
  • It’s possible to create a solution that is completely declarative (no Java, C#, JavaScript, or AJAX code)
  • It facilitates the Model-Driven Architecture (MDA) software design approach
  • When the data or content is managed in XML, a declarative programming model is a superior and simpler alternative to a conventional approach based on Java Server Faces (JSF), Spring, and Hibernate. For example, you get a more robust and simpler data and business rules validation mechanism and you can forgo ORM mapping altogether. However, declarative and imperative programming can co-exist in the same solution.

The following are some recipes for taking advantage of this new paradigm:


  • Define your data structures and types in the XML Schema.
  • Use XML Schema's xsd:appinfo element to:

    • Capture metadata such such as XML to RDBMS mapping information
    • Capture business rules using inline ISO Schematron rules
    • Control the labels, alerts, and appearances of UI components such as XForms controls.

  • Generate your UI by transforming the XML Schema into XForms with XSLT.
  • Alternatively, store your XML schema and other XML-formatted metadata artifacts in a native XML database and use XQuery to extract and manipulate the metadata during development and at run-time.
  • Use XBL for enabling custom UI controls for your XForms. As an example, you can integrate a rich text editor using XBL and Dojo.
  • Use ISO Schematron for business rules validation either post form-submission or by using XSLT to inject schematron rules into the XForms directly with the xforms:bind element.
  • Use the RESTful or AtomPub API to a native XQuery-enabled XML database (such as Exist) to CRUD (create, read, update, delete) the data.
  • If you're using SOAP-based web services, use XForms to send SOAP requests and display SOAP responses as well.
  • Leverage Atom syndication for pushing updates to data consumers. In general, when creating your application, think "Syndication Bus".

Many software architects and developers are not yet aware of the power of this new paradigm. However, I believe that rather than using only familiar development frameworks, it’s important to always evaluate emerging alternatives and select the best approach even it involves a learning curve.