Adventures in Computing

The New Economics of Healthcare and What It Means for Health IT

2011-12-11T19:44:00.020-05:00

Healthcare expenditures represented 17.6% of US GDP in 2009 and continue to rise. In a recently released report titled "Therapy or Surgery? A Prescription for Canada's Health Systems", Don Drummond, former chief economist at TD Bank warned that "assuming total program spending and revenues grow at the nominal GDP growth rate of 4 percent, healthcare would comprise 80 percent of the Ontario budget by 2030, up from 46 percent today."

To address quality and costs challenges, the concept of a value-based healthcare system is gaining significant ground. Articles on the subjects have recently appeared in prestigious publications such as the New England Journal of Medicine (NEJM) and Health Affairs. There is no clear consensus among industry experts that initiatives such as the Accountable Care Organization (ACO) and the Patient Centered Medical Home (PCMH) will lead to dramatic improvements in quality and costs reductions. However, everyone seems to agree that the ultimate goal of a healthcare system should be to maximize positive patient health outcomes per dollar spent.

Measuring Healthcare Value

Michael Porter is arguably the leading thinker on the subject of healthcare value. In an article titled "What Is Value in Health Care?" that appeared in the New England Journal of Medicine, he wrote:

"Value should always be defined around the customer, and in a well-functioning health care system, the creation of value for patients should determine the rewards for all other actors in the system."

Current process measures and the regulatory requirements to report them are necessary but not sufficient. It seems like the healthcare industry has escaped the business process reengineering and quality improvement movements that have permeated many industries in the nineties. Fortunately, thanks to the hard work of visionary leaders like Donald Berwick, former Administrator of the US Centers for Medicare & Medicaid Services (CMS) and co-founder of the Institute for Healthcare Improvement (IHI), quality measures reporting is now an essential component of incentive programs such as Meaningful Use and the ACO model. Healthcare transformation during the next few years will be focused on the migration from a paradigm based on the volume of services delivered to one that is based on measuring value (patient-centered outcomes) as well as the total costs incurred throughout the care cycle to achieve those outcomes.

Patient-centered outcome measures include essential metrics such as mortality, functional status, time to recovery, severity of side effects, and remission (e.g. depression remission at six and twelve months). These measures should take into account the values, goals, and wishes of the patient. Therefore patient-centered outcome should also include the patient's own evaluation of the care received. One important implication of this shift from fee-for-service to value is the growing importance of wellness, prevention, early screening, and disease and population management. In short, the elimination of waste and the optimization of healthcare delivery through the standardization of care pathways and treatment protocols based on the latest scientific evidence will become a top priority.

Measuring Healthcare Costs

The total costs incurred throughout the care cycle represent the assets used to achieve those outcomes such as: employees, consumables, facilities, medical devices, equipment, energy, and computing resources such as software and hardware. Measuring these costs would appear intuitive and straightforward at first. However, in the US healthcare system, Medicare reimbursement is based not on the actual usage of resources, but on so called relative value units (RVUs). Service charges are calculated based on formula that includes three RVUs: one for physician work, one for practice expense, and one for malpractice expense. Similar reimbursement schemes are used in other developed countries as well.

In an article titled "The Big Idea: How to Solve the Cost Crisis in Health Care", Michael Porter and Robert Kaplan proposed time-driven activity-based costing (TDABC) as a more accurate methodology for measuring costs in healthcare. Bundled Payment (using episodes of care as a basis for payment and value measurement) is emerging as a solution to contain healthcare costs.

Implications for Healthcare Information Technology

The trend toward bundle payments and other cost containment schemes will put an increasing pressure on providers to maximize the value of the assets involved in the care cycle. This will be achieved by optimizing the decision making process with software that can analyze large data sets and make specific and accurate recommendations faster than humanely possible.

In the clinical domain in particular, the following are examples of health IT solutions that will make a big difference in maximizing value for patients:

Automated execution of Clinical Practice Guidelines (CPGs), Care Pathways (CPs), and treatment protocols using technologies such as business rules, predictive analytics, and Business Process Management (BPM).
Creation of disease registries as well as secondary use of EHR data to track patient outcomes, compliance with CPGs, Comparative Effectiveness Research (CER), and Patient-Centered Outcome Research (PCOR). Advanced analytics will play an important role in providing important insights into clinical data on the effectiveness of various treatment options based on the clinical profile of a specific patient or subpopulation of patients. Personalized Medicine leveraging advances in genomics will play an important role here as well. This will require computing power to handle the large data sets involved in making clinical decisions based on genomic data.
Applications of Knowledge Representation, Reasoning, Natural Language Processing (NLP), Speech Recognition, Information Retrieval, and Machine Learning to enable next generation Clinical Question Answering (CQA).
Clinical Knowledge Management (CKM) to support a learning health system. The Institute of Medicine (IOM) released a report earlier this year titled "Digital Infrastructure for the Learning Health System: The Foundation for Continuous Improvement in Health and Health Care". The report describes the learning health system as:

"delivery of best practice guidance at the point of choice, continuous learning and feedback in both health and health care, and seamless, ongoing communication among participants, all facilitated through the application of IT."
Social Health Enterprise tools that allow clinicians to communicate and collaborate beyond email.
Leveraging mobile devices and tablets to provide information and cognitive support to clinicians, patients, and care givers while enforcing strict security.
Healthcare interoperability standards. As Doug Fridsma, Director of the Office of Standards and Interoperability puts it in a recent blog post: "standards are not optional". Standardization at the data, security, and transport level is necessary. However, care should be taken to ensure that these standards can be widely implemented by health IT vendors. Adopting healthcare profiles of cross-industry standards and creating Open Source reference implementations using the tools and techniques developers are familiar with (as was done by the ONC-sponsored Direct Project) can help meet that objective.

On the other hand, over-standardization in areas that are going through a rapid rate of technological innovation could have a negative impact.
Applications of human factors research to enable the effective use of technology in clinical settings. Examples include: implementation of usability guidelines to reduce alert fatigue in clinical decision support (CDS) and the use of speech recognition, checklists, simulation, Visual Analytics, and disease-specific documentation templates or Smart Forms.

There are many lessons to be learned from other mission-critical industries that have adopted automation. Following several incidents and accidents related to the introduction of the "Glass Cockpit" about 25 years ago, human factors training known as Cockpit Resource Management (CRM) is now standard practice in the aviation industry.
Lastly, Cloud Computing and Service-Oriented Architecture (SOA) will allow health enterprises to reduce costs and share computing resources while focusing on their core competency: medical care.

Thoughts on the Query Health Initiative

2011-11-12T21:55:00.019-05:00

I have been following the Query Health Initiative of the ONC Standards and Interoperability Framework with great interest. The following are key goals of the Query Health Initiative:

Identify standards and services for distributed population health queries to EHRs, HIEs, and other clinical data sources such as registries.
Define a framework to allow partners to create their own distributed query networks (with or without intermediaries called "Network Data Partners"). I believe that the solution should not mandate a specific implementation in order to foster innovation in the field.
Support a number of use cases such as quality measures reporting, public health surveillance, comparative effectiveness research (CER), and patient centered outcome research (PCOR).
Support queries over a common and extensible Clinical Information Model (CIM).
Support security, audit trails, privacy, patient consent directives, and other policy and legal requirements. Techniques such as the de-identification of data will be essential to maintaining privacy.
Create a solution that can be implemented with a financially sustainable model. I think there are many lessons to be learned here from failed or struggling Health Information Exchange (HIE) initiatives.

The Query Health Initiative supports a distributed model as a opposed to a centralized model. This allows data to be kept in the originating systems and securely queried and aggregated.

Previous initiatives to create such a distributed query health network include:

i2b2 (Informatics for Integrating Biology and the Bedside) and SHRINE (Shared Health Research Information Network) - a scalable informatics framework that will enable clinical researchers to use existing clinical data as well as genomic data for research and discovery.
hQuery - an open source project by MITRE which leverages the ability of certified EHRs to produce C32 or CCR documents. hQuery is based on a MongoDB document database and uses JavaScript Map and Reduce functions.
PopMedNet – a multi-purpose distributed networks for secondary use of EHR, administrative, claims, and registry data.

In addition, large health enterprises are investing considerable efforts and resources in building clinical data warehouses and analytics capabilities for their own internal needs. Examples are the Enterprise Data Trust at Mayo Clinic and the STRIDE (Stanford Translational Research Integrated Database Environment) project at Stanford University. These existing systems may have to eventually participate in distributed population health query networks. Most of these systems are based on SQL databases which have reached a high level of maturity in terms of scalability, performance, and the availability of data processing and analytics techniques and tools.

However, NoSQL and alternatives based on Big Data Analytics (e.g. Hadoop and Hive) are currently making significant inroads into the enterprise. Emerging NoSQL databases include key-value stores, document databases, graph databases, and triple stores such as those based on the SPARQL query language for RDF. In some cases, these NoSQL databases provide superior scalability when compared to SQL databases. They are increasingly popular with developers because they simplify the application development process by eliminating the need for Object Relational Mapping (ORM). For example, in document-oriented databases such as MongoDB (which is used in hQuery), objects are persisted as JSON documents.

So, I believe that the technical choices that are made for the Query Health Initiative should be grounded in that current reality of enterprise data management.

First, I think that queries should be formulated in a declarative as a opposed to a procedural manner. This rules out an approach based on JavaScript.
Second, I believe that queries should be formulated in an established query language. By established query language, I mean a standard like SQL, SPARQL, or XQuery that was design specifically for the purpose of querying data stores. This rules out standards such as the HL7 Health Quality Measures Format (HQMF) or any implementation of the HL7 CDA. In fact, quality measures reporting is just one of many use cases in Query Health. In my opinion, in a value-based healthcare system, patient-centered outcome measurement is even more important than quality measures which are essentially process measures and do not necessarily correlate with improved patient outcomes.

By the way, I believe this same principle should extend to other ONC Standards and Interoperability efforts such as the Data Segmentation Initiative which is trying to define an interoperable approach to implementing privacy policies, consent directives, and authorizations. The Data Segmentation Initiative should embrace the approach taken by the OASIS Cross-Enterprise Security and Privacy Authorization (XSPA) which consists in defining healthcare profiles for well established and recognized standards such as SAML, XACML, and WS-Trust. This contrasts with an approach that would consist in creating a CDA implementation for patient consent directives. This discussion on patient consent directives is relevant to the Query Health Initiative.

i2b2 which is one of the projects considered by the Query Health Initiative uses SQL. i2b2 is a well engineered, robust, and proven architecture. The i2b2 data model is based on the "star schema" which has a central "fact" table where each row represents an observation about a patient. The current implementation of the clinical research chart (CRC) in i2b2 is based on the Oracle and Microsoft SQL Server databases.

The maturity of SQL-based tools could be a deciding factor.
Third, the adopted solution should leave the door open to innovation, by giving participants the choice of embracing alternative and emerging solutions such as SPARQL or UnSQL, a newly proposed query language for NoSQL document databases. Erik Meijer and Gavin Bierman from Microsoft Research wrote a paper titled "A co-Relational Model of Data for Large Shared Data Banks" where they propose a new common query language for both SQL and noSQL databases called coSQL.

In the era of Clinical Question Answering (CQA), Natural Language Processing (NLP) and ontologies will play a critical role in clinical data repositories. SPARQL-based queries when combined with the use of ontologies could offer significant advantages over traditional SQL-based systems (see my previous post titled "Why do we Need Ontologies in Healthcare Applications"). Furthermore, standard vocabularies (such as SNOMED CT) and value sets are essential components of clinical data repositories. These terminologies are often derived from ontologies, so a solution that integrate well with ontologies will be important. I believe that the CTS2 specification satisfies all the vocabulary and value set requirements for Query Health. CTS2 is also currently being implemented by commercial vocabulary tool vendors and various open source projects.

The W3C R2RML (RDB to RDF mapping language) specification allows existing applications to provide an RDF view other relational databases. The SPARQL 1.1 query language supports key Query Health requirements such as aggregates, grouping, and subqueries, while the SPARQL 1.1 Federation Extensions specification supports federated queries.

The Translational Medicine Ontology is designed as a unifying ontology for the integration of EHR, genomic, treatment, drug, and other types of clinical data. This allows the creation of knowledge bases that can be queried with SPARQL to answer important questions related to clinical research as well as patient care. If you are interested in this topic, I highly recommend these two papers:

Why Do We Need Ontologies in Healthcare Applications

2011-08-27T16:46:00.129-04:00

There is an ongoing thread in the HL7 mailing list about "what can OWL do?" in the wake of Grahame Grieve's recent post titled: "HL7 needs a fresh look because V3 has failed".

This post is my answer to "what can OWL do?".

Ontology vs. Information Model

Ontologies are our conceptualization (understanding) of the world while information models (of data structures) describe and constrain how the data is stored and transmitted in messages. Thomas Gruber popularized the notion of ontology in the nineties when he wrote in a paper titled "A Translation Approach to Portable Ontology Specifications":

A conceptualization is an abstract, simplified view of the world that we wish to represent for some purpose. Every knowledge base, knowledge-based system, or knowledge-level agent is committed to some conceptualization, explicitly or implicitly.

An ontology is an explicit specification of a conceptualization. The term is borrowed from philosophy, where an ontology is a systematic account of Existence. For knowledge-based systems, what "exists" is exactly that which can be represented.

When people ask me to explain how ontologies are relevant to healthcare, I often use this quote from a report titled "Semantic Interoperability Deployment and Research Roadmap" by Alan Rector, an authority in the field of biomedical ontologies:

Ontologies are about the things being represented – patients, their diseases. They are about what is always true, whether or not it is known to the clinician. For example, all patients have a body temperature (possibly ambient if they are dead); however, the body temperature may not be known or recorded. It makes no sense to talk about a patient with a "missing" body temperature.

Data structures are about the artefacts in which information is recorded. Not every data structure about a patient need include a field for body temperature, and even if it does, that field may be missing for any given patient. It makes perfect sense to speak about a patient record with missing data for body temperature.

Hence, at the practical level, ontologies can help us verify the soundness of statements in messages based on our conceptualization of the world. Information models in healthcare often take the form of an XML schema, a Schematron schema, or a relational database schema. One distinguishing characteristic of ontologies is that they are based on an Open World Assumption (OWA) which is based on the AAA slogan or Anyone can say Anything about Any topic. Statements that are not included in an ontology are considered unknown as opposed to false. In contrast, information models of data structures such as XML messages and relational databases are based on a Closed World Assumption (CWA) which holds that any statement that is not known to the message or database to be true is false (this is also referred to as "negation as failure" or NF).

The OWA principle recognizes that our understanding of the world is incomplete, evolving, and that new knowledge can be discovered and added at any time. To return to Alan Rector's example, one cannot assume that because there is no mention of a patient's body temperature in an electronic health record message, that the patient does not have a body temperature. Another distinguishing characteristic of ontologies is the Nonunique Naming Assumption as opposed to the Unique Name Assumption (UNA) in CWA-based systems. People do use different labels to represent the same concept. This discussion of OWA vs. CWA is not just academic. The reality is that data about a patient can exist in multiple systems, organizations, jurisdictions, and even countries using different vocabularies and XML data structures. Concepts such as longitudinal or lifelong health record and medication reconciliation will soon reveal the limits of healthcare systems based on a CWA.

OWL2, a W3C Recommendation, is an expressive ontology language and provides reasoning and inferencing capabilities to software applications. Logical axioms specify restrictions through property domains and ranges. OWL2 also support negation and disjunction. OWL2 reasoning capabilites can be enhanced with a rule language such as the Semantic Web Rule Language (SWRL). Given the complexity and scale of medical knowledge today, the use of ontology-based reasoning will become essential in applications such as medical terminologies, clinical knowledge management for automated decision support, and even automatically verifying the accuracy of messages exchanged between healthcare applications.

Unfortunately, ontologies are not widely used in software engineering today. They are not well understood by the majority of developers. Undergraduate computer science programs don't usually teach ontologies. There is an urgent need to educate a new generation of ontology-savvy healthcare application developers.

Model Consistency

For obvious reasons, healthcare applications require a high degree of model quality and consistency. This is not always possible or easy to do with traditional approaches such as object-oriented design (the HL7 RIM is based on the UML) and data structures such as XML and relational database schemas.

A clear and clean separation of concerns is needed between the semantic model (the ontology) and the information model (the model of how the data is structured in an XML message or the health application's data store). The ontology can be used to verify that the content of a message is accurate in regard to our conceptualization of the world, while the information model is used to validate the data structure in the data stores and XML messages exchanged with other applications.

The HL7 RIM is definitely not an ontology and has been plagued by consistency issues. Futhermore, a consequence of the RIM model refinement process that is used to derive XML message exchange schemas from the RIM is that data structure concerns have leaked into what was touted as the semantic model. This lack of separation of concerns has led to an unwieldy information model and very complex XML message structures (in the CDA and other V3 messages) that are difficult to learn and implement in software applications. The GreenCDA is a possible answer to the message structure simplification challenge (see my previous post on the Greening of the HL7 CDA). However, it is not enough to solve the semantic interoperability challenge.

Ontologies and Medical Terminologies

In a paper titled "Why Do It the Hard Way? The Case for an Expressive Description Logic for SNOMED", Alan Rector and Sebastian Brandt argued in favour of using the OWL ontology language for SNOMED which is currently based on a Description Logic semantics known as EL++. The availability of computing power (particularly the elasticity and massive scalability of the cloud), reasoners, and tools have now made such a migration possible.

In a recent paper published in the Journal of the American Medical Informatics Association (JAMIA) and titled "Getting the foot out of the pelvis: modeling problems affecting use of SNOMED CT hierarchies in practical applications" (subscription required), Alan Rector, Sam Brandt, and Thomas Schneider used an OWL representation of SNOMED CT to unearth errors in SNOMED CT hierarchies for such common conditions as myocardial infarction, diabetes, and hypertension. This has significant practical implications for the use and interpretation of SNOMED codes in electronic health records (EHRs), post-coordination, and queries in software applications.

ICD-11 is being developed using OWL to allow consistency checking and linking to other biomedical terminologies and ontologies.

In addition to OWL, the Simple Knowledge Organization System (SKOS) specification can also used to represent thesauri, classification schemes, taxonomies, controlled vocabularies, and other concept schemes.

Overlap between the HL7 RIM and SNOMED CT

HL7 V3 messages like the CDA typically carry codes from SNOMED CT and other terminologies such as CPT, ICD9, and LOINC. However in certain cases such as family history, an observation can be expressed through a single SNOMED CT code or by using the RIM. To ensure model consistency, HL7 has released an implementation guide on using SNOMED CT in HL7 Version 3 documents such as the HL7 CDA (I refer to this implementation guide as Terminfo). In addition, HITSP C80 specifies vocabularies and terminologies to be used in various sections of a C32 document.

However, these guidelines have been difficult to enforce in practice due to the lack of automated validation tools. In a paper recently published in the Journal of Biomedical Semantics titled "Semantic validation of the use of SNOMED CT in HL7 clinical documents", Stijn Heymans, Matthew McKennirey, and Joshua Phillips described an approach using OWL ontologies to automatically validate Terminfo guidelines. The approach consisted in using the OWL representation of SNOMED CT, lifting (with XSLT) CDA XML instances into OWL individuals based on a CDA OWL ontology, and by expressing Terminfo guidelines as OWL integrity constraints. The latter were validated with the Pellet Integrity Constraint validator or Pellet-ICV.

Clarifying the relationship ("interface") between Ontologies, Coding Systems, and Information Models

I mentioned the need for a clean separation of concerns between the ontology and the information model. So what is the relationship between ontologies, coding systems (like SNOMED CT), and information models? I have long been intrigued by that question. In a paper titled "Binding Ontologies & Coding systems to Electronic Health Records and Messages", Alan Rector, Rahil Qamar, and Tom Marley write:

We contend that codes are also data structures – or more precisely symbols to be used in data structures – and that the model of codes is also at the level of the information model.

Although coding systems are derived from ontologies, we also need a separation of concerns between the coding system and the ontology. Remember that ontologies are based on an "Open World Assumption" (which means essentially that Anyone can say Anything about Any topic or the AAA slogan). Coding systems in contrast contain an enumerated list of codes to choose from.

In the same paper, the authors propose a code binding interface based on OWL DL between the model of meaning (i.e., the ontology), the model of codes (i.e., the terminology) and the information model.

To summarize our findings so far:

We first create an ontology to describe our conceptualization (or understanding) of the world.
We derive an enumerated list of codes called code system (itself a data structure) from the ontology.
We used the codes in EHR applications databases and messages which are data structures.
We can validate the binding between the ontology, the information model, and the code system (using the approach proposed by Alan Rector, Rahil Qamar, and Tom Marley).

Ontology Alignment

An ontology represents a specific world view that reflects the perspective of its origin (application, domain, people, or organization). Alignment consists in mapping concepts across ontologies. For example, in translational medicine, there could be a need to map an ontology used in biomedical research to an ontology used for clinical purposes. Several techniques can be used to achieve Ontology Alignment between two ontologies including:

Mapping each ontology to a third shared ontology called a foundational ontology
Mapping the two ontologies directly.

OWL facilitates ontology alignment through constructs such as owl:sameAs, owl:equivalentClass, and owl:equivalentProperty. These OWL constructs can be enhanced with a rule-based mapping using SWRL or RIF constructs. XSLT and SPARQL can also be useful in Ontology Alignment.

Clinical Knowledge Management (CKM)

Ontologies as knowledge representation formalism are well suited for modeling the medical knowledge contained in Clinical Practice Guidelines (CPGs) and Care Pathways (CPs). This enables automated reasoning and the execution of those guidelines based on patient data at the point of care.

Several ontology-based approaches to modelling CPGs and CPs have been proposed in the past including PROforma, HELEN, EON, GLIF, and SAGE. However, the lack of tooling has been a major impediment to a wide adoption of those standards. OWL has the advantage of being a widely implemented W3C Recommendation with available open source as well as commercial tools.

Ontologies and Enterprise Master Data Management (MDM)

As healthcare enterprises become larger and integrated (through the ACO model for example), there will be the need to consistently define and manage core business entities such as "patient", "provider", "payor", "care delivery", and "claim" across systems and business processes (e.g. research, clinical, reporting, and financial). The goal of Master Data Management (MDM) is to address those challenges.

One area of particular interest to MDM is the naming, meaning, equivalency, and relationships between those core business entities. Ontology constructs such as owl:sameAs, owl:equivalentClass, and owl:equivalentProperty can help establish common semantics across the enterprise when the same business entity is called by different names in different systems and business processes.

Linked Open Data (LOD)

Ontologies can help in building silo-busting applications that need to link data items (datum) to other data items (as opposed to web page to web page) over the web in order to perform entity correlation (or entity resolution). A datum can be a row in a relational database and technologies exist to provide an RDF view over a relational database table (see the R2RML: RDB to RDF Mapping Language). The RDF view itself can be defined in terms of an OWL ontology or RDFS vocabulary. Hence, LOD can integrate data across health applications and organizations by providing a semantic layer on top of existing applications.

The Linked Data design pattern is based on an open world assumption, uses dereferenceable HTTP URIs for identifying and accessing data items, RDF for describing metadada about those items, and semantic links to describe the relationships between those items. Other standards used in LOD applications include RDFS (for describing RDF vocabularies) and SQARQL (for querying RDF graphs). A practical application of LOD in healthcare is the Clinical Quality Linked Data project on health.data.gov.

Metadata and the PCAST Report

The Office of the National Coordination for Health Information Technology (ONC) recently released an Advance Notice of Proposed Rulemaking (ANPRM) on Metadata Standards to Support Nationwide Electronic Health Information Exchange. The ANPRM was driven by the PCAST Report released in December 2010.

Specifically, the ANPRM called for public comments on patient identity, provenance, and privacy. There are existing ontologies related to identity, provenance, and privacy that can be at least partially reused (ontology reuse is a recommended best practice to avoid the difficulties of ontology alignment). An example is the Provenance Vocabulary Core Ontology. Modeling metadata in healthcare using ontologies will enable reasoning, data integration through Linked Open Data mechanisms, and federated SPARQL queries. Please note that metadata expressed in XML syntax can be lifted into RDF (using techniques like XSLT or XQuery) to provide the same benefits.

UPDATE: IBM and Wellpoint announced in September 2011 that they will collaborate in using IBM Watson (an advanced Question Answering (QA) system) to help clinicians make better decisions. A tutorial titled Semantic Web Technology in Watson was presented at the 2011 International Semantic Web Conference (ISWC 2011) in October.

Service-Oriented Clinical Decision Support in the Cloud

2011-07-13T20:46:00.001-04:00

This is the presentation I gave today at the 2011 SOA In Healthcare Conference in Herdnon, VA.

Service-Oriented CDS in the Cloud

View more presentations from Joel Amoussou

How Checklists Can Enhance Clinical Decision Support (CDS)

2011-03-24T22:00:00.017-04:00

I have been reading "The Checklist Manifesto", a book written by Dr. Atul Gawande on the effectiveness of checklists in healthcare delivery. Another paper recently published in the Milbank Quarterly entitled "Counterheroism, Common Knowledge, and Ergonomics: Concepts from Aviation That Could Improve Patient Safety" suggests that beyond checklists, proven aviation safety practices such as Cockpit Resource Management (CRM), Joint safety briefings, and first-names-only rules could help improve patient safety.

I first became aware of the importance of checklists while I was being trained as a Flight Engineer. I spent a lot of time studying them carefully as an aviation student. Checklists are used during normal, abnormal, and emergency situations and pilots go through practical exercises in flight simulators to use them correctly. Let's not mince words: aviation as we know it today would not be possible without checklists.

In a study entitled "Missed and Delayed Diagnoses in the Emergency Department: A Study of Closed Malpractice Claims From 4 Liability Insurers", researchers found that:

The leading breakdowns in the diagnostic process were failure to order an appropriate diagnostic test (58% of errors), failure to perform an adequate medical history or physical examination (42%), incorrect interpretation of a diagnostic test (37%), and failure to order an appropriate consultation (33%). The leading contributing factors to the missed diagnoses were cognitive factors (96%), patient-related factors (34%), lack of appropriate supervision (30%), inadequate handoffs (24%), and excessive workload (23%).

Checklists can serve as cognitive aid in helping clinicians do their job safely. While the idea of using checklists and standard operating procedures has been fully embraced and adopted by aviation professionals for more than 70 years, it is only now making inroads into the field of medicine particularly in high pressure environments like intensive care units. The use of checklists in medicine has already shown the potential to save patients live and reduce human errors. However, the main challenge remains the acceptance of checklists by clinicians concerned about "Cookbook Medicine".

Checklists are just cognitive aids and the presence of an experienced and competent professional will always make a big difference in critical situations. As Captain Sullenberger (the airline pilot who successfully ditched US Airways Flight 1549 in the Hudson River in New York City, on January 15, 2009) said, "One way of looking at this might be that for 42 years, I've been making small, regular deposits in this bank of experience: education and training. And on January 15 the balance was sufficient so that I could make a very large withdrawal."

On modern airplanes, Electronic Centralised Aircraft Monitor (ECAM) systems or Engine Indicating and Crew Alerting Systems (EICAS) monitor aircraft systems and engines and displays messages in case of failure, as well as recommended remedial actions in the form of checklists. The National Transportation Safety Board (NTSB) accident report on US Airways Flight 1549 indicates that the First Officer "was able to promptly locate the [Engine Dual Failure checklist] procedure listed on the back cover of the [Quick Reference Handbook] QRH, turn to the appropriate page, and start executing the checklist."

In medicine, factors such as comorbidity can complicate the design of effective CDS. However, with the explosion of medical knowledge and evidence-based guidelines, CDS will become an essential tool in healthcare delivery. The design, development, implementation, and use of CDS is knowledge-intensive and require an effective collaborative knowledge management strategy. The challenge will be to integrate checklists into the different CDS modalities such as context-sensitive Infobuttons, order sets, alerts, reminders, data entry and visualization, and clinical workflows.

For example, the evaluation results (in the form of recommendations) of a CDS rule can be presented to the clinician as an electronic checklist. This in turn can be tied directly to quality measures in the era of Meaningful Use, Pay-For-Performance, and Accountable Care Organizations (ACOs). An interesting example would be a checklist that prompts clinicians to generate detailed discharge instructions to satisfy quality measures for patients with heart failure or acute myocardial infection.

There is an important Human Factors aspect to the design and use of cockpit checklists and flight-deck procedures. This has been the subject of advanced research at NASA Ames Research Center more than twenty years ago and the results have been widely disseminated and implemented in the aviation industry.

In an article entitled "The Checklist" published in the New Yorker, Atul Gawande wrote:

"But consider: there are hundreds, perhaps thousands, of things doctors do that are at least as dangerous and prone to human failure as putting central lines into I.C.U. patients. It’s true of cardiac care, stroke treatment, H.I.V. treatment, and surgery of all kinds. It’s also true of diagnosis, whether one is trying to identify cancer or infection or a heart attack. All have steps that are worth putting on a checklist and testing in routine care. The question—still unanswered—is whether medical culture will embrace the opportunity."

Peter Pronovost, an intensivist at Johns Hopkins Hospital and pioneer in the use of checklist in medicine, implemented a checklist at 127 Michigan intensive care units (ICUs) to reduce catheter-related blood stream infections (CRBSI). The project was so successful that it is estimated that it could significantly reduce the 28,000 deaths and 3 billion dollars in costs caused by these hospital-acquired infections.

The HL7 Clinical Decision Support (CDS) workgroup is working on standards for the vMR (Virtual Medical Record), Infobuttons, and Order Sets. There is also an effort at the OMG to publish a Clinical Decision Support Services specification for service-oriented CDS capabilities. The Flight Operation Interests Group (FOIG) of the Air Transport Association (ATA) is developing a data model and XML Schema for flight deck procedures and checklists. Developing a shareable content model for checklists in medicine could be an interesting idea.

The Greening of the HL7 CDA

2011-02-27T18:59:00.032-05:00

I attended the HIMSS 2011 Conference this week in Orlando, FL. The GreenCDA was one of the big themes at the HL7 booth. The goal of the HL7 GreenCDA project is to provide a simple intermediary XML representation of the CDA to facilitate quick learning and ease of use for developers building healthcare data exchange solutions. Using the GreenCDA should not require prior knowledge of the HL7 Reference Information Model (RIM) and the associated model refinement process.

Developers should be able to generate code from the GreenCDA XML schema using data binding tools in any programming language of their choice. It should also be possible to create a round-trip transformation between the GreenCDA and the CDA. These requirements also apply to CDA implementations such as the HITSP C32. The GreenCDA will be available as an HL7 Implementation Guide and the HL7 Structured Document Working Group recently issued a GreenCDA wire format position statement.

In a previous post entitled "XML Processing in Healthcare Applications", I described some of the issues with the HL7 CDA and HITSP C32 XML structure and suggested some ideas on dealing with the complexity of the CDA schema and C32 generation process. In this post, I will share some thoughts on what can be done to ensure that the GreenCDA lives up to its full potential as the answer to the simplification challenge in healthcare data exchange standards.

XML Schemas In the Software Development Lifecycle

The XML schema is an important part of the service contract in Service Oriented Architecture (SOA). Services contracts also include the WSDL and WS-Policy documents. Using the recommended contract-first approach to web services development, developers generate client as well as server code using various tools and APIs in their native programming language and framework. Even when not using a pre-existing industry XML schema, the contract-first approach allows developers to decouple the service contract from platform-specific idiosyncrasies and adhere to cross-platform interoperability standards such as the WSI-Basic Profile.

On the Java platform, JAX-WS and JAXB allow developers to generate code from the WSDL and XML schema with tools like WSDL2Java.

On the .NET platform, the Windows Communication Framework (WCF) and Visual Studio provide data binding tools out-of-the-box like the Svcutil. There is also an open source tool called WSCF.blue specifically designed to facilitate contract-first web services development on the .NET platform.

The GreenCDA XML schema could also be used in support of the "Canonical Data Model" enterprise integration pattern. Enterprise data architects typically extend industry XML schemas components to satisfy custom needs.

Finally, the PCAST Report released in December 2010 recommended a universal exchange language that is "structured as individual data elements, together with metadata that provide an annotation for each data element". The report suggests that the metadata attached to each of these data elements

"...would include (i) enough identifying information about the patient to allow the data to be located (not necessarily a universal patient identifier), (ii) privacy protection information—who may access the mammograms, either identified or de-identified, and for what purposes, (iii) the provenance of the data—the date, time, type of equipment used, personnel (physician, nurse, or technician), and so forth."

Put together, these requirements argue in favor of a GreenCDA XML schema that supports the following:

Reusability
Extensibility
A well-defined versioning strategy
Seamless code generation in a variety of programming languages and development frameworks
A metadata facility per the PCAST recommendations.

Designing for Reuse and Extensibility

I suggest that the GreenCDA should only use global and named simple and complex types to facilitate reuse and extensibility. In other words, anonymous type definitions should be avoided. Extensibility is typically implemented through the <xsd:extension> element. Reuse can also be achieved by assembling logically related schema components into separate schema documents and using the <xsd:include> and <xsd:import> constructs.

Common XML schema components (also called core components) such as Hl7 datatypes as well as person, address, and organization should be in a separate schema file ideally under a different namespace than the target namespace of the GreenCDA itself.

Component Naming and Documentation

It would be nice to have different naming conventions for types vs. elements and attributes. Also schema component names should be spelled out for readability. A component name like "ivlTs" is not obvious for someone who is not familiar with HL7 datatypes.

Each type, element, or attribute should have a required <xs:annotation> child element which describes the semantics of the element in its child <xs:documentation> element. In other words, all schema components should be documented.

Support for Data Binding Tools

Certain features of the XML Schema language such as mixed content models, <xsd:choice>, and dynamic type substitution with xsi:type are not well supported by various XML databinding tools. The need to use these constructs to accurately express the GreenCDA XML data structure should be balanced against the ability to seamlessly generate code from the GreenCDA XML schema using various XML databinding tools.

Before the GreenCDA is released for production use, I suggest at least two open source reference implementations in two different development platforms (such as Java and .NET) covering the end-to-end web services development cycle using the specific tooling provided by the respective platforms.

What Can Be Learned From the National Information Exchange Model (NIEM)

The ONC Standards and Interoperability Framework is leveraging the NIEM from a process perspective. However, I believe there is much to be learned from the design of the NIEM as an XML data exchange standard. This does not imply that the GreenCDA should use the NIEM Core. It simply means that the healthcare domain can leverage certain NIEM design principles that are not only backed by advanced research (at Georgia Tech Research Institute) in XML schema modeling, but are also proven by the numerous government agencies using the NIEM.

The NIEM embodies recognized XML Schema design patterns in its Naming and Design Rules (NDR). The NIEM provides a schematron-based tool to automatically validate XML schemas against the rules defined in the NDR. For example, the schematron schema can enforce component naming conventions or the requirement to document every schema component.

The PCAST Report says:

"We think that a universal exchange language must facilitate the exchange of metadata tagged elements at a more atomic and disaggregated level, so that their varied assembly into documents or reports can itself be a robust, entrepreneurial marketplace of applications."

The NIEM defines an extensible metadata facility for adding metadata to any data elemeent in the spirit of the PCAST recommendations. The NIEM itself support the exchange of "data items" at any level of granularity. These XML Schema Design Patterns are universal and can be applied to any domain including the healthcare domain.

A Therapeutic Layered Cake

2011-02-03T18:45:00.019-05:00

With all the talk about the PCAST Report, I've been doing some Systems thinking on semantic interoperability in healthcare IT. Trying to put all the pieces together, I remembered Tim Berners-Lee's "Semantic Web Layer Cake".

The Semantic Web layer Cake has gone through several iterations over the years (see James Hendler's presentation on that subject). However, I think it can still be very helpful in visualizing a unified framework for addressing the challenges of semantic interoperability in Healthcare IT.

As we move to Stage 2 of Meaningful Use, I believe Clinical Decision Support (CDS) will take center stage. Beyond currently used XML-based data structures (such as HL7 v3 messages), this will put an increased emphasis on medical terminologies, ontologies, and knowledge representation in OWL. For example, ICD-11 is being developed using OWL to allow consistency checking and linking to other biomedical terminologies and ontologies. Equally important to knowledge representation, but not shown in the layer cake above is the Simple Knowledge Organization System (SKOS) specification.

In a report entitled "Semantic Interoperability Deployment and Research Roadmap", Alan Rector summarized the difference between the notions of ontology, knowledge representation, and data model:

Ontology – A representation of what is universally true, including what is true by definition

Knowledge Representation or "Background knowledge resource" – a representation of what is generally true, or widely known to be true in some specific instance. In general, the knowledge representation is formulated in terms of and indexed by the Ontology.

Information model or Data model a model of how information is structured in a given software system, message, or electronic health record. In general, the data structures carry codes for the ontology as their content.

Clinical guidelines are published in the form of narrative text, sometimes with an evaluation algorithm. The translation of those guidelines into an executable representation is a complex and costly process. Several formalisms and standards have been proposed such as the Arden Syntax, GLIF, GELLO, and GEM. However, none of these standards has been widely adopted. Developed with inputs from the Business Rules, Logic Programming, and Semantic Web communities, the W3C Rule Interchange Format (RIF) can help with the interchange of executable Clinical Decision Support (CDS) rules in addition to adding reasoning capabilities to patient records. This example shows how decision support rules could be exchanged between two rules engines (Drools and Jess) using the RIF PRD syntax, a standard XML serialization format for production rule languages.

Existing patient records marked up in XML HITSP C32 or ASTM CCR can be lifted into RDF statements (with XSLT or XQuery for example) and queried using SPARQL.

Proof, Trust, and Cryptography are being currently addressed by various standards and specifications in the healthcare industry notably the OASIS Cross-Enterprise Security and Privacy Authorization (XSPA) Profiles of XACML, SAML, and WS-Trust.

On the User Interface side, I see HTML5 giving both Flex and Silverlight a run for their money in the next few years. This will be driven in part by the demand for mobile health (mHealth).

XML Processing in Healthcare Applications

2011-01-22T09:41:00.021-05:00

Meaningful Use certification requires the ability to create patient summaries in either C32 or CCR format. One of the most frequently asked questions on the HL7 Structured Document mailing list is related to the processing of the CDA XML schema with data binding tools such as JAXB or Castor. Initially, people are not able to generate Java classes with JAXB at all. After some changes to the schema, JAXB finally works and creates hundreds of classes which are not very easy to work with and maintain. Then someone suggests using the Model-Driven Health Tools (MDHT) CDA tools which are Java-based. You face additional headache if you're not developing on the Java platform.

In a paper presented at the Balisage 2009 conference, a team of engineers who implemented the "Laika" C32 compliance testing tool described the issues with the CDA and C32 XML structure:

Repeated use of overly abstract data structures: The HL7 CDA defines a number of very generic objects that are used to represent information in a given document. Differing information, such as medications and conditions, are represented using the same XML elements with very subtle changes in their nesting and attributes. This makes a CDA document difficult to process.

Underspecified implementation, including lack of a normative schema: While there is an XML schema for the HL7 CDA, a final schema does not exist for the HITSP C32 or other CDA-based documents due to their use of attributes for selecting templates. Thus, defining schemas for these documents is impossible. As a result, CDA-based constructs such as HITSP C32 cannot be automatically validated by XML parsers; standard object mapping tools, such as XML Beans or JAXB, cannot be used.

Ambiguous data types: Data can be represented in multiple ways in a CDA document. Consumers of CDA documents must, therefore, write software that handles any of the numerous permutations of these data types. This leads to bloated software, or more likely, software that does not implement the full specification and experiences interoperability problems when it receives data in an unexpected format.

Steep and long learning curve: Mastery of the CDA and its many specifications and constructs takes an experienced software engineer many months to achieve. Once learned, it is very cumbersome to employ in robust software applications and services. These difficulties drive up the cost and time to develop and maintain health care software, thus reducing the pace of innovation.

In a previous post entitled "The Future of Healthcare Data Exchange Standards", I suggested some ideas on how to develop standard XML schemas that support the software development process as opposed to hindering it. Since we're not there yet, in this post I will suggest some ideas on dealing with the complexity of the CDA schema and C32 generation process.

The key is to leverage the power of XML related technologies such as XPath2, XSLT2, XQuery, XProc, ISO Schematron, and even XML Schema 1.1 (for assertions or business rules constraints) to simplify the task. First, generate a simple and perhaps flat XML representation (let's call it simpleC32) of the patient summary from your domain objects or database (through a data transfer object or DTO for example). That simpleC32 contains all the content that is needed to populate the C32 templates and generate a valid C32 document. You can create your own XML schema for your simpleC32 and use it for validation and data binding.

Once you have a valid simpleC32 document, you can use XSLT2 to transform the patient summary from your simpleC32 representation into a C32 document that can be validated against the NIST Meaningful Use C32 Validator. This is roughly the idea behind the GreenCDA project. Use that as an inspiration on how to create a simple representation of the C32. You can even use the GreenCDA XML schema as your simpleC32. But don't hesitate to create your own simpleC32 if the GreenCDA does not work for you, because the target is still the C32, and the idea here is to have an intermediary representation (an Adapter) to make your life easier. It is also an approach that allows you to isolate your domain model and prevent the complexity of the C32 data model from leaking into your domain layer (see my previous post on the concept of Anti-Corruption Layer in Domain Driven Design).

Why is this approach not used more often? Some developers who code with imperative programming languages (such as Java, C#, or JavaScript) are not comfortable with declarative programming using languages like XSLT2 and XQuery. I've recently seen a Java developer use JAXB to create hundreds of classes and thousands of hard to maintain lines of code for a simple transformation from the CDA to a different target XML schema.

The basic difference between declarative (and functional) programming languages and imperative languages is that the former specify the "what" (the intent) as opposed to the "how" (the algorithm). However, declarative programming with XSLT2 and XQuery can be mastered through training and practice: see my previous posts entitled: "In Defense of XSLT", "Why XProc Rocks", and Putting XQuery to Work in Healthcare".

While Java and C# are general purpose languages, processing languages like XSLT2, XQuery, and XProc are actually based on the XQuery 1.0 and XPath 2.0 Data Model (XDM) and specifically designed for the purpose of manipulating XML documents. This is particularly helpful when dealing with a complex and deep structure such as the HL7 CDA and other HL7 V3 messages. These XML-centric processing languages use XPath2 to navigate the XML tree. In general, consider using them in the following cases:

Applications that require dealing with a complex industry data exchange XML schema which is not easy to process with your databinding and other development tools. In that case, create an intermediary simpe XML representation and map it to the industry data exchange XML schema using XSLT2 or XQuery (XQuery is not just for querying native XML databases, it is also a powerful language for processing XML documents).
Applications that require translation from an XML schema to another target XML schema (for example a mapping from the HL7 CCD to the ASTM CCR or from the C32 to XHTML).
Applications that require translation from an XML representation to a non-XML representation and round-trip (for example HL7 v2.x to HL7 V3, C32 XML to JSON, or C32 to a non-XML serialization of RDF).
Consider using XProc if you need to chain multiple XML processing steps such as: query a data source with XQuery, expand XIncludes, validate against XML schema, validate against a schematron schema, transform with XSLT2, generate a PDF document with XSL FO, and so on.

The Universal Exchange Language proposed by the PCAST Report could be an opportunity to address the issues listed above.

How Not to Build A Big Ball of Mud

2010-12-19T16:44:00.014-05:00

If you are a software developer or architect, in addition to the ever changing business requirements, you also need to deal with the myriad of application development frameworks and design patterns out there. There are frameworks for: the User Interface (UI), Dependency Injection (DI), Aspect Oriented Programming (AOP), and Object-Relational Mapping (ORM). On top of that, you will probably need a web services framework and perhaps an Enterprise Service Bus (ESB) if you need to integrate applications. As an architect, you also need to keep an eye on scalability, availability, security, usability, industry standards, and government regulations.

In such an environment, the lack of a disciplined approach to software architecture can quickly lead to a Big Ball of Mud. In a paper presented in 1997 at the Fourth Conference on Patterns Languages of Programs, Brian Foote and Joseh Yoder describe the Big Ball of Mud:

A BIG BALL OF MUD is haphazardly structured, sprawling, sloppy, duct-tape and bailing wire, spaghetti code jungle. We’ve all seen them. These systems show unmistakable signs of unregulated growth, and repeated, expedient repair. Information is shared promiscuously among distant elements of the system, often to the point where nearly all the important information becomes global or duplicated. The overall structure of the system may never have been well defined. If it was, it may have eroded beyond recognition. Programmers with a shred of architectural sensibility shun these quagmires. Only those who are unconcerned about architecture, and, perhaps, are comfortable with the inertia of the day-to-day chore of patching the holes in these failing dikes, are content to work on such systems.

The Big Ball of Mud remains the most pervasive architecture today. Note that these problems can be exacerbated by an agile software development approach that leaves little or no room for design and strategic thinking (see my previous post on software architecture documentation in agile projects).

Domain Driven Design (DDD) is a set of patterns that have been introduced by Eric Evans in his book entitled: "Domain-Driven Design: Tackling Complexity in the Heart of Software". I won't go into the details of what those patterns are. I do recommend that you read the book and there are other free DDD resources on the web as well. However, I will share with you some key DDD principles that have been helpful to me in wrapping my head around software architecture complexity:

Collaboration between software developers and domain experts is important to create a common understanding of the concepts of the domain. Note that we're not talking about UI components such as screens or fields here, nor are we talking about computer science abstractions such as classes and objects. We are talking about what the domain is made of conceptually. These domain concepts are expressed in a Ubiquitous Language that is used not only in conversations and software documentation, but also in the code. So in essence, DDD is model-driven design: the model can be translated into code and frameworks like Naked Objects can help you do exactly that. Continuously refine the model.
Experimentation and rapid prototyping are an efficient way to collaborate with domain experts and business analysts. This is where the Naked Objects Framework can be very helpful. Note that you can use Naked Objects for the initial prototyping and domain modeling, and then use your preferred frameworks for the remaining layers of your application.
Pay attention to the correct implementation of DDD building blocks such as: entities (behaviour-rich with business rules), value objects, aggregates, aggregate roots, domain events, factories, repositories, and services. Avoid an anemic domain model and know how to recognize value objects and persist them properly. Value objects (such as money and time interval) are immutable and are manipulated through side-effect free functions. Move complexity and behaviour out of your entities into those value objects.
Domain concepts are grouped into modules to delineate what Eric Evans calls the "conceptual contours" of the domain. To reduce coupling, OO patterns such as the dependency inversion principle, the interface segregation principle, and the acyclic dependency principles are applied.
DDD recommends the following four layers: the presentation layer, the application layer, the domain layer, and the infrastructure layer. Although the idea of a multi-tier architecture is not new, the anti-patterns are typically: a fat application layer, an anemic domain model, and a tangled mess in general. So, properly layer your architecture:
- Repository interfaces are in the domain layer, but their implementation are in the infrastructure layer to allow "Persistence Ignorance"
- Both the interface and implementation of factories are in the domain layer
- Domain and infrastructure services are injected into entities using dependency injection (some argue that DDD is not possible without DI, AOP, and ORM)
- The application layer takes care of cross-cutting concerns such as transactions and security. It can also mediate between the presentation layer and domain layer through Data Transfer Objects (DTOs).
DDD enables Object Oriented User Interfaces (OOUI) which expose the richness of the domain layer as opposed to obscuring it.
Models exist within bounded contexts and the latter should be clearly identified. In his book, Eric Evans talks about "strategic design" and "context maps" and suggests the following options for integrating applications:
- Published language
- Open host service
- Shared kernel
- Customer/supplier
- Conformist
- Anti-corruption layer
- Separate ways.
In industries such as healthcare where an XML-based data exchange standard exists, the "Published Language" approach is the pattern typically used. Each healthcare application participating in an exchange represents a separate context. On the other hand, an "Anti-Corruption Layer" can be created as an adapter to isolate the model against an industry standard model that is not considered best practice in data modeling, is inconsistent, immature, or subject to change. However, since there is tremendous value in exchanging data, we hope not to go "Separate Ways".
DDD is a solid foundation for next-generation architecture based on the Command Query Responsibility Segregation (CQRS) pattern. The UI sends commands which are handled by command handlers. These command handlers change the state of aggregate roots. However, the aggregate roots still define behavior and business rules and are responsible for maintaining invariants. Changes to aggregate roots generate events that are stored in an event store (this is called event sourcing). Aggregate roots are persisted by storing these streams of events in the event store. That way, the aggregate roots can be reconstructed by replaying the events from the event store. The events are published to subscribers including denormalizers for enhanced query performance. The separation of the read side from the write side allows:
- Increased performance and scalability on the read and reporting side particularly when combined with a cloud deployment model
- Complete audit trails through the event store
- New data mining capabilities leveraging temporal queries
- The opportunity to eliminate the need for Object Relational Mapping (ORM) through the use of high performance NoSQL databases.

Toward a Universal Exchange Language for Healthcare

2010-12-13T22:07:00.023-05:00

The US President's Council of Advisors on Science and Technology (PCAST) published a report last week entitled: "Realizing the Full Potential of Health Information Technology to Improve Healthcare for Americans: The Path Forward". The report calls for a universal exchange language for healthcare (abbreviated as UELH in this post). Specifically, the report says:

"We believe that the natural syntax for such a universal exchange language will be some kind of extensible markup language (an XML variant, for example) capable of exchanging data from an unspecified number of (not necessarily harmonized) semantic realms. Such languages are structured as individual data elements, together with metadata that provide an annotation for each data element."

First, let me say that I fully support the idea of a UELH. I've written in the past about the future of healthcare data exchange standards. The ASTM CCR and the HL7 CCD have been adopted for Meaningful Use Stage 1 and that was the right choice. In my opinion, the UELH proposed by PCAST is about the next generation healthcare data exchange standard that is yet to be built. It's part of the natural evolution and innovation that are inherent to the information technology industry. It is also a very challenging task that should be informed by the important work that has been done previously in this field including:

The ASTM CCR
The HL7 RIM, CDA, CCD, and greenCDA
Archetype-based EN 13606 from OpenEHR
The National Information Exchange Model (NIEM)
HITSP C32
Biomedical Ontologies using semantic web technologies such as OWL2, SKOS, and RDF.
Medical Terminologies such as SNOMED and RxNorm.

This new language should focus on identifying, addressing, and solving the issues with the use of the current set of healthcare data exchange standards. This will require a public discourse that is cordial and focused on solutions and innovative ideas. Most importantly, it will require listening to the concerns of implementers. This proposal should not be about reinventing the wheel. It should be about creating a better future by learning lessons from the past while being open-minded about new ideas and approaches to solving problems.

Note that the report talks about the syntax of this new language as some kind of an "XML variant". It also mentioned that the language must be extensible. This is important in order to enable innovation in this field. For example, we've recently seen a serious challenge to XML coming from JSON in the web APIs space (Twitter and Foursquare removed support for XML in their APIs and now only provide a JSON API). Similarly, in the Semantic Web space, alternatives to the RDF/XML serialization syntax have emerged such as the N-triples notation. This is not to say that XML is the wrong representation for healthcare data. It simply means that we should be open to innovation in this area.

Metadata and the Semantic Web in Healthcare

Closely related to the notion of metadata is the idea of the Semantic Web. Although semantic web technologies are not widely used in healthcare today, they could help address some of the issues with current healthcare standard information models including: model consistency, reasoning, and knowledge integration across domains (e.g. the genomics and clinical domains). In a report entitled "Semantic Interoperability Deployment and Research Roadmap", Alan Rector, an authority in the field of biomedical ontologies, explains the difference between ontologies and data structures:

A second closely related notion is that of an "information model" of "model of data structures". Both Archetypes and HL7 V3 Messages are examples of data structures. Formalisms for data structures bear many resemblances to formalisms for ontologies. The confusion is made worse because the description logics are often used for both. However, there is a clear difference.

Ontologies are about the things being represented – patients, their diseases. They are about what is always true, whether or not it is known to the clinician. For example, all patients have a body temperature (possibly ambient if they are dead); however, the body temperature may not be known or recorded. It makes no sense to talk about a patient with a "missing" body temperature.
Data structures are about the artefacts in which information is recorded. Not every data structure about a patient need include a field for body temperature, and even if it does, that field may be missing for any given patient. It makes perfect sense to speak about a patient record with missing data for body temperature.

A key point is that "epistemological issues" – issues of what a given physician or the healthcare system knows – should be represented in the data structures rather than the ontology. This causes serious problems for terminologies coding systems, which often include notions such as "unspecified" or even "missing". This practice is now widely deprecated but remains common.

One of the Common Terminology Services (CTS 2) submissions to the OMG is based on Semantic Web technologies such as OWL2, SKOS, and SPARQL. The UELH proposed by the PCAST should leverage the work that has been done by the biomedical ontology community.

The NIEM Approach to Metadata-Tagged Data Elements

The report goes on to say that the metadata attached to each of these data elements

"...would include (i) enough identifying information about the patient to allow the data to be located (not necessarily a universal patient identifier), (ii) privacy protection information—who may access the mammograms, either identified or de-identified, and for what purposes, (iii) the provenance of the data—the date, time, type of equipment used, personnel (physician, nurse, or technician), and so forth."

The report does not explain exactly how this should be done. So let's combine the wisdom of the NIEM, HL7 greenCDA, and OASIS XSPA (Cross-Enterprise Security and Privacy Authorization Profile of XACML for healthcare) to propose a solution. Let's assume that we need to add metadata about the equiment used for the lab result as well as patient consent directives to the following lab result entry which is marked up in greenCDA format:


<result>
  <resultID root="107c2dc0-67a5-11db-bd13-0800200c9a66" />
  <resultDateTime value="200003231430" />
  <resultType codeSystem="2.16.840.1.113883.6.1" code="30313-1"
  displayName="HGB" />
  <resultStatus code="completed" />
  <resultValue>
    <physicalQuantity value="13.2" unit="g/dl" />
  </resultValue>
  <resultInterpretation codeSystem="2.16.840.1.113883.5.83"
  code="N" />
  <resultReferenceRange>M 13-18 g/dl; F 12-16
  g/dl</resultReferenceRange>
</result>

In the following, an s:metadata attribute is added to the root element (s:metadata is of type IDREFS and for brevity, I am not showing the namespace declarations):


<result s:metadata="equipment consent">
  <resultID root="107c2dc0-67a5-11db-bd13-0800200c9a66" />
  <resultDateTime value="200003231430" />
  <resultType codeSystem="2.16.840.1.113883.6.1" code="30313-1"
  displayName="HGB" />
  <resultStatus code="completed" />
  <resultValue>
    <physicalQuantity value="13.2" unit="g/dl" />
  </resultValue>
  <resultInterpretation codeSystem="2.16.840.1.113883.5.83"
  code="N" />
  <resultReferenceRange>M 13-18 g/dl; F 12-16
  g/dl</resultReferenceRange>
</result>

The following is the lab test equipment metadata:


<LabTestEquipmentMetadata s:id="equipment">
  <SerialNumber>93638494749</SerialNumber>
  <Manufacuturer>MedLabEquipCo.</Manufacturer>
</LabTestEquipmentMetadata>

And here is the patient consent directives marked in XACML XSPA format (this snippet is taken from the NHIN Access Consent Policies Specification):


<ConsentMetadata s:id="consent">
<Policy xmlns="urn:oasis:names:tc:xacml:2.0:policy:schema:os"
PolicyId="12345678-1234-1234-1234-123456781234"
RuleCombiningAlgId="urn:oasis:names:tc:xacml:1.0:rule-combining-algorithm:first-applicable">
<Description>Sample XACML policy for NHIN</Description>
<!-- The Target element at the Policy level identifies the subject to whom the Policy applies -->
<Target>
<Resources>
<Resource>
<ResourceMatch MatchId="http://www.hhs.gov/healthit/nhin/function#instance-identifier-equal">

<AttributeValue DataType="urn:hl7-org:v3#II"
xmlns:hl7="urn:hl7-org:v3">
  <hl7:PatientId root="2.16.840.1.113883.3.18.103"
  extension="00375" />
</AttributeValue>
<ResourceAttributeDesignator AttributeId="http://www.hhs.gov/healthit/nhin#subject-id"
DataType="urn:hl7-org:v3#II" />
</ResourceMatch>
</Resource>
<Actions>
<!-- This policy applies to all document query and document retrieve transactions -->
<Action>
<ActionMatch MatchId="urn:oasis:names:tc:xacml:1.0:function:anyURI-equal">

  <AttributeValue DataType="http://www.w3.org/2001/XMLSchema#anyURI">
  urn:ihe:iti:2007:CrossGatewayRetrieve</AttributeValue>
  <ActionAttributeDesignator AttributeId="urn:oasis:names:tc:xacml:2.0:action"
  DataType="http://www.w3.org/2001/XMLSchema#anyURI" />
</ActionMatch>
</Action>
<Action>
<ActionMatch MatchId="urn:oasis:names:tc:xacml:1.0:function:anyURI-equal">

  <AttributeValue DataType="http://www.w3.org/2001/XMLSchema#anyURI">
  urn:ihe:iti:2007:CrossGatewayQuery</AttributeValue>
  <ActionAttributeDesignator AttributeId="urn:oasis:names:tc:xacml:2.0:action"
  DataType="http://www.w3.org/2001/XMLSchema#anyURI" />
</ActionMatch>
</Action>
</Actions>
<Rule RuleId="133" Effect="Permit">
<Description>Permit access to all documents to all
physicians and nurses</Description>
<Target>
<Subjects>
  <Subject>
 <SubjectMatch MatchId="urn:oasis:names:tc:xacml:1.0:function:string-equal">

   <!-- coded value for physicians -->
   <AttributeValue DataType="http://www.w3.org/2001/XMLSchema#string">
   112247003</AttributeValue>
   <SubjectAttributeDesignator AttributeId="urn:oasis:names:tc:xacml:2.0:subject:role"
   DataType="http://www.w3.org/2001/XMLSchema#string" />
 </SubjectMatch>
  </Subject>
  <Subject>
 <SubjectMatch MatchId="urn:oasis:names:tc:xacml:1.0:function:string-equal">

   <!-- coded value for nurses -->
   <AttributeValue DataType="http://www.w3.org/2001/XMLSchema#string">
   106292003</AttributeValue>
   <SubjectAttributeDesignator AttributeId="urn:oasis:names:tc:xacml:2.0:subject:role"
   DataType="http://www.w3.org/2001/XMLSchema#string" />
 </SubjectMatch>
  </Subject>
</Subjects>
<!-- since there is no Resource element, this rule applies to all resources -->
</Target>
</Rule>
<Rule RuleId="134" Effect="Permit">
<Description>Allow access Dentists and Dental Hygienists
Access from the Happy Tooth dental practice to documents
with "Normal" confidentiality for a defined time
period.</Description>
<Target>
<Subjects>
  <Subject>
 <SubjectMatch MatchId="urn:oasis:names:tc:xacml:1.0:function:string-equal">

   <!-- coded value for dentists -->
   <AttributeValue DataType="http://www.w3.org/2001/XMLSchema#anyURI">
   106289002</AttributeValue>
   <SubjectAttributeDesignator AttributeId="urn:oasis:names:tc:xacml:2.0:subject:role"
   DataType="http://www.w3.org/2001/XMLSchema#string" />
 </SubjectMatch>
 <SubjectMatch MatchId="urn:oasis:names:tc:xacml:1.0:function:anyURI-equal">

   <AttributeValue DataType="http://www.w3.org/2001/XMLSchema#anyURI">
   http://www.happytoothdental.com</AttributeValue>
   <SubjectAttributeDesignator AttributeId="urn:oasis:names:tc:xspa:1.0:subject:organization-id"
   DataType="http://www.w3.org/2001/XMLSchema#anyURI" />
 </SubjectMatch>
  </Subject>
  <Subject>
 <SubjectMatch MatchId="urn:oasis:names:tc:xacml:1.0:function:string-equal">

   <!-- coded value for dental hygienists -->
   <AttributeValue DataType="http://www.w3.org/2001/XMLSchema#string">
   26042002</AttributeValue>
   <SubjectAttributeDesignator AttributeId="urn:oasis:names:tc:xacml:2.0:subject:role"
   DataType="http://www.w3.org/2001/XMLSchema#string" />
 </SubjectMatch>
 <SubjectMatch MatchId="urn:oasis:names:tc:xacml:1.0:function:anyURI-equal">

   <AttributeValue DataType="http://www.w3.org/2001/XMLSchema#anyURI">
   http://www.happytoothdental.com</AttributeValue>
   <SubjectAttributeDesignator AttributeId="urn:oasis:names:tc:xspa:1.0:subject:organization-id"
   DataType="http://www.w3.org/2001/XMLSchema#anyURI" />
 </SubjectMatch>
  </Subject>
</Subjects>
<Resources>
  <Resource>
 <ResourceMatch MatchId="urn:oasis:names:tc:xacml:1.0:function:string-equal">

   <AttributeValue DataType="http://www.w3.org/2001/XMLSchema#string">
   N</AttributeValue>
   <ResourceAttributeDesignator AttributeId="urn:oasis:names:tc:xspa:1.0:resource:patient:hl7:confidentiality-code"
   DataType="http://www.w3.org/2001/XMLSchema#string" />
 </ResourceMatch>
  </Resource>
</Resources>
<Environments>
  <Environment>
 <EnvironmentMatch MatchId="urn:oasis:names:tc:xacml:1.0:function:date-greather-than-or-equal">

   <AttributeValue DataType="http://www.w3.org/2001/XMLSchema#date">
   2009-07-01</AttributeValue>
   <EnvironmentAttributeDesignator AttributeId="http://www.hhs.gov/healthit/nhin#rule-start-date"
   DataType="http://www.w3.org/2001/XMLSchema#date" />
 </EnvironmentMatch>
  </Environment>
  <Environment>
 <EnvironmentMatch MatchId="urn:oasis:names:tc:xacml:1.0:function:date-less-than-or-equal">

   <AttributeValue DataType="http://www.w3.org/2001/XMLSchema#date">
   2009-12-31</AttributeValue>
   <EnvironmentAttributeDesignator AttributeId="http://www.hhs.gov/healthit/nhin#rule-end-date"
   DataType="http://www.w3.org/2001/XMLSchema#date" />
 </EnvironmentMatch>
  </Environment>
</Environments>
</Target>
</Rule>
<Rule RuleId="135" Effect="Deny">
<Description>deny all access to documents. Since this
rule is last, it will be selected if no other rule
applies, under the rule combining algorithm of first
applicable.</Description>
<Target />
</Rule>
</Resources>
</Target>
</Policy>
</ConsentMetadata>

Please note the following:

Metadata "LabTestEquipmentMetadata" asserts the equipment used for the lab test.
Metadata "ConsentMetadata" asserts the patient consent directives leveraving the XSPA XACML format.
Metadata can be declared once and reused by multiple elements.
An element can refer to 0 or more metadata objects.

In NIEM, an appinfo:AppliesTo element in a metadata type declaration is used to indicate the type to which the metadata applies as in the following example (note this is not enforced by the XML schema validating parser, but can be enforced at the application level):


<xsd:complexType name="LabTestEquipmentMetadataType">
  <xsd:annotation>
    <xsd:appinfo>
      <i:AppliesTo i:name="LabResultType" />
    </xsd:appinfo>
  </xsd:annotation>
  <xsd:complexContent>
<xsd:extension base="s:MetadataType">
...
</xsd:extension>
</xsd:complexContent>
</xsd:complexType>

<xsd:element name="LabTestEquipmentMetadata" type="LabTestEquipmentMetadataType" nillable="true"/>

NIEM defines a common metadata type that can be extended by any type definition that requires metadata:


<schema
    targetNamespace="http://niem.gov/niem/structures/2.0"
    version="alpha2"
    xmlns:i="http://niem.gov/niem/appinfo/2.0"
    xmlns:s="http://niem.gov/niem/structures/2.0"
    xmlns="http://www.w3.org/2001/XMLSchema">

 
  <attribute name="id" type="ID"/>
  <attribute name="linkMetadata" type="IDREFS"/>
  <attribute name="metadata" type="IDREFS"/>
  <attribute name="ref" type="IDREF"/>
  <attribute name="sequenceID" type="integer"/>
   
  <attributeGroup name="SimpleObjectAttributeGroup">
    <attribute ref="s:id"/>
    <attribute ref="s:metadata"/>
    <attribute ref="s:linkMetadata"/>
  </attributeGroup>

   <element name="Metadata" type="s:MetadataType" abstract="true"/>

  <complexType name="ComplexObjectType" abstract="true">
      <attribute ref="s:id"/>
      <attribute ref="s:metadata"/>
      <attribute ref="s:linkMetadata"/>
  </complexType>

  <complexType name="MetadataType" abstract="true">
      <attribute ref="s:id"/>
  </complexType>

</schema>

Any type definition that needs metadata can simply extend ComplexObjectType as follows for lab result type:


<xsd:complexType name="LabResultType">
  <xsd:complexContent>
    <xsd:extension base="s:ComplexObjectType">
      <xsd:sequence>...</xsd:sequence>
    </xsd:extension>
  </xsd:complexContent>
</xsd:complexType>

Software Architecture Documentation in Agile Projects

2010-10-13T20:11:00.014-04:00

One misconception that I often hear in Agile circles is that there is no need for software architecture documentation in Agile because "code is self-documenting". The emphasis in agile is not to eliminate the need for design and documentation, but to avoid Big Up Front Design (BDUF). Design and architecture documentation are still important in Agile. However, you only need just enough design and documentation to start coding. In other words, don't over-document.

As you code and refactor, some of the software architecture documentation will become quickly obsolete and should be discarded. Use tools such as Maven, SchemaSpy, Doxygen, and UmlGraph to auto-generate up-to-date documentation from your source code. A wiki is also a good tool for publishing and sharing architecture documentation. For consistency, I recommend using a template for documenting the architecture.

Provide the documentation only if it is really needed and used by stakeholders. So, don't try to document everything. You do need to document the following:

Design decisions and their rationale
Design patterns and development frameworks used
The architecture viewpoints and quality attributes that cannot be easily gleaned from the code alone.

Far too often, software architecture documentation only covers the code view. This is not enough. Stakeholders are not limited to developers, but also include end users, testers, the operational staff, compliance auditors, etc. When writing software architecture documentation, I first identify all stakeholders and their concerns. To ensure that I provide a 360-degree view of the architecture, I develop the architecture documentation based on the viewpoints and perspectives described by Nick Rozanski and Eoin Woods in their book "Software Systems Architecture: Working With Stakeholders Using Viewpoints and Perspectives" (Addison Wesley, April 2005)

The following are the Architecture Viewpoints:

Functional
Information
Concurrency
Development
Deployment
Operational

And here are the Architecture Perspectives:

Security
Performance and Scalability
Availability and Resilience
Evolution
Accessibility
Development Resource
Internationalization
Location
Regulation
Usability

These viewpoints and perspectives can be described using different notations such as UML (using stereotypes and profiles like SoaML for service oriented architecture), Business Process Modeling Notation (BPMN), and Domain Specific Languages (DSLs).

The Future of Healthcare Data Exchange Standards

2010-09-22T22:11:00.023-04:00

Meaningful Use Final Rule has finally been released and I think now is a good time to start thinking about where we want to be five years from now in terms of healthcare data exchange standards.

Listening to the Concerns of Implementers

I think it is very important that we listen to the concerns of the implementers of the current set of standards. They are the users of those standards and good software engineers like to get feedback from their end users to fix bugs and improve their software. The following post details some of the concerns out there regarding the current HL7 v3 XML schema development methodology and I believe they should not be ignored: Why Make It Simple If It Can Be Complicated?

Using an Industry Standard XML Schema: A Developer's Perspective

XML documents are not just viewed by human eyeballs through the use of an XSLT stylesheet. The XML schema has become an important part of the service contract in Service Oriented Architecture (SOA). SOA has emerged during the last few years as a set of design principles for integrating applications within and across organizational boundaries.

In the healthcare sector for example, the Nationwide Health Information Network (NHIN) and many Health Information Exchanges (HIEs) are being built on a decentralized service-oriented architecture using web services standards such as SOAP, WSDL, WS-Addressing, MTOM, and WS-Policy. The Web Services Interoperability (WS-I) Profiles WS-I Basic and WS-I Security provide additional guidelines that should be followed to ensure cross-platform interoperability for example between .NET and Java EE platforms. Some of the constraints defined by the WS-I Basic Profile are related to the design of XML schemas used in web services.

An increasingly popular alternative to the WS-* stack is to use RESTful web services. The REST architectural style does not mandate the use of web services contract such as XML schema, WSDL, and WS-Policy. However, the web application description language (WADL) has been proposed to play the role of service contract for RESTful web services. This post will not engage in the SOAP vs. REST debate except to mention that both are used in implementation projects today.

On top of these platform-agnostic web services standards, each platform defines a set of specifications and tooling for building web services applications. In the Java world, these specifications include:

The Java API for XML Web Services (JAX-WS)
The Java Architecture for XML Binding (JAXB)
The Java API for RESTful Web Services (JAX-RS)
The Streaming API for XML (StAX).

JAX-WS and JAXB allow developers to generate a significant amount of Java code from the WSDL and XML schema with tools like WSDL2Java. The quality of a standard XML schema largely depends on how well it supports the web services development process and that's why I believe that creating a reference implementation should be a necessary step before the release of new standards. An industry standard XML schema that is hard to use will directly translate into high implementation cost resulting from development project delays for example.

Embracing Design Patterns

Beyond our personal preferences (such as the NIEM vs. HL7 debate), there are well established engineering practices and methodologies that we can agree on. In terms of software development, design patterns have emerged as a well known approach to building effective software solutions. For example, the following two books have had a strong influence in the fields of object-oriented design and enterprise application integration respectively (and they sit proudly on my bookshelf):

Design Patterns: Elements of Reusable Object-Oriented Software by Erich Gamma, Richard Helm, Ralph Johnson, and John Vlissides
Enterprise Integration Patterns: Designing, Building, and Deploying Messaging Solutions by Gregor Hohpe and Bobby Woolf.

An interesting design pattern from the "Enterprise Integration Patterns" book that is relevant to the current discussion on industry standard XML schemas is the "Canonical Data Model" design pattern. Enterprise data architects tasked with creating such canonical data models often reuse components from industry standard XML schemas. That approach makes sense but cannot succeed if the industry standard XML schema is not designed to support reusability, extensibility, and a clearly specified versioning strategy.

Modeling Data In Transit vs. Data at Rest

Modeling data at rest (e.g. data stored in relational databases) is a well established discipline. For example, data modeling patterns for relational data have been captured by Len Silverston and Paul Agnew in their book entitled "The Data Model Resource Book, Vol. 3: Universal Patterns for Data Modeling".

There is a need to apply the same engineering rigor to modeling data in transit (e.g. data in web services messages). The XML Schema specification became a W3C Recommendation more than 9 years ago and I think there is now enough implementation experience to start building consensus around a set of XML Schema Design Patterns. The latter should address the following issues:

Usability: the factors that affect the ability of an average developer to quickly learn and use an XML schema in a software development project
Component Reusability
Web services cross-platform interoperability constraints. Some of those constraints are defined by the WS-I Basic Profile
Issues surrounding the use of XML databinding tools such as JAXB. This is particularly important since developers use those tools for code generation in creating web services applications. It is well known that existing databinding tools do not provide adequate support for all XML Schema language features
Ability to manipulate instances with XML APIs such as StAX
Schema extensibility, versioning, and maintainability.

These design patterns should be packaged into a Naming and Design Rules (NDR) document to ensure a consistent and proven approach to developing future XML vocabularies for the healthcare domain.

The XML Schema 1.1 specification is currently a W3C Candidate Recommendation. It defines new features such as conditional type assignments and assertions which allow schema developers to consolidate structural and business rules constraints into a single schema. This could help alleviate some of the pain associated with the multiple layers of Schematron constraints currently specified by HITSP C32, IHE PCC, and the HL7 CCD (sometimes referred to as the "HITSP Onion"). Saxon supports some of these new features.

Developing Standards the Way We Develop Software

The final point I'd like to make is that we should start creating healthcare standards the same way we develop software. I am a proponent of agile development methodologies such as Extreme Programming and Scrum. These methodologies are based on practices such as user stories, iteration (sprint) planning, unit test first, refactoring, continuous integration, and acceptance testing. Agile programming helps create better software and I believe it can help create better healthcare standards as well.

Secure Cloud Computing for the Health Enterprise

2010-08-30T14:24:00.002-04:00

Some thoughts on securing healthcare applications in the cloud.

Secure Cloud Computing for the Health Enterprise

View more presentations from Joel Amoussou.

Health IT Standards in Canada

2010-08-18T22:35:00.012-04:00

In Canada, health IT standards are established by Canada Health Infoway's Standard Collaborative. Canada Health Infoway is a not-for-profit organization funded by the federal government to develop pan-Canadian health IT standards and provide incentives for the adoption of health information technologies.

HIT Standards

The following are key standards approved by the Standard Collaborative:

Systematized Nomenclature of Medicine Clinical Terms (SNOMED CT) for problem lists, procedures, and other clinical observations. 35,000 SNOMED CT concepts have been translated to Canadian French
HL7 Version 3 (HL7 V3) for clinical, ﬁnancial, and administrative messaging and document exchange. Canada played a key role in the development of the HL7 v3 modeling methodology and tooling. The pan-Canadian HL7 v3 is used for the following core areas:
- Laboratory
- Immunization
- Pharmacy (Drugs)
- Client Registry (patient demographics)
- Provider Registry
- Shared Health Records
- Electronic Claims
- Public health surveillance
HL7 Clinical Document Architecture (CDA) standards enables pan-Canadian EHR interoperability
The pan-Canadian LOINC Observation Code Database (pCLOCD) for lab test results is used by the Lab Messaging and Nomenclature and Public Health Surveillance standards. PCLOCD adds and excludes certain records from the original LOINC standard to support Canadian requirements. Unified Code for Units of Measure (UCUM) is used for units of measures
Diagnostic Imaging (DI) Standards are based on DICOM and IHE XDS-I
The Health Canada Drug Product Database (HCDPD) provides coding for medications.

Certification

Infoway offers certification for the following classes of HIT software:

Client registry
Consumer health application
Consumer health platform
Diagnostic Imaging (DI)
Drug Information System (DIS)
Immunization registry
Provider registry

The assessment criteria cover functionality, privacy, security, interoperability and management and are based on the following standards:

Functionality – Canada Health Infoway Electronic Health Record Privacy and Security Requirements.

Privacy – Canada Health Infoway Electronic Health Record Infostructure (EHRi) Privacy & Security Conceptual Architecture; Government of Canada’s Personal Information Protection and Electronic Documents Act (PIPEDA); The Canadian Standards Association’s Model Code for the Protection of Personal Information – CAN-CSA-Q830-03.

Security – Canada Health Infoway Electronic Health Record Infostructure (EHRi) Privacy & Security Conceptual Architecture; The International Organization for Standardization’s Code of Practice for Information Security Management – ISO/IEC 17799:2005; The National Institute of Standards and Technology’s Recommended Security Controls for Federal Information Systems – NIST SP800-53;The USA Health Insurance Portability and Accountability Act (HIPAA) Security Rule.

Interoperability – Canada Health Infoway pan-Canadian Standards and Conformance Profile Definitions for diagnostic imaging, laboratory, drug, shared health record, and demographic information.

Management – The IT Governance Institute Control Objectives for Information and Related Technology (COBIT); The Office of Government Commerce’s Information Technology Infrastructure Library (ITIL).

Architecture

The following diagram from Infoway's web site depicts the high level architectural vision from an end user perspective (click to enlarge).

My Assessment

Overall, I am pleased with the choices that have been made by Infoway's Standard Collaborative. I believe that HL7 V3 is a step forward compared to HL7 v2.x because it is based on XML, it is more amenable to a Service-Oriented Architecture (SOA), and it defines a healthcare Reference Information Model (RIM) with associated modeling methodology and tooling. Beyond ICD-9 and CPT, SNOMED CT is definitively the medical terminology language of the future. Consistency is needed for units of measure for lab test results and UCUM is a good choice.

Some flexibility will be needed in standardizing transport protocols to allow lightweight solutions such as SMTP for point-to-point connections (what our friends in the US call NHIN Direct). The e-Health certification process should include some HIT usability testing (see my previous post on the experience of the British NHS in this area).

I am not aware of any pan-Canadian standard in the area of quality reporting. I would like to see Canada Health Infoway put more efforts into creating specifications (messaging, security, privacy) and open source tools for Health Information Exchanges (HIEs) at the provincial and federal levels. More work needs to be done in promoting the use of Computerized Physician Order Entry (CPOE) systems and Clinical Decision Support Systems (CDSS) for the automated execution of clinical guidelines. Finally, more guidance is needed in the area of patient consent in the context of electronic health information exchanges.

Content Integration in the Aviation Industry Using CMIS

2010-08-01T22:41:00.004-04:00

The recently approved Content Management Interoperability Services (CMIS) specification could play a very important role in ensuring that aircraft operators receive up-to-date maintenance and operation documentation from aviation manufacturers.

The safe and efficient maintenance and operation of air vehicles require clear, technically accurate, and up-to-date technical documentation. The technical documentation is supplied by original equipment manufacturers (OEMs), regulatory agencies, and the aircraft operator's own engineering staff. OEMs (e.g. airframe, engine, and component manufacturers) provide regular publications such as Aircraft Maintenance Manuals (AMM) and Flight Crew Operating Manuals (FCOM) as well as time-sensitive supplements such as Service Bulletins (SBs) and Temporary Revisions (TRs). Regulatory agencies like Transport Canada and the US Federal Aviation Administration (FAA) also publish technical information that affects the maintenance and operation of air vehicles and equipments. Examples are Advisory Circulars (ACs), Airworthiness Directives (ADs), and various forms and regulations.

A typical airline faces the following challenges:

The elimination of the high costs associated with the shipping, storage, and distribution of physical products (paper, CDs, and DVDs) containing the technical documentation.
The safety and regulatory compliance concerns related to the use of out-of-date technical information (currently, some airlines receive revisions to technical manuals only four times a year).

The aerospace industry is in the process of adopting the new S1000D technical publications standard. S1000D is based on the concepts of modularity, reuse, and metadata (see my post on S1000D Content Reuse). The Flight Operation Interests Group (FOIG) of the Air Transport Association (ATA) is developing a data model and XML Schema for flight deck procedures and checklists also based on the S1000D data model. While S1000D is the right payload format, the exchange between content management and publishing systems within the industry must be orchestrated in an efficient manner.

Airlines, repair stations, regulatory agencies, and original equipment manufacturers (OEMs) manage and publish technical content using proprietary content management systems (CMS) each with its own proprietary API. Some companies now provide online portals where customers can login to get the latest documentation. However, pilots and technicians don't really want to login into the support sites of all those content providers to find out what is new and updated. To minimize aircraft downtime, aircraft mechanics want to connect to the aircraft's health and usage monitoring system (HUMS), determine what problem needs to be fixed, and have the appropriate content aggregated (work package) and presented to them.

With CMIS, an airline or aircraft operator can create a portal to aggregate content from the repositories of its OEM suppliers using a single standardized web services interface based on either SOAP or AtomPub (the RESTful alternative). This allows the aircraft operator to keep their maintenance and operation documentation updated at all time without having to wait for a CD or paper manual to be shipped by the OEM.

The second scenario is distributed authoring driven by the shift to distributed aircraft manufacturing. For example, the content of the Aircraft Maintenance Manual (AMM) can be provided by different aviation manufacturers participating in a consortium to design, manufacture, and support a new aircraft. In such as a scenario, a centralized CMIS-compliant content repository (hosted by the airframe manufacturer acting as the content integrator) can provide the following CMIS services to other members of the consortium:

Policy and ACL Services to obtain the policies (such as retention) and Access Control List (ACL) currently applied to a document
Navigation Services to programmatically navigate the content repository
Discovery services to query content. CMIS supports SQL-92 with some extensions and full-text search and can handle federated search across multiple repositories
Relationship Services to obtain the relationships (such as links) associated with a document
Versioning Services to check-out and check-in a document
Object Services to obtain the properties of a document and create folders and documents
Filing Services to add a document to a folder.

The third example use case is the ability for a SCORM-compliant Learning Management System to integrate with CMIS-compliant S1000D Common Source DataBases (CSDB) in order to repurpose technical publications content for training purposes. The International S1000D-SCORM Bridge Project is an interesting initiative to create such an integration.

In general, CMIS will enable new capabilities such as the remote access to library services, cross-repository exchange, cross-repository aggregation, and cross-repository observation (or notification).

CMIS is now supported by major CMS vendors including EMC, IBM, Alfresco, and Microsoft. A list of open source and commercial implementations of CMIS is available at this page.

Toward An Enterprise Reference Architecture for the Meaningful Use Era

2010-07-24T11:22:00.008-04:00

Meaningful Use is not just about financial incentives. In fact, the financial incentives provided under the HITECH Act will only cover a portion of the total costs associated with migrating the healthcare industry to health information technologies. Consider the following Meaningful Use criteria:

Submit electronic data to immunization registries
Incorporate clinical lab-test results into certified EHR technology as structured data
Submit electronic syndromic surveillance data to public health agencies
Generate and transmit permissible prescriptions electronically (eRx)
Provide patients with an electronic copy of their health information
Exchange key clinical information among providers of care and patient authorized entities electronically.

These criteria are clearly designed to bridge information silos within and across organizational boundaries in the healthcare industry. This integration is necessary to improve the coordination and quality of care, eliminate the costs associated with redundancies and other inefficiencies within the healthcare system, empower patients with their electronic health records, and enable effective public health surveillance. This integration requires a new health information network such as those provided by state and regional Health Information Exchanges (HIEs) and the Nationwide Health Information Network (NHIN) initiative. HIEs and NHIN Exchange are based on a decentralized, service-oriented architecture (SOA) whereby authorized health enterprises securely exchange health information.

Health CIOs who see Meaningful Use as part of a larger transformational effort will drive their organizations toward success. Creating a coherent and consistent Enterprise Architecture for tackling these new challenges should be a top priority. Not having a coherent Enterprise Architecture will lead to a chaotic environment with increased costs and complexity. The following are some steps that can be taken to create an Enterprise Reference Architecture that is aligned with with the organization's business context, goals, and drivers such as Meaningful Use:

Adopt a proven architecture development methodology such as TOGAF.
Create an inventory of existing systems such as pharmacy, laboratory, radiology, patient administration, electronic medical records (EMRs), order entry, clinical decision support, etc. This exercise is necessary to gain an understanding of current functions, redundancies, and gaps.
Create a target enterprise service inventory to eliminate functional redundancies and maximize service reuse and composability. These services can be layered into utility, entity, and task service layers. For example, Meaningful Use criteria such as "Reconcile Medication Lists", "Exchange Patient Summary Record", or "Submit Electronic Data to Immunization Registries" can be decomposed into two or more fine-grained services.
Task services based on the composition of other reusable services can support the orchestration of clinical workflows. The latter previously required clinicians to log into multiple systems and re-enter data, or relied on point-to-point integration via the use of interface engines. These task services can also invoke services across organizational boundaries from other providers participating in a HIE in order to compose a longitudinal electronic health record (EHR) of the patient. Other services such as clinical decision support services or terminology services can be shared by multiple healthcare organizations to reduce costs.
Consider wrapper services to encapsulate existing legacy applications.
The Enterprise Reference Architecture should support standards that have been mandated in the Meaningful Use Final Rule, but also those required by NHIN Exchange, NHIN Direct, and the HIE in which the organization participates. These standards are related to data exchange (HITSP C32, CCR, HL7 2.x), messaging (SOAP, WSDL, REST, SMTP, WS-I Basic), security and privacy (WS-I Security, SAML, XACML), IHE Profiles (XDS, PIX, and PDQ), and various NHIN Exchange and NHIN Direct Specifications. While standards are not always perfect, healthcare interoperability is simply not possible without them.
Select a SOA infrastructure (such as an Enterprise Service Bus or ESB) that supports the standards listed above. Consider both open source and commercial offerings.
Consider non-functional requirements such as performance, scalability, and availability.
The Enterprise Reference Architecture should also incorporate industry best practices in the areas of SOA, Security, Privacy, Data Modeling, and Usability. These best practices are captured in various specifications such as the HL7 EHR System Functional Model, the HL7 Service Aware Interoperability Framework (SAIF), and the HL7 Decision Support Service (DSS) specification.
Finally, create a Governance Framework to establish and enforce enterprise-wide technology standards, design patterns, Naming and Design Rules (NDRs), policies, service metadata and their versioning, quality assurance, and Service Level Agreements (SLAs) such as the NHIN Data Use and Reciprocal Support Agreement (DURSA).

Adopting the NIEM for Health Information Exchange

2010-07-12T09:40:00.013-04:00

A National Information Exchange Model (NIEM) Information Exchange Package Documentation (IEPD) for health information exchange is being considered by the U.S. Department of Health and Human Services Office of the National Coordinator (ONC). After years of work developing the HL7 CCD and HITSP C32, will a NIEM IEPD represent a step forward in health data exchange standards? Will the NIEM IEPD process work in the healthcare domain? In adopting the NIEM process, can we leverage the work that has been done so far by organizations such as HITSP, HL7, and the ASTM?

NIEM vs. RIM

Having previously used the NIEM standard in a project, my first reaction when I started to work with the HL7 CDA and the related Reference Information Model (RIM) refinement process was, "why not use the NIEM?". The NIEM approach is proven and has been successfully used in complex data exchange scenarios in criminal justice and other sensitive domains.

The HL7 RIM expressed as a set of UML class diagrams defines a foundation model for health care and clinical data. The HL7 Clinical Document Architecture (CDA) is derived from the HL7 RIM through a refinement process that generates a Refined Message Information Model (R-MIM) for the CDA. The HL7 Continuity of Care Document (CCD) and HITSP C32 specifications define additional constraints on the HL7 CDA. Although the HL7 CCD was in fact an original attempt to harmonize the HL7 CDA and the ASTM CCR, there is still a need to define a unified data model for health information exchange. The Meaningful Use Interim Final Rule (IFR) allows both the CCD and the CCR for data exchange.

Ease of Use

From a software development point of view, the HL7 CCD and HITSP C32 standards are complex to learn and use. Data exchange standards don't provide any value by themselves. They provide value when they are used to create software solutions that improve the quality and safety of care and reduce costs. Therefore, ease of use represents an important requirement for a health data exchange standard. How do you create an XML data exchange standard that developers can learn quickly and that is easy to use?

Something as simple as adopting a component naming convention that conveys the semantics of those components is a good start. For example, ISO 11179, Part 5 entitled "Naming and Identification Principles" defines the following convention for naming data components:

Object-class qualifier terms (0 or more).
An object class term (1).
Property qualifier terms (0 or more).
A property term (1).
Representation qualifier terms (0 or more).
A representation term (1).

An example of an element name that follows that recommendation is <PersonBirthLocation>. Not having a component naming convention or using cryptic names make it hard for developers to understand a new XML vocabulary. Disk space is cheap these days, so there is no reason to keep using those cryptic names.

Defining components is also important. ISO 11179, Part 4 entitled "Formulation of Data Definitions" provides the following recommendations for defining data components:

state the essential meaning of the concept
be precise and unambiguous
be concise
be able to stand alone
be expressed without embedding rationale, functional usage, or procedural information
avoid circular reasoning
use the same terminology and consistent logical structure for related definitions
be appropriate for the type of metadata item being defined

In general, consistency based on a predefined XML Schema Naming and Design Rules (NDRs) document which is itself based on recognized XML Schema design patterns can contribute to ease of use.

Developing Software

Key to adoption is the ability for an average software developer to use familiar tools including existing open source software to generate and process instances of the exchange XML schema. Examples of these tools include:

XML databinding tools such as JAXB
XML APIs such as JDOM or DOM4J
Mapping and storing the data in existing relational databases (this is what the majority of developers know)
Web Services Framework such as Apache CXF.

Reference Implementation and Testing Ease of Use

Creating a reference implementation with existing open source tools and testing ease of use should be part of releasing new health data exchange standards.

The reference implementation should support a complete SOA-based exchange scenario and provide artifacts that are typically generated in a real web services project. This includes designing a data model for storage, WSDLs, precompiling the XML schema with a databinding tool such as JAXB, and using an open source web services tool such as Apache CXF.

One way to determine ease of use is to ask average developers to process instances of the XML schema using familiar XML APIs and then measure how long it took them to write the code as well as the Cyclomatic Complexity (and other code quality metrics) of the written code.

Benefits of Adopting NIEM in the Healthcare Domain

I believe that NIEM is a good choice for moving healthcare interoperability forward for the following reasons:

The NIEM is a proven data exchange standard that has been used in mission critical environments such as the justice and intelligence domains.
The NIEM embodies recognized XML Schema design patterns in its Naming and Design Rules (NDR). For example, the NIEM component naming convention is based on ISO 11179 Parts 4 and 5. Per the NIEM NDR, all schema type definitions must be global to enable reuse. The NIEM provides a schematron-based tool to automatically validate XML schemas against the rules defined in the NDR.
The NIEM provides core components that can be used or extended for the healthcare domain. Core NIEM data components such as <Person>, <Organization>, and <Address> are universal components that are needed in healthcare as well. NIEM also specifies an elegant solution for modeling roles, associations between entities, and code lists.
The NIEM provides various tools to facilitate the IEPD process. These tools include the Schema Subset Generation Tool (SSGT), the NIEM Wayfarer, and conformance validation tools. The SSGT presents a shopping cart metaphor for assembling existing components into new exchange packages. The ONC anticipates the need for additional tools for the healthcare domain.
A companion specification called the Logical Entity Exchange Specification (LEXS) can be used to define a messaging model for web services. LEXS has its own set of tools.

A NIEM Concept of Operations for the Healthcare Domain

The diagram below is extracted from a presentation made at the HIT Standards Committee meeting on June 30, 2010 and entitled: Standards & Interoperability Framework Concept of Operations (ConOps) Overview (click to enlarge).

NIEM Healthcare IEPD vs. NIEM Healthcare Domain

My personal recommendation is that the new harmonization structure (the successor to HITSP) should create a NIEM Healthcare domain as opposed to a NIEM IEPD for the healthcare domain. The healthcare domain is complex enough to warrant its own domain within the NIEM. A NIEM Healthcare domain should provide a data exchange standard for Electronic Health Records (EHRs) and leverage the work done by HITSP C32, C80, and C83 in defining required data elements and code lists (terminology) for various EHR content modules. Instead of mapping those data elements to the HL7 CCD XML schema as was done by HITSP, those elements will be mapped to existing, extended, or new NIEM components. Specific health data exchanges can then create IEPDs to satisfy their unique requirements by reusing data components from the NIEM.

It should be possible to map the CDA RMIM into the NIEM model using the NIEM Component Mapping Tool or CMT. It should also be possible to leverage at least some components from the HL7 CCD XML schema in building a NIEM Healthcare domain or IEPD. However, these components will have to be renamed and restructured to comply with the NIEM naming and design rules.

Patient Consent: State of the Union

2010-07-01T11:44:00.007-04:00

The viability and sustainability of an electronic health information exchange (HIE) largely depends on the trust and desire of patients to participate in the exchange. What are the laws, regulations, standards, practices, and current directions in the field of patients' privacy consents?

State and Federal Laws

The Health Insurance Portability and Accountability Act of 1996 (HIPAA) does not require patient consent and authorization for the exchange of health information among healthcare providers for the purpose of medical care. The patient consent is implied by her general consent to be treated.

Some states have adopted heightened laws (higher than those imposed by HIPAA) that require explicit patient consent and authorization for specially-protected health information such as sexually transmitted diseases, human immunodeficiency virus tests, viral hepatitis, genetic information, substance abuse, mental health, and developmental disabilities.

Building Trust

The lack of patient trust can be a significant barrier to the implementation of a HIE. Therefore, a common practice in HIEs is to offer individual patients the opportunity to opt-out of exchanging their health information even if patient consent is not required by existing laws and regulations. Patients are notified and informed of their consent options through an outreach program.

Consent Requirements

The HHS Office of the National Coordinator (ONC) releases a Consumer Preferences Requirements Document in October 2009. The document describes consent stakeholders, functional needs, policy implications, scenarios, and processes including HIEs.

Consent Options

The ONC released a whitepaper in March 2010 entitled "Consumer Consent Options for Electronic Health Information Exchange: Policy Considerations and Analysis". The whitepaper identified the following consent options:

No consent. Health information of patients is automatically included—patients cannot opt out;

Opt-out. Default is for health information of patients to be included automatically, but the patient can opt out completely;

Opt-out with exceptions. Default is for health information of patients to be included, but the patient can opt out completely or allow only select data to be included;

Opt-in. Default is that no patient health information is included; patients must actively express consent to be included, but if they do so then their information must be all in or all out; and

Opt-in with restrictions. Default is that no patient health information is made available, but the patient may allow a subset of select data to be included.

The granularity of patient consent preference can be based on the type of data, healthcare provider, time range, and intended use.

Standardization

The IHE Basic Patient Privacy Consents (BPPC) provides a mechanism to record and enforce patient consents. It complements the IHE Cross-Enterprise Document Sharing (XDS) standard by specifying how an XDS affinity domain can create privacy policies and enforce those policies through the access control mechanisms of an XDS Actor such as an EHR system. Patient consent is captured in an HL7 Clinical Document Architecture (CDA) document using a scanned document (wet signature on paper) or a digital signature.

HL7 is working on a set of specifications for the creation and use of privacy consent directives.

The Nationwide Health Information Network (NHIN) Access Consent Policies Specification is based on the eXtended Access Control Markup Language (XACML), an OASIS standard.

Health Record Banks (HRBs)

An emerging pattern is to include a Health Record Bank (HRB) containing Personal Health Records (PHRs) as a participating node in the HIE. The HRB is accessible to patients via a web portal and allows patients to exercise fine-grained control over the exchange of their health records within the HIE through a consent configuration interface.

UPDATE: On Thursday, July 8, 2010, the Department of Health and Human Services (HHS) announced proposed modifications to the HIPAA Privacy & Security Rules.

Health Information Exchanges (HIEs): Emerging Architectural Patterns

2010-06-22T17:23:00.012-04:00

The following are some emerging architectural patterns in the nascent field of HIEs:

Decentralized and Hybrid Models utilizing a service-oriented architecture (SOA). A centralized registry stores metadata about the type and location of clinical data available in edge systems connected to the HIE. For privacy and security reasons, the clinical data itself is kept at its source as opposed to a centralized repository. Upon request, a Record Locator Service (RLS) finds the data in edge systems and securely routes the data to authorized requestors.
A Data Use and Reciprocal Support Agreement (DURSA) provides the legal framework for participation in the HIE.
Use of a master patient index (MPI) as a core infrastructure for patient matching. The MPI stores identifying information on all patients with records in participating systems.
Use of Integrating the Health Enterprise (IHE) profiles such as Patient Identity Cross-Reference (PIX) and Cross Enterprise Document Exchange (XSD.b) to facilitate patient discovery and the query and retrieval of clinical documents.
Health Information Event Messaging to provide the ability to subscribe to health information.
Interoperability with NHIN Exchange to enable exchange of data with other state and federal agencies such as the Department of Veterans Affairs (VA) and the Department of Defense (DoD). This requires support for NHIN messaging standards such as the Web Services Interoperability (WS-I) Profiles WS-I Basic v 2.0 and WS-I Security v 1.1. WS-I Basic specifies the use of SOAP 1.2, WSDL 1.1, WS-Addressing 1.0, WS-Policy 1.5 , MTOM 1.0 , and UDDI v3.0.2 for the NHIN Web Services Registry. WS-I Security defines the security standards for NHIN Exchange including TLS, AES 128, X.509, XML D-Sig, and Attachment Security. NHIN has adopted an authentication and authorization framework based on SAML 2.0.
Use of the HL7 Continuity of Care Documents (CCD) as the data exchange standards for clinical documents. Meaningful Use criteria allow the ASTM CCR specification as well.
Health Record Banks (HRBs) containing Personal Health Records (PHRs) as participating nodes in the HIE. The HRBs allow patients to exercise control over their health records by granting permissions to specific providers to view those health records.
Ability to connect to the HIE through a local EMR or a web-based portal (for example to allow access for physicians without an EMR).
For simple and secure interoperability, the NHIN Direct draft proposal at the time of this writing is to use:
- SMTP as a backbone protocol
- S/MIME-signed and encrypted messages for security
- IHE XDM for content and metadata packaging
- IHE XDR, REST, and Email (POP/IMAP) as edge protocols
- TLS (with a server certificate only) for on-the-wire security
- XDR as the backbone for NHIN Exchange.

Putting XQuery to Work in Healthcare

2010-06-12T23:02:00.005-04:00

The following are some of the challenges that healthcare organizations will be facing during the next few years:

Conversion from HIPAA 4010 to 5010
Conversion from ICD-9 to ICD-10
Efficiently storing, querying, processing, and exchanging Electronic Health Records (EHRs)
Mapping from HL7 2.x to HL7 v3 messages
Assembling EHRs by aggregating data from multiple organizations participating in Health Information Exchanges (HIEs).

XQuery is not just a query language for XML data sources. It is also a very powerful declarative, strongly typed, and side-effect free programming language for processing and manipulating XML documents. XQuery is a natural solution for querying and aggregating data coming from heterogeneous sources such as relational databases, native XML databases, file systems, and legacy data formats such as EDI. Some developers will find XQuery easier to use than XSLT because XQuery has a SQL-like syntax.

Migration to HIPAA 5010 and ICD 10

Conversion from HIPAA 4010 to 5010 and ICD-9 to ICD-10 will be a priority on the agenda in the next three years (details on final compliance dates can be found on this HHS web page).

The XQuery and XQuery Update Facility specifications provide a simple and elegant solution to this conversion challenge.

Health Information Exchanges (HIEs) and the Virtual Health Record

In a HIE with multiple participating organizations, EHR data must be assembled either through a centralized, federated, or hybrid data model. The data needed to assemble a longitudinal EHR (a virtual health record) for a patient could be coming from several providers, payers, lab companies, and medical devices. XQuery was designed to handle that type of XML processing use case.

Storing, Updating, and Querying EHRs

The HL7 CCD and ASTM CCR have been retained as Meaningful Use XML data exchange standards for EHRs. Mapping between an XML HL7 CCD representation (which is derived from the HL7 UML-based Reference Information Model or RIM) and an existing relational database structure is not trivial. IBM has been granted a patent entitled "Conversion of hierarchically-structured HL7 specifications to relational databases". The HL7 RIMBAA project provides some best practices on mapping RIM objects to a relational database structure.

With the emergence of native XML databases such as Oracle XML DB and IBM pureXML, XML is no longer just a messaging format. It can be used as a format for storing and querying data as well.

This article shows a sample code of updating an EHR stored in HL7 CDA format in an IBM DB2 pureXML native XML database.

Mapping from HL7 2.x to HL7 v3 messages

In countries like Canada where HL7 v3 has been adopted, a frequent challenge is to map from legacy HL7 2.x messages to HL7 v3 messages for example for lab results. An XQuery-based transform can be used to map from an HL7 v2.x XML structure to an HL7 v3 XML structure.

An Alternative to GELLO?

In a previous post entitled "Clinical Decision Support: Crossing the Chasm", I argued that Clinical Decision Support Systems (CDSS) implementers should be free to use any programming language of their choice. GELLO is an HL7 standard which specifies an expression and query language for CDSS. The following are the requirements for a CDSS expression and query language as specified in the GELLO specification:

vendor-independent
platform-independent
object-oriented and compatible with the vMR
easy to read/write
side-effect free
flexible
extensible

XQuery satisfies all these requirements except the third. XQuery is a functional programming language with no side effect as opposed to an object-oriented programming language. GELLO settled on the OMG Object Constraint Language (OCL). The following paragraph from the GELLO specification explains why XQuery (known as XQL at the time) wasn't selected:

XQL is a query language designed specifically for XML documents. XML documents are unordered, labeled trees, with nodes representing the document entity, elements, attributes, processing instructions and comments. The implied data model behind XML neither matches that of a relational data model nor that of an object-oriented data model. XQL is a query language for XML in the same sense as SQL is a query language for relational tables. Since the HL7 RIM data model and the vMR data model are both object-oriented, it is clear that XQL is not an appropriate approach for an object-oriented query and expression language.

That might have been true back in 2004 in an object-oriented world. Today, if the inputs to a CDSS are EHRs represented in HL7 CCD or ASTM CCR format, and those EHRs are stored in an XQuery compliant native XML database, then XQuery could be a strong candidate for an expression and query language for the CDSS.

Data Modeling for Electronic Health Records (EHR) Systems

2010-06-09T00:27:00.008-04:00

Getting the data model right is of paramount importance for an Electronic Health Records (EHR) system. The factors that drive the data model include but are not limited to:

Patient safety
Support for clinical workflows
Different uses of the data such as input to clinical decision support systems
Reporting and analytics
Regulatory requirements such as Meaningful Use criteria.

Model First

Proven methodologies like contract-first web service design and model driven development (MDD) put the emphasis on deriving application code from the data model and not the other way around. Thousands of line of code can be auto-generated from the model, so it's important to get the model right.

Requirements Gathering

The objective here is to determine the entities, their attributes, and the relationships between those entities. For example, what are the attributes that are necessary to describe a patient's condition and how do you express the fact that a condition is a manifestation of an allergy? The data modeler should work closely with clinicians to gather those requirements. Industry standards should be leveraged as well. For example, HITSP C32 defines the data elements for each EHR data module such as conditions, medications, allergies, and lab results. These data elements are then mapped to the HL7 Continuity of Care Document (CCD) XML schema.

The HL7 CCD is itself derived from the HL7 Reference Information Model (RIM). The latter is expressed as a set of UML class diagrams and is the foundation model for health care and clinical data. A simpler alternative to the CCD is the ASTM Continuity of Care Records (CCR). Both the CCD and CCR provide an XML schema for data exchange and are Meaningful Use criteria. Another relevant data model is the HL7 vMR (Virtual Medical Record) which aims to define a data model for the input and output of Clinical Decision Support Systems (CDSS).

These standards can be cumbersome to use as such from a software development perspective. Nonetheless, they can inform the design of the data model for an EHR system. Alignment with the CCD and CCR will facilitate data exchange with other providers and organizations. The following are Meaningful Use criteria for data exchange:

Electronically receive a patient summary record, from other providers and organizations including, at a minimum, diagnostic test results, problem list, medication list, medication allergy list, immunizations, and procedures and upon receipt of a patient summary record formatted in an alternative standard specified in Table 2A row 1, displaying it in human readable format.
Enable a user to electronically transmit a patient summary record to other providers and organizations including, at a minimum, diagnostic test results, problem list, medication list, medication allergy list, immunizations, and procedures in accordance with the standards specified in Table 2A row 1.

Applying Data Modeling Patterns

Applying data modeling patterns allows model consistency and quality. Relational data modeling is a well established discipline. My favorite resource for relational data modeling patterns is: The Data Model Resource Book, Vol. 3: Universal Patterns for Data Modeling.

Some XML Schema best practices can be found here.

Data Stores

Today, options for data store are no longer limited to relational databases. Alternatives include: native XML databases (e.g. DB2 pureXML), Entity-Attribute-Value with Classes and Relationships (EAV/CR), and Resource Description Framework (RDF) stores.

Native XML databases are more resilient to schema changes and do not require handling the impedance mismatch between XML documents, Java objects, and relational tables which can introduce design complexity, performance, and maintainability issues.

Storing EHRs in an RDF store can enable the inference of medical facts based on existing explicit medical facts. Such inferences can be driven by an ontology expressed in OWL or a set of rules expressed in a rule language such SWRL. Semantic Web technologies can also be helpful in checking the consistency of a model, data and knowledge integration across domains (e.g. the genomics and clinical domains), and for managing classification schemes like medical terminologies. RDF, OWL, and SWRL have been successfully implemented in Clinical Decision Support Systems (CDSS).

The data modeling notation used should be independent of the storage model or at least compatible with the latter. For example, if native XML storage is used, then a relational modeling notation might not be appropriate. In general, UML provides the right level of abstraction for implementation-agnostic modeling.

Due Diligence

When adopting a "noSQL" storage model, it is important to ensure that (a) the database can meet performance and scalability criteria and (b) the team has the skills to develop and maintain the database. Due diligence should be performed through benchmarking using a tool such as the IBM Transaction Processing over XML (TpoX). The team might need formal training in a new query language like XQuery or SPARQL.

A Longitudinal View of the Patient Health

Maintaining an up-to-date and truly longitudinal view of a patient's medical history requires merging and reconciling data from heterogeneous sources including providers' EMR systems, lab companies, medical devices, and payers' claim transaction repositories. The data model should facilitate the assembly of data from such diverse sources. XML tools based on XSLT, XQuery, or XQuery Update can be used to automate the merging.

The Importance of Data Validation

Data validation can be performed at the database layer, the application layer, and the UI layer. The data model should support the validation of the data. The following are examples of techniques that can be used for data validation:

XML Schema for structural validation of XML documents
ISO Schematron (based on XPath 2.0 and XSLT 2.0) for business rules validation of XML documents
A business rules engine like Drools
A data processing framework like Smooks
The validation features of a UI framework such as JSF2
The built-in validation features of the database.

The Future: Modeling with the NIEM IEPD

The HHS ONC issued an RFP for using the National Information Exchange Model (NIEM) Information Exchange Package Documentation (IEPD) process for healthcare data exchange. The ONC will release a NIEM Concept of Operations (ConOps). The NIEM IEPD process is explained here.

Health IT Usability: What Can Be Learned From the British NHS

2010-05-31T13:13:00.006-04:00

What can possibly go wrong when the user interface of a health IT system is inadequate. The following are examples of patient safety incidents that can occur when using an electronic medical record (EMR), a Computerized Physician Order Entry (CPOE), or a clinical decision support (CDS) system:

Incorrect medication dosage due to a miscalculation of body weight resulting from a mix-up of kilograms and pounds.
Missing data or incorrect data entry into a patient's medical record.
Misreading of patient's medical record due to a font size that is too small on a computer screen.
Errors in medication dosing frequency due to a non-intuitive menu.
Incorrect test ordered due to a clinician clicking on the wrong test.
Incorrect identification of a patient resulting in the administration of the wrong drug.

These patient safety concerns prompted the US Agency for Healthcare Research and Quality (AHRQ) to order a study entitled: Electronic Health Record Usability Vendor Practices and Perspectives. The report noted the lack of standards for the design, testing, and monitoring of the usability of EHR products. The following are some of the recommendations made in the report:

Encourage formal usability testing early in the design and development phase as a best practice, and discourage dependence on post-deployment review supporting usability assessments

Increase research and development of best practices supporting designing for patient safety

Design certification programs for EHR usability in a way that focuses on objective and important aspects of system usability

Support an independent body for vendor collaboration and standards development to overcome market forces that discourage collaboration, development of best practices, and standards harmonization in this area.

Since November 2004, the British National Health Service (NHS) Common User Interface (CUI) Program has been creating standards and guidance in support of the usability of clinical applications with inputs from user interface design specialists, usability experts, and hundreds of clinicians with a diversity of background in using health information technology.

The program is based on a rigorous development process which includes: research, design, prototyping, review, usability testing, and patient safety assessment by clinicians. The NHS CUI standards and guidance have been tested in production through the NHS Summary Record Application (SRA) project. The Guidance Catalog currently covers the following subjects with high impact on patient safety:

Patient Identification
Consistent Navigation
Accessibility
Abbreviations and Acronyms
Clinical Noting and Terminology
Decision Support
Handover
Information Entry and Display
Medications Management.

The NHS CUI Program is being developed in collaboration with Health IT vendors. The vendors provide feedback to the NHS on the feasibility of the proposed guidelines. The usability standards and guidance inform the development of functional and user interface specifications of vendors' health IT systems.

The benefits are:

Increased patient safety
Increased productivity of healthcare staff through a user-friendly, predictable, and intuitive interface
Reduced need for staff training due to a consistent look and feel across clinical applications.

The US National Institute of Standards and Technology (NIST) will be hosting a workshop on July 13 in Gaithersburg, Maryland entitled Usability in Health IT: Strategy, Research, and Implementation.

Architecting the Health Enterprise with TOGAF 9

2010-05-25T23:31:00.014-04:00

Several factors are currently driving the increased complexity of health information technology (HIT). These factors include: a new regulatory framework, innovations in the practice of healthcare delivery, standardization, cross-enterprise integration, usability, mobility, security, privacy, and the imperative to improve care quality and reduce costs.

A methodology and governance framework is needed for creating a coherent and consistent enterprise architecture (EA). The latter should not be driven by vendors and their offerings. Instead, health enterprises should develop an EA that is aligned with their unique overarching business context, drivers, and vision. Developing an architecture capability that is based on a proven framework should be a top priority for health IT leaders.

TOGAF 9 is an Open Group standard that defines a methodology, standardized semantics, and processes that can be used by Enterprise Architects to align IT with the strategic goals of their organization. TOGAF 9 covers the following four architecture domains:

Business Architecture
Data Architecture
Application Architecture
Technology Architecture.

The diagram below from the TOGAF 9 documentation provides an overview (click on the image to enlarge).

The Architecture Development Method (ADM) is the core of TOGAF and describes a method for developing an enterprise architecture. TOGAF 9 includes specific guidance on how the ADM can be applied to service-oriented architecture (SOA) and enterprise security (two areas of interest in health IT). The different phases of the ADM are depicted on the following diagram (click on the image to enlarge).

The Architecture Capability Framework provides guidelines and resources for establishing an architecture capability within the enterprise. This capability operates the ADM. The Content Framework specifies the artifacts and deliverables for each Architecture Building Block (ABB). These artifacts are stored in a repository and classified according to the Enterprise Continuum.

The Open Group has been working on the adoption of an open EA modeling standard called ArchiMate. ArchiMate provides a higher level view of EA when compared to modeling standards such as BPMN and UML. It can be used to depict different layers of EA including business processes, applications, and technology in a way that can be consumed by non-technical business stakeholders. A sample of an ArchiMate enterprise view of a hospital can be found here.

HL7 has published the Services-Aware Interoperability Framework (SAIF), an architectural framework for facilitating interoperability between healthcare systems. SAIF includes the following four components: the Enterprise Conformance and Compliance Framework (ECCF), the Governance Framework (GF), the Behavioral Framework (BF), and the Information Framework (IF).

For guidance on using SOA in healthcare, the Healthcare Services Specification Project (HSSP) has published the Practical Guide for SOA in Healthcare based on the TOGAF Architecture Development Method (ADM) and the SAIF ECCF. The Practical Guide for SOA in Healthcare contains a sample Reference Enterprise Architecture. The Practical Guide for SOA in Healthcare Volume II describes an immunization case study.

Also noteworthy is the HL7 EHR System Functional Model (EHR-S FM) and the HSSP Electronic Health Record (EHR) System Design Reference Model (EHR SD RM).

Clinical Decision Support: Crossing The Chasm

2010-05-22T01:14:00.016-04:00

Clinical Decision Support (CDS) is certainly a "meaningful use" of electronic health records (EHRs). Despite its potential to improve the quality of care, CDS is not widely used in health care delivery today. In tech marketing parlance, CDS has not crossed the chasm. There are several issues that need to be addressed including: physicians buy-in into the concept of automated execution of evidence-based clinical guidelines, seamless integration into clinical workflows, usability, standardization, and CDS software architecture and design.

When I first started to explore CDS systems, I was quite overwhelmed by the number of different competing formalisms, standards, and academic projects in the field. To move us past the current gridlock in CDS adoption, I propose an agile approach to the development of CDS software with an emphasis on:

Working CDS software that delivers results for providers and their patients
Close daily collaboration between clinicians and software developers during the development process
The use of agile techniques such as automated acceptance testing to facilitate the involvement of clinicians in CDS software quality assurance.

Working CDS Software

Different formalisms, methodologies, and architectures have been proposed for representing the medical knowledge in clinical guidelines for their automated execution. Examples include, but are not limited to the following:

The Arden Syntax
GLIF (Guideline Interchange Format)
GELLO (Guideline Expression Language Object-Oriented)
GEM (Guidelines Element Model)
PROforma
EON
PRODIGY
Asbru
GUIDE
SAGE
SEBASTIAN
The HL7 Decision Support Service (DSS) Functional Model Specification.

Many academic papers have been written to explain and compare these different approaches. Each of these projects represents an important contribution to the CDS field and will inform the design of future CDS software. However, it is quite easy to fall into analysis paralysis when reviewing and debating which formalism or standard is better. At the end of the day, what really matters to the pragmatic developer is tested and working CDS software that delivers results for clinicians and their patients.

Using business rule engines is not the only way to develop CDS software. However, I believe that because they are written in languages and frameworks that are accessible to mainstream software developers, business rule engines can accelerate the development, deployment, and availability of CDS software. Furthermore, many viable open source business rule engines are available today and can be leveraged.

Getting CDS Done with a Business Rule Engine

ARRA interim certification criteria for electronic health record (EHR) technology include the following CDS-related requirements:

Implement automated, electronic clinical decision support rules (in addition to drug-drug and drug-allergy contraindication checking) according to speciality or clinical priorities that use demographic data, specific patient diagnoses, conditions, diagnostic test results and/or patient medication list.
Automatically and electronically generate and indicate (e.g., pop-up message or sound) in real time, alerts and care suggestions based upon clinical decision support rules and evidence grade.
Automatically and electronically track, record, and generate reports on the number of alerts responded to by a user.

These requirements can be satisfied with simple conditional statements in any programming language. However, it is recognized that clinical decision support rules (like other complex types of business rules) are better implemented with a business rule engine. This allows the developer to externalize the medical knowledge in the clinical guideline in the form of declarative rules as opposed to embedding that knowledge in procedural code. This approach has many benefits notably resilience to change, ease of maintenance, and enabling collaboration with business users (in this case clinicians).

Collaboration between Clinicians and Software Developers

One of the most challenging aspects of CDS has been the translation of the medical knowledge in clinical guidelines into executable code. There are not that many people who are expert in the both the medical and software development fields. The Agile prescription to this problem is close and daily collaboration between clinicians and software developers.

Business Rules engines like JBoss Drools provide features such as Excel-based decision tables or the ability to write rules in a DSL (domain specific language) to allow clinicians to actively participate in the development and maintenance of decision support rules.

Automated Acceptance Testing

The quality of CDS software is of paramount importance for care safety reasons. The Agile prescription here is test-driven development (TDD), particularly the automated integration and acceptance testing of the proper execution of clinical decision support rules. "FIT for Rules" is an example of an automated acceptance framework for rule engines like ILog and Drools. Such frameworks allow both the developer and the clinician to participate in the acceptance testing process.

Service-Oriented CDS

The complexity and cost of developing CDS software strongly argue in favor of a service-oriented approach whereby CDS software capabilities are exposed as a set of services that can be consumed by other client health IT systems such as EHR and Computerized Physician Order Entry (CPOE) systems. To reduce costs, these CDS software services can be shared by several health care providers.

In this regard, the HL7 Decision Support Service (DSS) Functional Model Specification represents one of the most important specifications for CDS implementers today.

Interchange Standard

The complexity and cost inherent in capturing the medical knowledge in clinical guidelines and translating that knowledge into executable code remains an impediment to the widespread adoption of CDS software. Therefore, there is still a need for a standard for the sharing and interchange of executable clinical guidelines. Several formalisms and standards have been proposed such as the Arden Syntax, GLIF, GELLO, and GEM. However, none of these standards has been widely adopted. Although there is a lot that can be learned from these standards, I believe that they are not widely used because they are complex and specific to the healthcare domain, and therefore not accessible to mainstream developers. There is also a lack of open source and even commercial implementations of some of these standards.

If business rule engines are the pragmatic path to CDS adoption, then I would argue that the Rule Interchange Format (RIF) specification might be a solution to the interchange problem. The RIF Production Rule Dialect (PRD) is designed as a common XML serialization for multiple rule languages to enable rule interchange between different business rule management systems (BRMS). RIF is currently a W3C candidate recommendation and is backed by several BRMS vendors.

UPDATE: The following is the Final Meaningful Use criteria for Clinical Decision Support (CDS):

Implement rules. Implement automated, electronic clinical decision support rules (in addition to drug-drug and drug-allergy contraindication checking) based on the data elements included in: problem list; medication list; demographics; and laboratory test results.

Notifications. Automatically and electronically generate and indicate in real-time, notifications and care suggestions based upon clinical decision support rules.

And here are the draft stage 2 requirements:

Use CDS to improve performance on high- priority health conditions.

Establish CDS attributes for purposes of certification:

Authenticated (source cited);
Credible, evidence-based;
Patient-context sensitive;
Invokes relevant knowledge;
Timely;
Efficient workflow;
Integrated with EHR;
Presented to the appropriate party who can take action

How Health Plans Can Help Jumpstart EHR Adoption

2010-05-11T23:33:00.015-04:00

ARRA incentives will help physicians and hospitals adopt computerized medical records software in the next four years. The next big challenge will be how to obtain patients' electronic health records (EHRs) in order to make a "Meaningful Use" of these investments in health information technology (HIT). Most patients don't currently have an EHR. The goal is to ensure that within four years, all patients in the US have an EHR.

It will take some time before physicians and hospitals have comprehensive medical records for their patients in their newly acquired Electronic Medical Record (EMR) software. EMR systems that are already deployed by providers don't typically contain clinical data about the care delivered by other providers. In contrast to EMRs, EHRs contain a longitudinal view of a patient's medical history across providers and can be populated from other sources including lab companies, at-home medical devices, health plans, and patient self-reported data (for example, data about over-the-counter drug that could potentially generate an adverse drug reaction).

Discussions on Meaningful Use don't usually mention health plans since the incentives are only directed at eligible professionals and hospitals. However, health plans can play a very important role in jumpstarting EHR adoption by extracting health records from the vast amount of clinical data in claim transaction repositories and by making that data available to members and providers as well.

Quality and Care Coordination with Claim-Based Health Records

The following examples demonstrate how health records extracted from claim data, when made available to providers, can significantly improve the quality and coordination of care :

Pharmacy claim data can enable medication reconciliation during transitions of care.
Lab and radiology claim data help avoid redundant and potentially costly orders.
Diagnosis codes in medical claims can provide a view of a patient's problem list for care coordination purposes.

When delivered in standard compliant data format, these claim-based data can be fed into clinical decision support systems for the automated execution of clinical guidelines and for generating alerts and reminders.

Data Integration and the Importance of Data Standards

Maintaining an up-to-date and truly longitudinal view of a patient's medical history requires merging and reconciling data from heterogeneous sources including providers' EMR systems, lab companies, medical devices, and payers' claim transaction repositories. This data integration will only be possible if all stakeholders adhere to common data exchange standards such as the ASTM Continuity of Care Records (CCR) and the HL7 Continuity of Care Documents (CCD) standards.

Both the CCR and the CCD specify an XML Schema for constraining and validating EHR data. Therefore XML tools based on XSLT or XQuery/XQuery Update can be used to automate the merging of data from multiple sources.

To ensure data portability when members switch jobs or health plans, the HL7 standard development organization published a Plan-to-Plan Personal Health Records Implementation Guide of the CCD specification with active participation by America's Health Insurance Plans (AHIP) and the Blue Cross and Blue Shield Association (BCBSA).

Health Plans and HIEs

Collaboration between all stakeholders in the healthcare industry requires an IT infrastructure to ensure effective data exchange orchestration and security. This infrastructure can take the form of a Health Information Exchange (HIE) network at the regional, state, or even federal level. A HIE provides capabilities such as a document registry and repository, an authorization policy engine, a consumer preferences manager, a HIPAA-compliant audit log, and a master patient index (MPI) to reliably aggregate patient's data across payers, providers, and other data sources. Such exchange networks can be built with open source software such as NHIN Connect and the emerging NHIN Direct.

The Value of Claim-Based Health Records for Health Plans

Claim-based health records can help providers meet health plans' pay-for-performance incentives and quality measures initiatives. Electronic Health records will also be an important component of health plans' Population Health Management programs.

Some Accountable Care Organizations (ACOs) will be led by health plans. Data integration will be essential to the success of those ACOs.

Health Plans vs. Google Health and Microsoft Health Vault

Health Plans have a huge competitive advantage compared to online personal health records (PHRs) services such as Google Health and Microsoft Health Vault: they can pre-populate the PHRs of their members with health records extracted from claim transactions and those PHRs are likely to be more useful and reliable to providers than patient self-reported data.

A Roadmap for Health Plans

Health plans can position themselves for their new role as enablers of EHR adoption by doing the following:

Create an EHR view of the health records in claim transaction repositories
Map the data to industry exchange standards such as the ASTM CCR or the HL7 CCD
Make the data available to their members (through a web portal for example) and partners (subject to patient's consent)
Get ready to contribute data to HIEs by adopting other necessary standards such as Integrating the Health Enterprise (IHE) profiles for cross-enterprise document exchange (XDS.b), security, and patient consent
Embrace and participate in open source projects such as NwHIN Connect and Direct
Actively participate in the health IT standards development process
Prepare their infrastructure to send and receive health records from providers.

Green EHRs, Genomics, Decision Support, and Checklists

2010-04-11T14:49:00.007-04:00

I am back in the blogosphere after a three month hiatus. As you may have noticed from my last post, my focus lately has been on health information technology and how to unleash its potential to improve the quality of care and reduce costs. Healthcare Reform has been signed into law in the US as I have predicted in my last post. I believe that the healthcare sector is going through what former Intel CEO Andy Grove calls a "Strategic Inflection Point" driven by a combination of the following:

A new regulatory framework including Healthcare Reform, ARRA and the HITECH Act
Technology: electronic health records, advances in Genomics, and leveraging the ubiquity and ease of use of Web 2.0/3.0 platforms
Innovations in the practice of healthcare delivery such as personalized medicine, automated execution of clinical guidelines (evidence-based medicine and clinical decision support), and the growing awareness of the effectiveness of checklists in high-pressure clinical settings such as surgery and intensive care.

Developing robust and easy to use health IT standards is key, so I've joined the HL7 standard development organization to follow and hopefully contribute. Specifically, I am interested in the following HL7 working groups:

The HL7 Structured Document Working Group and the Greening of the CDA

This group develops structured document standards for healthcare such as the Clinical Document Architecture (CDA) and the Continuity of Care Document (CCD). I am particularly interested in the greenCDA effort to simplify the CDA. I believe that simplifying the CDA can greatly facilitate and accelerate adoption. A simpler version of the CDA will allow adopters to quickly learn it and use their familiar software development tools (such as XML databinding tools and IDEs) to write software applications. To achieve that objective, data components of the greenCDA will be based on data semantics as opposed to the CDA R-MIM (Refined Message Information Model) classes. The CDA R-MIM is a graphical representation of the CDA specification and a step in the HL7 V3 Reference Information Model (RIM) refinement process.

HITSP C32 took the right approach by first defining required data elements for various content modules such as conditions and medications and then mapping those data elements to the HL7 CCD XML Schema structure. However the latter is quite complex. Fortunately, it is possible to map those HITSP C32 data elements to a simpler XML Schema than what is defined by the HL7 CCD. That simpler XML Schema can take different forms. A National Information Exchange Model (NIEM) Information Exchange Package Documentation (IEPD) for healthcare data is an approach that is being considered by the ONC. The NIEM approach is proven and has been successfully used in complex data exchange scenarios in criminal justice and other sensitive domains. Another approach is to create a lighter version of the HITSP C32 specification like the L32 proposed by the MITRE corporation.

There are many new interesting features of the XML Schema 1.1 specification such as assertions and conditional type assignments that can be leveraged as well to consolidate both structural and business rules data constraints in a single schema.

While there are different approaches to simplifying the CDA, instances of any well-designed simplified version of the CDA/CCD should easily be mapped to the original plain vanilla CDA/CCD schemas for exchange.

The Clinical Decision Support Working Group

This group develops standards such as the Virtual Medical Record (vMR), GELLO (an expression and query language for computer-interpretable clinical guidelines), Infobutton (a context-aware information retrieval standard specification), and Clinical Decision Support Services (CDSS).

Clinical Decision Support systems had a limited adoption but are likely to get a boost from the new ARRA EHR certification requirements.

The Clinical Genomics Data Working Group

This group develops standards for clinical and personalized genomic data such as DNA sequence variations and gene expression levels. Combined with clinical decision support services, this work has the potential to revolutionize personalized medicine as we know it. For example, a medication dose can be adjusted based on a patient's genotype. Family history and genetic screening will play an important role in preventive medicine for diseases such as diabetes and cancer.

Procedures and Checklists in the Cockpit and Intensive Care Units

The Flight Operation Interests Group (FOIG) of the Air Transport Association (ATA) of which I am a member is developing a data model and XML Schema for flight deck procedures and checklists. Adherence to well designed flight deck procedures and checklists can contribute significantly to the improvement of flight safety. For example, several no-flap/no-slat takeoff crashes have been caused by a deviation from basic operating procedures.

Aircraft manufacturers have different philosophies and policies on the design of flight deck procedures that must be reconciled. Another challenge is the need to take human factors into consideration in any effort related to the design and execution of flight deck procedures.

While the idea of using checklists and standard operating procedures has been fully embraced and adopted by aviation professionals for the last 70 years, it is only now making inroads into the field of medicine particularly in intensive care units. The use of checklists in medicine has already shown the potential to save patients live and reduce human errors. However, the main challenge remains acceptance and full embrace of checklists by clinicians. If you are interested in learning more about checklists in medicine, I recommend the book "The Checklist Manifesto: How to Get Things Right" by Atul Gawande.

A checklist could take the form of a tasks list automatically generated as output of clinical decision support services that take as inputs electronic health records as well as patient genomic data for a more effective form of personalized medicine.

Unlocking the Potential of Health Information Technology (HIT)

2009-12-27T10:29:00.022-05:00

Around the world, HIT is widely seen as a key factor in containing rising health care costs and improving the quality of patient care. President Obama, in a speech at George Mason University in January 2009, declared:

To improve the quality of our health care while lowering its cost, we will make the immediate investments necessary to ensure that, within five years, all of America's medical records are computerized.

Yet, recent studies have raised some concerns about the effectiveness of Electronic Health Records (EHRs). A recent survey of 4000 hospitals conducted by Dr. David Himmelstein and his team of researchers at Harvard Medical School and published in the American Journal of Medicine concluded that the adoption of EHRs have had little impact on administrative costs and quality of care.

Another study published in the Milbank Quaterly raised some questions about the effectiveness of EHRs (as compared to paper records) in primary clinical work.

In the US, the American Recovery and Reinvestment Act (ARRA) of 2009 (also known as the "economic stimulus package") contains a portion called the Health Information Technology for Economic and Clinical Health Act (HITECH Act) which provides $19.2 billions of incentives to physicians and hospitals for the adoption of EHRs. No matter where you stand in the political spectrum, ARRA, the HITECH Act, and Health Care Reform are now realities to contend with in the US health care sector.

What can be done to ensure that these massive investments in HIT and EHRs live up to their expectations? A research paper entitled "Use of Electronic Health Records in US Hospitals" and published in April 2009 in The New England Journal of Medicine reveals:

On the basis of responses from 63.1% of hospitals surveyed, only 1.5% of U.S. hospitals have a comprehensive electronic-records system (i.e., present in all clinical units), and an additional 7.6% have a basic system (i.e., present in at least one clinical unit). Computerized provider-order entry for medications has been implemented in only 17% of hospitals.

The EHR is the cornerstone of the emerging Nationwide Health Information Network (NHIN). Software developers can do all sorts of interesting things (in the spirit of "meaningful use") with patient health data once it becomes available in electronic form. In 2010, EHRs will be a top priority for providers because of the incentives available under the HITECH Act. The ECRI Institute puts EHRs on the number two spot on its list of Top 7 Technologies for Health Plans in 2010.

The following factors will be key to unlocking the full potential of HIT:

Standardization
Open Source
Training the HIT Workforce

Standardization

Standards are important for the seamless exchange of data between the different stakeholders (hospitals, physicians, payers, lab companies, etc.) in the health care industry. Furthermore, due to the complexity of the health care domain, organizations adopting HIT cannot afford to reinvent the wheel when deciding on key issues such as:

A data model for representing EHR data and quality measure reporting content.
Coding systems for representing patient health information in a machine-processable format.
A language for expressing clinical practice guidelines for their automated execution in clinical decision support systems (CDSS) as a "meaningful use" of EHRs.
A protocol for cross-organization exchange of patient data.
Technologies for privacy, security, audit trails, and patient consents.

Fortunately, some of these standards exist today and should be adopted. Organizations such as the Health Information Technology Standards Panel (HITSP) and Integrating the Health Enterprise (IHE) in the US and Canada's Health Infoway's Standards Collaborative are working on harmonizing standards for data exchange as well as security and privacy in the health care domain.

EHR Data and Quality Reporting

The HL7 Reference Information Model (RIM) expressed as a set of UML class diagrams is the foundation model for health care and clinical data. The HL7 Clinical Document Architecture (CDA) which is derived from the HL7 RIM has been widely adopted around the world for EHR implementation projects. The HL7 Continuity of Care Documents (CCD) is a constraint on the CDA driven by the requirements of the ATSM Continuity of Care Record (CCR) specification. The HITSP C32 specification which is a further constraint on the HL7 CCD is emerging as the standard for the exchange of EHR data in the US realm.

On the CCR vs. CCD debate, I have come to appreciate and respect the exhaustive metadata provided by the CCD which is important for medico-legal reasons, but also for building software applications such as clinical decision support systems (CDSS) that rely heavily on the availability of such metadata. However, I also realize that some patient-facing web applications for managing personal health records (PHRs) can benefit from the simplicity of the CCR.

For collecting and reporting performance measure data to improve the quality of care, the Physician Quality Reporting Initiative (PQRI) Registry XML Specification and the HL7 Quality Reporting Document Architecture (QRDA) are available.

Laboratory content can be exchanged in HL7 2.5.1 messages or HITSP CCD documents. Digital Images and Communications in Medecine (DICOM), DICOM Structured Reporting (SR), and Web Access to DICOM objects (WADO) are used for medical imaging.

Medical Terminologies

The HITSP C80 Clinical Document and Message Terminology Component specification defines the vocabularies and terminologies used by various HITSP specifications. HITSP C80 recommends Logical Observation Identifiers Names and Codes (LOINC) for laboratory observations, RxNorm for standard names of clinical drugs, Unique Ingredient Identifier (UNII) for allergies, SNOMED CT for Problem Lists, and Unified Code for Units of Measure (UCUM) for units of measure.

Integrating terminologies is a big challenge in Health IT. An example is the mapping from SNOMED-CT to ICD-10 or Current Procedural Terminology (CPT). Semantic web technologies such as RDF, OWL2, SWRL, SPARQL, and SWASDL have proven their usefulness in semantic mediation across coding systems as well as in building Clinical Decision Support Systems (CDSS).

Administrative Transactions and E-Prescribing

Relevant standards include X12 4010 and 5010 for administrative transactions, Council for Affordable Quality Healthcare (CAQH) Core for online patient eligibility and benefits inquiries, and NCPDP SCRIPT 10.6 for electronic prescriptions (e-prescribing). Conversion from 4010 to 5010 and ICD-9 to ICD-10 will be a priority on the agenda in the next three years (details on final compliance dates can be found on this HHS web page). XQuery, XQuery Update, and XSLT2 are likely to play an important role in that conversion effort.

An Expression and Query Language for Clinical Decision Support Systems (CDSS)

There is currently no consensus on a single standard for expressing clinical practice guidelines for their automated execution in CDSS. Existing specifications include the Arden Syntax, the Guideline Interchange Format (GLIF), and GELLO (roughly Guideline Expression Language Object-Oriented). Some wonder if a standard is needed in this space at all. My position is that a standard with an open source reference implementation with support for some publicly available and approved clinical guidelines can be useful in speeding up adoption.

In addition, since EHR data is one of the primary inputs into a CDSS, the CDSS expression language should at least share the same underlying data model with the EHR data input. That model is the HL7 Reference Information Model (RIM). The "virtual medical record" (or vMR) specified by GELLO is defined as a view of the HL7 RIM. It would be a great help if HITSP could step in and harmonize standards in the CDSS space.

Cross-Enterprise Document Sharing

The Integrating the Health Enterprise (IHE) Cross-Enterprise Document Sharing (XDS) Integration Profile specifies a protocol for the sharing and access to EHRs across health enterprises. XDS defines concepts such as an "Affinity Domain" (a group of collaborating health enterprises), the "Document Source", the "Document Repository", and the "Document Registry". It also specifies transactions such as "Provide and Register Document", "Query Document" and "Retrieve document".

Other relevant IHE Integration Profiles include the Cross Community Access profile (XCA), the Cross Enterprise Document Reliable Interchange (XDR), and the Cross Enterprise Document Media Interchange (XDM).

Privacy and Security

On the privacy and security front, the following publicly available standards can be used to comply with ARRA and HIPAA requirements:

Transport Layer Security (TLS)
Advanced Encryption Standard (AES)
Secure Hash Algorithm (SHA)
OASIS eXtensible Access Control Markup Language (XACML)
OASIS Security Assertion Markup Language (SAML)
OASIS WS-Security
OASIS WS-Trust
OASIS WS-Policy.

The following IHE Profiles provide specific security guidelines for health enterprises and Health Information Exchanges (HIEs):

Audit Trail and Node Authentication (ATNA)
Consistent Time (CT)
Basic Patient Privacy Consents (BPPC)
Enterprise User Authentication (EUA)
Cross-Enterprise User Assertion (XUA)
Patient Demographics Query (PDQ)
Patient Identifier Cross-Referencing (PIX)
Digital Signatures (DSG).

For the de-identification and re-identification of content which are important for privacy as well as public health research and epidemiology, the following specifications are available:

45 CFR Parts 160 and 164. Standards for Privacy of Individually Identifiable Health Information; Final Rule. August 14, 2002. Section 164.514(a-b) Deidentification of protected health information. (Deidentification)
46 CFR Parts 160 and 164. Standards for Privacy of Individually Identifiable Health Information; Final Rule. August 14, 2002. Section 164.514(c) Reidentification (Pseudonymization)
ISO/TS 25237:2008 Health Informatics --Pseudonymisation, Unpublished Technical Specification (Pseudonymization).

Open Source

High costs, total failure, disappointing return on investment (ROI), and adverse impact on end-users productivity are always risk factors to deal with in any large scale software implementation project. HIT projects are no exception. Driving costs out of health care will be a top priority for all stakeholders in 2010. So, companies will look for viable alternatives to insane software licensing and maintenance fees.

Open source implementations of some of the HIT standards mentioned above can be leveraged to reduce such risks. In addition to lowering costs and risk, open source can provide the transparency that is needed to ensure that HIT software provides adequate privacy and security protections. Examples of these tools include:

The Open eHealth Integration Platform which is based on Apache Camel and implements HIT standards such as HL7 2.x, HL7 CDA/CCD, and IHE XDS.
The Open Health Tools Project provides tooling for developing HIT applications.
The Connect Gateway allows health enterprises to easily connect their HIT systems to health information exchanges (HIE). It implements key components such as a Master Patient Index (MPI), XDS.b Document Registry and Repository, Authorization Policy Engine, Consumer Preferences Manager, and a HIPAA-compliant Audit Log.
The WorldVistA EHR open source EHR system which is based on the U.S. Department of Veterans Affairs (VA) sponsored VistA software. Work is underway to add CCR and CCD support to WorldVistA.

Training the HIT workforce

Finally, the potential of HIT will not be fully realized without the availability of a fully trained and competent workforce. All stakeholders in the health care industry should commit the necessary investments to educating the future HIT workforce through college and university programs as well as corporate training.

Lastly, the usability of EHRs in clinical settings deserves more attention and additional research.

Relational, XML, or RDF?

2009-09-21T22:02:00.028-04:00

During the last 15 years, I have had the opportunity to work with different data models and approaches to application development. I started with SGML in the aerospace content management space back in 1995, then saw the potential of XML and fully embraced it in 1998. Since that time, I have been continuously following the evolution of XML related specifications and have been able to leverage the bleeding edge including XForms, XQuery, XSLT2, XProc, ISO Schematron, and even XML Schema 1.1.

However, being a curious person, I decided to explore other approaches to data management and application development. I worked on systems using a relational database backend and application development frameworks like Spring, Hibernate, and JSF. I've been involved in SOAP-based web services projects where XML data (constrained and validated by an XML Schema) was unmarshalled into Java objects, and then persisted into a relational table with an Object-Relational Mapping (ORM) solution such as Hibernate.

I also had the opportunity to work with the Java Content Repository (JCR) model in magazine content publishing, and the Entity-Attribute-Value with Classes and Relationships (EAV/CR) model in the context of medical informatics. EAV/CR is suited for domains where entities can have thousands of frequently changing parameters.

Lately, I have been working on Semantic Web technologies including the RDF data model, OWL (the Web Ontology Language), and the SPARQL query language for RDF.

Clients often ask me which of these approaches is the best or which is the most appropriate for their project. Here is what I think:

Different approaches should be part of the software architect's toolkit (not just one).
To become more productive in an agile environment, every developer should become a "generalizing specialist".
The software architect or developer should be open minded (no "not invented here syndrome" or "what's wrong with what we're doing now" attitude).
The software architect or developer should be willing to learn new technologies outside of their comfort zone and IT leadership should encourage and reward that learning.
Learning new technologies sometimes requires a new way of thinking about the problems at hand and "unlearning" old knowledge.
It is important not to have a purist or religious approach to selecting any particular approach, since each has its own merits.
Ultimately, the overall context of the project will dictate your choice. This includes but is not limited to: skills set, learning curve, application performance, cost, and time to market.

Based on my personal experience, here is what I have learned:

The XPath 2.0 and XQuery 1.0 Data Model (XDM)

The roots of SGML and XML are in content management applications in domains such as law, aerospace, defense, scientific, technical, medical, and scholarly publishing. The XPath 2.0 and XQuery Data Model (XDM) is particularly well suited for companies selling information products directly as a source of revenues (e.g. non-ad based publishers).

XSLT2 facilitates media-independent publishing (single sourcing) to multiple devices and platforms. XSLT2 is also a very powerful XML transformation language that allows these publishers to perform the series of complex transformations that are often required as the content is extracted from various data sources and assembled into a final information product.

With XQuery, sophisticated contextualized database-like queries can be performed. In addition, XQuery allows the dynamic assembly of content where new compound documents can be constructed on the fly from multiple documents and external data sources. This allows publishers to repurpose content into new information products as needed to satisfy new customer demands and market opportunities.

XInclude enables the chunking of content into reusable pieces. XProc is an XML pipeline language that essentially allows you to automate a complex publishing workflow which typically includes many steps such as content assembly, validation, transformation, and query.

The second category of application for which XML is a strong candidate is what is sometimes referred to as an "XML Workflow" application. The typical design pattern here is XRX (XForms, REST, and XQuery) where user inputs are captured with an XForm front end (itself potentially auto-generated from an XML schema) and data is RESTfully submitted to a native XML database, then queried and manipulated with XQuery. The advantages of this approach are:

It is more resilient to schema changes. In fact the data can be stored without a schema.
It does not require handling the impedance mismatch between XML documents, Java objects, and relational tables which can introduce design complexity, performance, and maintainability issues even when using code generation.

A typical example of an "XML Workflow" application would be a Human Resources (HR) form-based application that allows employees to fill and submit a form and also provides reporting capabilities.

The third and last category of application are Web Services (RESTful or SOAP-based) that consume XML data from various sources, store the data natively in an XML database directly (bypassing the XML databinding and ORM layers altogether), and perform all processing and queries on the data using a pure XML approach based on XSLT2 and XQuery. An example is a dashboard or mashup application that stores all of the submitted data in a native XML database. In this scenario, the data can be cached for faster response to web services requests. Again the benefits listed for "XML Workflow" applications apply here as well.

The Relational Model

The relational model is well established and well understood. It is usually an option for data-oriented and enterprise-centric applications that are based on a closed world assumption. In such a scenario, there is usually no need for handling the data in XML and a conventional approach based on JSF, Spring, Hibernate and a relational database backend is enough.

Newer Java EE frameworks like JBoss Seam and its seam-gen code generation tools are particularly well-suited for this kind of task. There is no running away from XML however, since these frameworks use XML for their configuration files. Unfortunately, there is currently a movement away from XML configuration files toward Java annotations due to some developers complaining about "XML Hell".

The relational model supports transactions and is scalable although a new movement called NoSQL is starting to challenge that last assumption. An article entitled "Is the Relational Database Doomed?" on readwriteweb.com describes this emerging trend.

The RDF Data Model

Semantic Web technologies like RDF (an incarnation of the EAV/CR model mentioned above), OWL, SKOS, SWRL, and SPARQL and Linked Data publishing principles have received a lot of attention lately. They are well suited for the following applications:

Applications that need to infer new implicit facts based on existing explicit facts. Such inferences can be driven by an ontology expressed in OWL or a set of rules expressed in a rule language such SWRL.
Applications that need to map concepts across domains such as a trading network where partners use different e-commerce XML vocabularies.
Master Data Management (MDM) applications that provide an RDF view and reasoning capabilities in order to facilitate and enhance the process of defining, managing, and querying an organization's core business entities. A paper on IBM's Semantic Master Data Management (SMDM) project is available here.
Applications that use a taxonomy, a thesaurus, or a similar concept scheme such as online news archives and medical terminologies. SKOS, recently approved as a W3C recommendation was designed for that purpose.
Silo-busting applications that need to link data items to over data items on the web, in order to perform entity correlation or allow users to explore a topic further. The Linked Data design pattern is based on an open world assumption, uses dereferenceable HTTP URIs for identifying and accessing data items, RDF for describing metadada about those items, and semantic links to describe the relationships between those items. An example is an Open Government application that correlates campaign contributions, voting records, census, and location data. Another example is a semantic social application that combines an individual's profiles and social networks from multiple sites in order to support data portability and fully explore the individual's social graph.

Of course, these different approaches are not mutually exclusive. For example, it is possible to provide an RDF view or layer on top of existing XML and relational database applications.

Adding Semantics to SOA

2009-08-11T22:43:00.017-04:00

What can Semantic Web technologies such as RDF, OWL, SKOS, SWRL, and SPARQL bring to Web Services. One of the most difficult challenges of SOA is data model transformation. This problem occurs when services don't share a canonical XML schema. XML transformation languages such as XSLT and XQuery are typically used for data mediation in such circumstances.

While it is relatively easy to write these mappings, the real difficulty lies in mapping concepts across domains. This is particularly important in B2B scenarios involving multiple trading partners. In addition to proprietary data models, it is not uncommon to have multiple competing XML standards in the same vertical. In general, these data interoperability issues can be syntactic, structural, or semantic in nature. Many SOA projects can trace their failure to those data integration issues.

This is where semantic web technologies can add significant value to SOA. The Semantic Annotations for WSDL and XML Schema (SAWSDL) is a W3C recommendation which defines the following extension attributes that can be added to WSDL and XML Schema components:

The modelReference extension attribute associates a WSDL or XML Schema component to a concept in a semantic model such as OWL. The semantic representation is not restricted to OWL (for example it could be an SKOS concept). The modelReference extension attribute is used to annotate XML Schema type definitions, element and attribute declarations as well as WSDL interfaces, operations, and faults.
The liftingSchemaMapping and loweringSchemaMapping extension attributes typically point to an XSLT or XQuery mapping file for transforming between XML instances and ontology instances.

A typical example of how SAWSDL might be used is in an electronic commerce network where trading partners use various standards such as EDI, UBL, ebXML, and RosettaNet. In this case, the modelReference extension attribute can be used to map a WSDL or XML Schema component to a concept in a common foundational ontology such as one based on the Suggested Upper Merged Ontology (SUMO). In addition, lifting and lowering XSLT transforms are attached to XML Schema components in the SAWSDL with liftingSchemaMapping and loweringSchemaMapping extension attributes respectively. Note that any number of those transforms can be associated with a given XML schema component.

Traditionally, when dealing with multiple services (often across organizational boundaries), an Enterprise Services Bus (ESB) provides mediation services such as business process orchestration, business rules processing, data format and data model transformation, message routing, and protocol bridging. Semantic mediation services can be added as a new type of ESB service. The SAWSDL4J API defines an object model that allows SOA developers to access and manipulate SAWSDL annotations.

Ontologies have been developed for some existing e-commerce standards such as EDI X12, RosettaNet, and ebXML. When required, ontology alignment can be achieved with OWL constructs such as subClassOf , equivalentClass , and equivalentProperty.

Semantic annotations provided by SAWSDL can also be leveraged in orchestrating business processes using the business process execution language (BPEL). To facilitate service discovery in SOA Registries and Repositories, interface definitions in WSDL documents can be associated with a service taxonomy defined in SKOS. In addition, once an XML message is lifted to an ontology instance, the data in the message becomes available to Semantic Web tools like OWL and SWRL reasoners and SPARQL query engines.

From Web 2.0 to the Semantic Web: Bridging the Gap in Newsmedia

2009-07-26T17:42:00.004-04:00

In this presentation, I explain the Semantic Web value proposition for the newsmedia industry and propose some concrete steps to bridge the gap.

Welcome to the world of news in the Web 3.0 era.

From Web 2.0 to the Semantic Web: Bridging the Gap in the Newsmedia Industry

View more presentations from Joel Amoussou.

Semantic Social Computing

2009-07-08T21:32:00.005-04:00

The Web 2.0 revolution has produced an explosion in social data that is fundamentally transforming business, politics, culture, and society in general. Using tools such as wikis, blogs, online forums, and social networking sites, users can now express their point of view, build relationships, and exchange ideas and multimedia content. Combined with portable electronic devices such as cameras and cell phones, these tools are enabling the citizen journalist who can report facts and events faster than traditional media outlets and government agencies.

One of the challenges posed by this explosion in social data is data portability between social networking sites. But the next biggest challenge will be the ability to harvest all that social data in order to extract actionable intelligence (e.g. a better understanding of consumer behavior or the events unfolding at a particular location). In addition, in a world where security has become the number one priority, various sensors from traffic cameras to satellite sensors are also collecting huge amounts of data. The integration of sensor data and social data offers new possibilities.

Those are the types of integration challenges that Semantic Web technologies are designed to solve. The SIOC (Semantically Interlinked Online Communities) Core ontology describes the structure and content of online community sites. A comprehensive list of SIOC tools is available at the SIOC Applications page. Using these tools, developers can export SIOC compliant RDF data from various data sources such as blogs, wikis, online forums, and social networking sites such as Twitter and Flickr. Once exported, the SIOC data can be crawled, aggregated, stored, indexed, browsed, and queried (using SPARQL) to answer interesting questions. Natural Language Processing (NLP) techniques can be used to facilitate entity extraction from user generated content.

SIOC leverages the FOAF ontology to describe the social graph on social networking sites. For example, this can offer deeper insights for marketers into how social recommendations affect consumer behavior.

One unique capability offered by Semantic Web technologies is the ability to infer new facts (inference) from explicit facts based on the use of an ontology (RDFS or OWL) or a set of rules expressed in a rule language such as the Semantic Web Rule Language (SWRL). Using constructs such as owl:sameAs or rdfs:seeAlso, it becomes easy to express the fact that two or more different web pages relate to the same resource (e.g. different profile pages of the same person on difference social networking sites). Linked Data principles can help in linking social data, therefore building bridges between the data islands that today's social networking sites represent.

SIOC compliant social data can be meshed up with other data sources such as sensor data to reveal very useful information about events related to logistics, public safety, or political unrest at a particular location for example. With the advent of GPS-enabled cameras and cell phones, temporal and spatial context can be added to better describe those events. The W3C Time OWL Ontology (OWL-Time) and the Basic Geo Vocabulary have been developed for that purpose.

Publishing Government Data to the Linked Open Data (LOD) Cloud

2009-06-11T23:43:00.018-04:00

In a previous post, I outlined a roadmap for migrating news content to the Semantic Web and the Linked Open Data (LOD) cloud. The BBC has been doing some interesting work in that space by using Linked Data principles to connect BBC Programmes and BBC Music to MusicBrainz and DBpedia. SPARQL endpoints are now available for querying the BBC datasets.

It is clear that Europe is ahead of the US and Canada in terms of Semantic Web research and adoption. The Europeans are likely to further extend their lead with the announcement this week that Tim Berners-Lee (the visionary behind the World Wide Web, the Semantic Web, and the Linked Open Data movement) will be advising the UK Government on making government data more open and accessible.

In the US, data.gov is part of the Open Government Initiative of the Obama Administration. The following is an excerpt from data.gov:

A primary goal of Data.gov is to improve access to Federal data and expand creative use of those data beyond the walls of government by encouraging innovative ideas (e.g., web applications). Data.gov strives to make government more transparent and is committed to creating an unprecedented level of openness in Government. The openness derived from Data.gov will strengthen our Nation's democracy and promote efficiency and effectiveness in Government.

Governments around the world have taken notice and are now considering similar initiatives. It is clear that these initiatives are important for the proper functioning of democracy since they allow citizens to make informed decisions based on facts as opposed to the politicized views of special interests, lobbyists, and their spin doctors. These facts are related to important subjects such as health care, the environment, the criminal justice system, and education. There is an ongoing debate in the tech community about the best approach for publishing these datasets. There are several government data standards available such as the National Information Exchange Model (NIEM). In the Web 2.0 world, RESTful APIs with ATOM, XML, and JSON representation formats have become the norm.

I believe however that Semantic Web technologies and Linked Data principles offer unique capabilities in terms of bridging data silos, queries, reasoning, and visualization of the data. Again, the methodology for adopting Semantic Web technologies is the same:

Create an OWL ontology that is flexible enough to support the different types of data in the dataset including statistical data. This is certainly the most important and challenging part of the effort.
Convert the data from its source format to RDF. For example XSLT2 can be used to convert from CSV or TSV to RDF/XML and XHTML+RDFa. There are also RDFizers such as D2R for relational data sources.
Link the data to other data sources such as Geonames, Federal Election Commission (FEC), and US Census datasets.
Provide an RDF dump for Semantic Web Crawlers and a SPARQL endpoint for querying the datasets.

The following are some of the benefits of this approach:

It allows users to ask sophisticated questions against the datasets using the SPARQL query language. These are the kind of questions that a journalist, a researcher, or a concerned citizen will have in mind. For example, which airport has the highest number of reported aircraft bird strikes? (read more here about why Transportation Secretary Ray LaHood rejected a proposal by the FAA to keep bird strikes data secret). Currently data.gov provides only full-text and category-based search.
It bridges data silos by allowing users to make queries and connect data in meaningful ways across datasets. For example, a query that correlates health care, environment, and census data.
It provides powerful visualizations of the data through Semantic Web meshups.

S1000D and SCORM Integration

2009-06-02T17:29:00.005-04:00

This is a presentation I gave at the DocTrain Boston 07 conference on how to reduce product lifecycle costs by integrating the S1000D and SCORM specifications.

S1000D is the International Specification for Technical Publications utilizing a Common Source Database (CSDB). Based on open XML standards, the latest issue (4.0) has been developed by the AeroSpace and Defence Industries Association of Europe (ASD), the Aerospace Industries Association of America (AIA), and the Air Transport Association of America (ATA).

Sharable Content Object Reference Model (SCORM) is a specification for online learning content developed by the Advanced Distributed Learning (ADL) Initiative.

The presentation has been updated to reflect the addition of SCORM support in S1000D 4.0.

S1000D and SCORM Integration

View more OpenOffice presentations from Joel Amoussou.

Why XProc Rocks

2009-05-19T22:31:00.012-04:00

These are exciting times to be a technologist in the news publishing business. The industry is going through fundamental changes driven by the current economic downturn. Innovation has become an imperative. Examples of recent innovations include news content APIs (newspaper as a Platform), specialized desktop news readers, the ability to publish to an increasing number of mobile devices (Kindle, iPhone, etc.), and migrating news content to the Semantic Web and Linked Open Data (LOD) cloud.

The result is that news publishing workflows are getting more complex. The following is an example of what can be expected from such a workflow:

Retrieve content from a native XML database such as Exist using XQuery and from a MySQL database and combine the result as an XML document.
Expand XIncludes and other content references in the XML.
Apply various processing to the XML depending on the content type.
Make a REST call to data.gov and recovery.gov to retrieve some government published data in XML for a graphical mashup visualization of the data.
Transform the XML into XHTML+RDFa for consumption by Yahoo's SearchMonkey and the recently unveiled Google Rich Snippets.
Transform the XML into RDF/XML and validate the result using the Jena command line ARP RDF parser. The RDF/XML output will provide an RDF dump for RDF crawlers and will be searched via a SPARQL endpoint.
Transform the XML into an Amazon Kindle-friendly format such as Text.
Publish the content as a PDF document with a print layout (e.g. header, footer, multi-column, pagination, etc.).
Transform the XML into a NewsML document and validate the result against the NewsML XML Schema and an ISO Schematron schema.
Generate an Atom feed containing NewsML elements and validate the result using NVDL (Namespace-based Validation Dispatching Language). NVDL is required here because the Atom feed will contain nodes that will be validated against the Atom RelaxNG schema as well as nodes that will be validated against the NewsML XML schema.

As you can see, this publishing workflow is XML document-centric. If you are a Java developer, your first instinct might be to automate all those processing steps with an Ant build file. There is now a better alternative and it is called XProc (an XML Pipeline Language) currently a W3C candidate recommendation. Here are some reasons why XProc is a superior alternative when dealing with an XML document-centric publishing workflow:

With XProc, XML documents are first-class citizens. Java objects are first-class citizens in Ant. In fact, using Ant to automate a complex XML document-centric publishing workflow can quickly lead to spaghetti code.
XProc allows you to add, delete, rename, replace, filter, split, wrap, unwrap, compare, insert, and conditionally process nodes in XML documents by addressing these nodes using XPath.
XProc comes with built-in steps for XML processing such as p:xquery, p:xinclude, p:xslt, p:validate-with-xml-schema, p:validate-with-relax-ng, p:validate-with-schematron, p:xsl-formatter, and p:http-request for making REST calls.
XProc is declarative, and programming language and platform neutral. For example, work is underway for an XQuery implementation for the Exist database. Note that step 1. above can be implemented with the SQL extension function provided by Exist for retrieving data from relational databases using JDBC.
XProc is extensible, so you can add custom steps. For example the XML Calabash XProc implementation provides extension steps such as cx:nvdl and cx:unzip.
XProc provides exception handlers which are essential for any complex workflow.
XProc pipelines are amenable to streaming.
XProc can simplify the design, documentation, and maintenance of very complex publishing workflows. Since XProc is in XML format, one can envision a visual designer for building XProc pipelines or a tool (such as Graphviz) to visualize XProc documents in the form of processing diagrams.

There are other pipeline technologies that can be useful as well, particularly if you're building mashup applications. Yahoo Pipes is a good choice for combining feeds from various sources, manipulating them as needed, and outputting RSS, JSON, KML, and other formats. And if you are into Semantic Web meshups, DERI Pipes provides support for RDF, OWL, and SPARQL queries. DERI Pipes works as a command line tool and can be hooked into an XProc pipeline with the XProc built-in p:exec step.

Thoughts on SOAP vs. REST

2009-04-26T19:34:00.009-04:00

REST is now an increasingly popular architectural style for building web services. The question for developers is: should REST always be the preferred mechanism for building web services or is SOAP still relevant for certain use cases?

In my opinion, REST is usually a no-brainer when you are exposing a public API over the internet and all you need is basic CRUD operations on your data. However, when designing a truly RESTful web services interface (as opposed to some HTTP API), care must be taken to adhere to key principles:

Everything is a URI addressable resource
Representations (media types such as XHTML, JSON, RDF, and Atom) describe resources and use links to describe the relationships between those resources
These links drive changes in application state (hence Representational State Transfer or REST)
The only type that is significant for clients is the representation media type
URI templates as opposed to fixed or hard coded resource names
Generic HTTP methods (no RPC-style overloaded POST)
Statelessness (the server keeps no state information)
Cacheability.

Adherence to these principles is what enables massive scalability. One good place to start is the AtomPub protocol which embodies these principles. In the Java space, the recently approved Java API for RESTful Web Services (JAX-RS) specification greatly simplifies REST development with simple annotated POJOs.

Within the enterprise and in B2B scenarios, SOAP (and its WS-* family of specifications) is still very attractive. This is not to say that REST is not enterprise ready. In fact, there are known successful RESTful implementations in mission critical applications such as banking. However, enterprise applications can have specific requirements in the areas of security, reliable messaging, business process execution, and transactions for which SOAP, the WS-* specifications, and supporting tools provide solutions.

These specifications include:

WS-Addressing
WS-Policy
WS-ReliableMessaging
WS-SecureConversation
WS-Security
WS-SecurityPolicy
WS-Trust
WS-AtomicTransaction
WS-BPEL (Business Process Execution Language)

RESTafarians will tell you that REST can handle these requirements as well. For example, RESTful transactions can be implemented by treating the transactions themselves as URI addressable REST resources. This approach can work, but is certainly not trivial to implement. In fact, it is often difficult to support some of these requirements without resorting to overloaded POST, which works more like SOAP and is a clear departure from a pure REST architectural style.

One characteristic of enterprise SOA is the need to expose pieces of application logic (as opposed to data) as web services and this can be more amenable to a SOAP-based approach. Existing SOAP web services toolkits such as Apache CXF provide support for WS-* specifications. More importantly, they greatly simplify the development process by providing various tools such as the ability to create new services with a contract-first approach where JAX-WS annotated services and server stubs can be automatically generated from an existing WSDL.

Furthermore, during the last ten years, organizations have made significant investments in SOAP-based infrastructure such as Enterprise Service Buses (ESBs) and Business Process Management (BPM) software based on WS-BPEL. The Content Management Interoperability Services (CMIS) specification which is currently being developed by OASIS specifies protocol bindings for both SOAP and AtomPub. The SOAP binding will allow organizations to leverage those investments in building interoperable content repositories.

Architecting an SOA solution is a balancing act. It's important not to dismiss any particular approach too soon. Both SOAP and REST should carefully be considered for new web services projects.

News Content APIs: Uniform Access or Content Silos

2009-03-26T19:20:00.007-04:00

The first generation of online news content applications was designed for consumption by humans. With the massive amounts of online content now available, machine processable structured data will be the key to findability and relevance. Major news organizations like the New York Times (NYT), the National Public Radio (NPR), and The Guardian have recently opened up their content repositories through APIs.

These APIs have generated a lot of excitement in the content developers community and are certainly a significant step forward in the evolution of how news content is processed and consumed on the web. The APIs allow developers to create new interesting mashup applications. An example of such a mashup is a map of the United States showing how the stimulus money is being spent by municipalities across the country with hotspots to local newspaper articles about corruption investigations related to the spending. The stimulus spending data will be provided by the Stimulus Feed on the recovery.gov site as specified by the "Initial Implementation Guidance for the American Recovery and Reinvestment Act" document. This is certainly an example of mashup that US tax payers will like.

For news organizations, these APIs represent an opportunity to grow their ad network by pushing their content to more sites on the web. That's the idea behind the recent release of The Guardian Open Platform API.

APIs and Content Silos

The emerging news content APIs typically offer a REST or SOAP web services interface and return content in XML, JSON, or ATOM feeds. However, despite the excitement that they generate, these APIs can quickly turn into content silos for the following reasons:

The structure of the content is often based on a proprietary schema. This introduces several potential interoperability issues for API users in terms of content structure, content types, and semantics.
It is not trivial to link content across APIs
Each API provides its own query syntax. There is a need for universal data browsers and a query language to read, navigate, crawl, and query structured content from different sources.

XML, XSD, XSLT, and XQuery

Migrating content from HTML to XML (so called document-oriented XML) has many benefits. XSLT enables media-independent publishing (single sourcing) to multiple devices such as Amazon's Kindle e-reader and "smart phones". With XQuery, sophisticated contextualized database-like queries can be performed, turning the content itself into a database. In addition, XQuery allows the dynamic assembly of content where new compound documents can be constructed on the fly from multiple documents and external data sources. This allows publishers to repurpose content into new information products as needed to satisfy new customer demands and market opportunities.

However XSD, XSLT, and XQuery operate at the syntax level. The next level up in the content technology stack is semantics and reasoning and that's where RDF, OWL, and SPARQL come into play. To illustrate the issue, consider three news organizations, each with their own XML Schema for describing news articles. To describe the author of an article, the first news organization uses the <creator> element, the second the <byline> element, and the third the <author> element. All of these three distinct element names have exactly the same meaning. Using an OWL ontology, we can establish that these three terms are equivalent.

Semantic Web and Linked Data to the Rescue

Semantic web technologies such as RDF, OWL, and SPARQL can help us close the semantic gap and also open up new opportunities for publishers. Furthermore, with the decline in ad revenues, news organizations are now considering charging users for accessing content online. Semantic web technologies can enrich content by providing new ways to discover and explore content based on user context and interests. An interesting example is a mashup application built by Zemanta called Guardian topic researchr which extract entities (people, places, organizations, etc.) from The Guardian Open Platform API query results and allows readers to explore these entities further. In addition, the recently unveiled Newssift site by the Financial Times is an indication that the industry is starting to pay attention to the benefits of "semantic search" as opposed to keyword search.

The rest of this post outlines some practical steps for migrating news content to the Semantic Web. For existing news content APIs, an interim solution is to create Semantic Web wrappers around these APIs (more on that later). The long term objective however should be to fully embrace the Semantic Web and adopt Linked Data principles in publishing news content.

Adopt the International Press Telecommunication Council (IPTC) News Architecture (NAR)

The main reason for adopting the NAR is interoperability at the content structure, content types, and semantic levels. Imagine a mashup developer trying to integrate news content from three different news organizations. In addition to using three different element names (<creator>, <byline>, and <author>) to describe the same concept, these three organizations use completely different XML Schemas to describe the structure and types of their respective news content. That can lead to a data mapping nightmare for the mashup developer and the problem will only get worse as the number of news sources increases.

The NAR content model defines four high level elements: newsItem, packageItem, conceptItem, and knowledgeItem. You don't have to manage your content internally using the XML structure defined by the NAR. However, you should be able to map and export your content to the NAR as a delivery format. If you have fields in your content repository that do not map to the NAR structure, then you should extend the standard NAR XML Schema using the appropriate XML Schema extension mechanism that allows you to clearly identify your extension elements in your own XML namespace. Provide a mechansim such as dereferenceable URIs to allows users to obtain the meaning of these extensions elements.

The same logic applies to the news taxonomy that you use. Adopting the IPTC News Codes which specified 1300 terms used for categorizing news content will greatly facilitate interoperability as well.

Adopt or Create a News Ontology

Several news ontologies in RDFS or OWL format are now available. The IPTC is in the process of creating an IPTC news ontology in OWL format. To facilitate semantic interoperability, news organizations should use this ontology when it becomes available. In mapping XML Schemas into OWL, ontology best practices should be followed. For example, if mapped automatically, container elements in the XML Schema could generate blank nodes in the RDF graph. However, blank nodes cannot be used for external RDF links and are not recommended for Linked Data applications. Also, RDF reification, RDF containers, and RDF collections are not SPARQL-friendly and should be avoided.

While creating the news ontology, you should reuse or link to other existing ontologies such as FOAF and Dublin Core using OWL elements like owl:equivalentProperty, owl:equivalentClass, rdfs:subClassOf, or rdfs:subPropertyOf.

Similarly, existing taxonomies should be mapped to an RDF compatible format using the SKOS specification. This makes it possible to use an owl:Restriction to constrain the value of a property in the OWL ontology to be an skos:Concept or skos:ConceptScheme.

Generate RDF Data

Assign a dereferenceable HTTP URI for each news item and use content negotiation to provide both an XHTML and an RDF/XML representation of the resource. When the resource is requested, an HTTP 303 See Other redirect is used to serve XHTML or RDF/XML depending on whether the browser's Accept header is text/html or application/rdf+xml. The W3C Best Practice Recipes for Publishing RDF Vocabularies explains how dereferenceale URIs and content negotiation work in the Semantic Web.

The RDF data can be generated using a variety of techniques. For example, you can use an XSLT-based RDFizer to extract RDF/XML from news item already marked up in XML. There are also RDFizers for relational databases. Entity extraction tools like Open Calais can also be useful particularly for extracting RDF metadata from legacy news items available in HTML format.

Link the RDF data to external data sources such as DBPedia and Geonames by using RDF links from existing vocabularies such as FOAF. For example, an article about US Treasury Secretary Timothy Geithner can use foaf:base_near to link the news item to a resource describing Washington, DC on DBPedia. If there is an HTTP URI that describes the same resource in another data source, then use owl:sameAs links to link the two resources. For example, if a news item is about Timothy Geithner, then you can use owl:sameAs to link to Timothy Geithner's data page on DBPedia. An RDF browser like Tabulator can traverse those links and help the reader explore more information about topics of interest.

Expose a SPARQL Endpoint

Use a Semantic Web Crawler (an extension to the Sitemap Protocol) to specify the location of the SPARQL endpoint or an RDF dump for Semantic Web clients and crawlers. OpenLink Virtuoso is an RDF store that also provides a SPARQL endpoint.

Provide a user interface for performing semantic searches. Expose the RDF metadata as facets for browsing the news items.

Provide a Semantic Web Wrapper for existing APIs.

A wrapper provides a deferenceable URI for every news item available through an existing news content API. When an RDF browser requests the news item, the Semantic Web wrapper translates the request into an API call, transforms the response from XML into RDF, and send it back to the Semantic Web client. The RDF Book Mashup is an example of how a Semantic Web Wrapper can be used to integrate publicly available APIs from Amazon, Google, and Yahoo into the Semantic Web.

Conclusion

The Semantic Web is still an obscure topic in the mainstream developers community. I hope I've outlined few practical steps you can take now to take advantage of the new Web of Linked Data.

TOGAF 9: The missing link between IT and the business

2009-02-21T09:01:00.011-05:00

In a recent and famous blog about the death of SOA, Anne Thomas Manes wrote:

After investing millions, IT systems are no better than before. In many organizations, things are worse: costs are higher, projects take longer, and systems are more fragile than ever. The people holding the purse strings have had enough. With the tight budgets of 2009, most organizations have cut funding for their SOA initiatives.
...
The small select group of organizations that has seen spectacular gains from SOA did so by treating it as an agent of transformation. In each of these success stories, SOA was just one aspect of the transformation effort. And here’s the secret to success: SOA needs to be part of something bigger. If it isn’t, then you need to ask yourself why you’ve been doing it.

In my opinion, that "something bigger" that Anne is referring to is Enterprise Architecture (EA), the missing link between IT and the business. There are several reasons why IT projects fail to deliver on their promise. The lack of EA expertise and practice is certainly one of them. If you ask a group of developers how they will go about architecting an SOA solution, the typical answer that you will hear is the use of some kind of agile or UML-based methodology for gathering the requirements and modelling the application. While these steps are required in any software development project, the lack of a methodology and governance framework for aligning IT with the overarching business context, vision, and drivers can lead to chaos and total failure. In the case of SOA, this situation creates a phenomenon known as JBOWS (Just a Bunch of Web Services) Architecture.

For people who are interested or responsible for EA in their organizations, this month has seen the release of two interesting publications:

"97 Things Every Software Architect Should Know: Collective Wisdom from the Experts" by O'Reilly Media
TOGAF 9 by the Open Group

In the first publication, several software architects share their experiences and lessons learned from the trenches. One of those "97 things" is entitled "Architecting is about balancing". I couldn't agree more. The wisdom of these experts will be a great asset to any software architect.

TOGAF 9 is an ambitious project by the Open Group to create a set of standardized semantics, methodology, and processes in the field of EA. IT professionals have been told repeatedly that they need to become business savvy in order to be more effective to their organizations. TOGAF 9 help them bridge the gap. The diagram below from the TOGAF 9 documentation provides an overview (click on the image to enlarge).

TOGAF 9 has a modular structure which permits an incremental approach to implementation and allows different stakeholders to focus on their respective concerns. The Architecture Development Method (ADM) describes a method for developing an enterprise architecture and is the core of TOGAF. The diagram below from the TOGAF 9 documentation shows the different phases of the ADM (click on the image to enlarge)..

The Content Framework specifies deliverables and artifacts for each architectural building block. These artifacts represent the output of the ADM. The specification provides guidance on how the ADM can be applied to SOA and enterprise security. TOGAF 9 also addresses the important issues of architecture partitioning and architecture repository.

Finally, the Open Group has been working on the adoption of an open EA modeling standard called ArchiMate. ArchiMate provides a higher level view of EA when compared to modeling standards such as BPMN and UML. It can be used to depict different layers of EA including business processes, applications, and technology in a way that can be consumed by non-technical business stakeholders. A sample of an ArchiMate enterprise view of a hospital can be found here.

In Defense of XSLT

2009-01-24T14:34:00.007-05:00

I recently had a conversation with a Java programmer about why he doesn't like XSLT. The following describes his objections and how I handled them as an XSLT salesman.

XSLT is hard to read and debug

This is a matter of using the right tool for the job. While an Eclipse plug-in like EclipseXSLT is better than a simple text editor, using a full blown XML IDE like OxygenXML can greatly increase productivity.

OxygenXML can help you navigate both the input XML document (XML input document view) and the XSLT transform (XSLT template view). The contextual XPath and XSLT content assistants are very helpful given the large number of XSLT 2.0 elements and XPath 2.0 functions now available. The XSLT refactoring feature allows you to turn a selection into a named template or an included XSLT fragment. Finally, the XSLT debugging perspective allows execution with breakpoints as well as an XSLT call stack and XPath watch views and many other goodies.

In addition, the XSLT 2.0 specification itself defines constructs to facilitate the debugging process. You can use the select attribute (an XPath expression) of the <xsl:message> instruction or the content of <xsl:message> as a sequence constructor to output useful information to standard output or even to a log file. The <xsl:comment> instruction and the XPath 2.0 trace() function can be helpful in debugging as well.

XSLT does not support the reusability and maintainability of code

In addition to XSLT named templates inherited from XSLT 1.0, you can now create your own custom functions in XSLT 2.0. While the content models of both <xsl:template> and <xsl:function> are the same, <xsl:function> is preferred for computing new values and selecting nodes (because functions are called from XPath expressions), while <xsl:template> is preferred for constructing new nodes. The “as” attribute which can be specified on <xsl:template>, <xsl:function>, and their <xsl:param> children allows you to constrain the type of returned value and input parameters.

For reusability and maintainability, the <xsl:include>, <xsl:import>, and <xsl:apply-imports> elements inherited from XSLT 1.0 are still available. When used properly, <xsl:import> provides capabilities that are similar to inheritance in object-oriented languages like Java. <xsl:apply-imports> and the XSLT 2.0 <xsl:next-match> instructions are reminiscent of a call to super() in Java.

There is no way to integrate XSLT with my Java libraries

With Saxon, you can represent a Java class as the namespace URI of a function and you can call Java methods and constructors directly from your XSLT transform. This allows you to reuse existing pieces of application logic build in Java without rewriting them in XSLT.

There is no type checking and no way to verify that the result of a transformation is valid against a schema

Again, this argument is no longer valid with XSLT 2.0. You can now validate both the input and output by using a schema-aware XSLT processor. I strongly recommend the schema-aware version of Saxon. This allows you to root out errors and correct bugs early. In addition to the built-in XML Schema types such as xs:decimal and xs:dateTime, you can define custom types in an XSD. You can then write a template that matches all elements of a certain type. XSD type hierarchies and substitution groups are fully supported as well.

After schema validation, a Post Schema Validation Infoset (PSVI) is generated and each node is assigned a typed value (which can be obtained using the data() function) and a type annotation (which is the schema type used to validate the node). To ensure that a string has a given type annotation, constructor functions are available for built-in and custom types such as in xs:date(“2009-01-24”).

Saxon supports “optimistic static type-checking”. The following is an excerpt from the Saxon FAQ:

“Saxon does not do static type-checking in the sense that the term is used in the W3C language specifications (this refers to pessimistic type checking, in which any construct that might fail at run-time is rejected at compile time). This is an optional feature of the W3C specifications. Saxon does however perform optimistic static analysis of queries and stylesheets, in which an error is reported only for constructions that must always fail at run-time. The information derived from this static analysis is also used to optimize the run-time code."

Unlike Java, XSLT 2.0 lacks a try/catch feature. However you can use the handy “castable as” and “instance of” operators to detect programming errors as well as data errors in the input document.

With XSLT, it is "run and pray": there is no unit and functional testing framework like in Java

Automated unit and functional testing are essential in agile software development. Type checking (or schema-aware XSLT) can help you reduce the number of unit tests needed to fully test your XSLT transform but may not be enough to detect all errors such as those related to business rules violations in the output.

Unit testing is performed by testing individual XSLT functions and templates. You can try Jeni Tennison's XSpec framework which is inspired by the Ruby RSpec framework itself based on a Behavior Driven Development (BDD) approach.

Functional testing consists in testing whole outputs of the XSLT transform. One way of doing this is to run a Schematron schema which contains XPath 2.0-based assertions against the output of the XSLT transform. Since the Schematron schema validation process is itself based on XSLT, you can chain all these transformations together with a build tool like Ant which produces an HTML report with friendly diagnostic messages. With XML Schema 1.1, you will be able to build these assertions directly into your schema, although you will have less control other the diagnostic messages that are produced by the XSD validator.

Conclusion

In a world marked up in XML, XSLT 2.0 is a very powerful language available to developers. However, to take full advantage of the language it is important to use the right tools and take the time to explore the full capabilities of the language particularly its code reuse and type checking features. Finally, to bring XSLT into the realm of agile software development, unit and functional testing should become an integrated part of the XSLT development process.

From Web 2.0 to the Semantic Web: Bridging the Gap in the News and Media Industry

2008-12-29T20:18:00.015-05:00

The news and media industry is going through fundamental changes. This transformation is driven by the current economic downturn and the emergence of the web as the new platform for creating and publishing content. The decline in ad revenues has forced some media companies to cancel their print publications (e.g. PC Magazine and the Christian Science Monitor). Others such as Forbes Media are consolidating their online and editorial groups into a single entity.

What are the opportunities and challenges of the Semantic Web (sometimes referred to as Web 3.0) for the industry and how can these companies embrace and extend the new web of Linked Data?

Widgets and APIs

Media companies are looking for ways to gain a competitive advantage from their web offerings. In addition to Web 2.0 features like blogs and RSS feeds, major news organizations such as the New York Times (NYT) and the National Pubic Radio (NPR) are opening their content to external developers through APIs. These APIs allow developers to mash-up content in new ways limited only by their own imagination. The ultimate goal is to drive ad revenues by pushing content to places like social networking sites and blogs where readers “hang out” online. Another interesting example is the Times Widget (from the NYT) which allows readers to insert NYT content such as news headlines, stock quotes, and recipes on their personal web pages or blogs.

Beyond Web 2.0: the Semantic Web

Today's end users obtain information from a variety of sources such as online newspapers, blogs, Wikipedia, YouTube videos, Flickr photos, cartoons, animations, and social networking sites such as Facebook and Twitter. I personally get some of the news I read from the people I follow on Twitter (including political cartoons at http://twitter.com/dcagle). The challenge is to integrate all these sources of information into a seamless search and browsing experience for readers. Publishers are starting to realize the importance of this integration as illustrated by the recently unveiled "Times Extra" feature from the NYT.

The key to this integration is metadata and this is where Semantic Web technologies such as RDF, OWL, and SPARQL have an important role to play in presenting massive amounts of content to end users in a way that is compelling. Semantic search and browsing as well as inference capabilities can help publishers in their effort to attract and retain readers and boost ad revenues.

News and Media Industry Metadata Standards

Established in 1965, the International Press Telecommunications Council (IPTC) is a consortium of news agencies and publishers including the Associated Press, the NYT, Reuters, and the Dow Jones Company. The IPTC maintains and publishes a set of news exchange and metadata standards for media types such as text, photos, graphics, and streaming media like audio and video. This includes:

NITF which defines the content and structure of news articles
NewsML 1 for the packaging and exchange of multimedia news
NewsML-G2, the latest standard for the exchange of all kinds of media types
EventsML-G2 for news events
SportsML for sport data

These standards are all based on XML and use XML Schema Definitions (XSDs) to describe the structure and content of the media types. In addition, IPTC defines a taxonomy for news items called NewsCodes as well as a Photo Metadata Standard based on Adobe's XMP specification.

Another interesting standard in the news and media industry is the Publishing Requirements for Industry Standard Metadata (PRISM) specification. Also based on XML, PRISM is more applicable to magazines and journals and is compatible with RDF. There is a certain degree of overlap between PRISM and the IPTC news metadata standards. Media companies that have adopted these standards are well positioned to bridge the gap to the Semantic Web.

From XML/XSD to RDF/OWL

The W3C Semantic Web Activity and the Semantic Web community have been working on a number of specifications and tools to facilitate the transition to the Semantic Web. The W3C Gleaning Resource Descriptions from Dialects of Languages (GRDDL) specification defines a method for using XSLT to extract RDF statements from existing XML documents. For example, using GRDDL, a news organization would be able to generate RDF from content items marked up in NITF or NewsML.

The recently approved RDFa W3C specification allows publishers to get their content ready for the Semantic Web by adding extension attributes to XHTML content to capture semantic information based on Dublin Core or FOAF for example.

The W3C Simple Knowledge Organization System (SKOS) is an application of RDF that can be used to represent existing taxonomies and classification schemes (such as the IPTC NewsCodes) in a way that is compatible with the Semantic Web. SPARQL is a query language for RDF that is supported in Semantic Web toolkits such as Jena.

There are also tools that can be used to derive an OWL ontology from an existing XSD. This transformations can be straightforward if the XSD was designed to be compatible with RDF. As an example, an OWL ontology can be derived from the IPTC's NewsML-G2 XSDs. Once created, new ontologies can be linked to existing ontologies such as Dublin Core, FOAF, and DBPedia as described by Tim Berner Lee in his vision of Linked Data. Other relevant ontologies for the news and media industry include MPEG-7, the Core Ontology for Multimedia (COMM), and Geonames.

From Web 2.0 Tagging to Semantic Annotations

To facilitate the transition of the industry to the Semantic Web, it will be important to design content management systems (CMS) interfaces that make it easy for content contributors to add semantic annotations to their content. These systems certainly have a lot to learn from Web 2.0 tagging interfaces such as Flickr clustering and machine tags. However the complexity of content in the news and media industry demands more sophisticated annotation capabilities than those that are available to the masses on YouTube and Flickr. Therefore full support for Semantic Web standards like RDF, OWL, SKOS, and SPARQL will be expected.

An interesting application in this space is the Thompson Reuters Calais application which allows content publishers to automatically extract RDF-based metadata on entities such as people, places, facts, and events. LinkedFacts.com is an example of a semantic news search application powered by Calais.

Calais has been integrated with the open source content management system (CMS) Alfresco to enable auto-tagging of content as well as the automatic suggestion of tags to content contributors. The latest release of Calais adds the ability to generate Linked Data to other sources such as DBPedia, GeoNames, and the CIA World Fact Book.

Conclusion

With the rapid growth of news content online, relevance is going to become very important. This will require metadata and structured data in general. By exposing content in XML format, news content APIs are certainly a step in the right direction. However, these APIs do have their own limitations and can create content silos in the future. Beyond APIs, Semantic Web technologies and Linked Data principles will ensure a uniform and intelligent access to news content. This will be the key to reader retention and content monetization.

Keeping Data Under Control in SOA with XSLT 2.0 and XQuery

2008-12-03T21:23:00.018-05:00

In a typical SOA project, several artifacts are represented in XML format. This includes SOAP messages, XML Schema definitions (XSDs), WSDL, WS-Policy, BPMN, BPEL, and various configuration files. The following are some examples of how XSLT 2.0 and XQuery can be leveraged in an SOA project.

Data Model and Data Format Transformation

When the services don't share the same data model (XSD) or the same data format (e.g. EDI vs. XML), there is a need to transform the data. An Enterprise Service Bus (ESB) typically provides data transformation as part of its mediation services. Some developers will find XQuery easier to use than XSLT 2.0 for transforming XML data because XQuery has a SQL-like syntax. In Oracle Aqualogic ESB, XQuery is used for data transformation while the Apache ServiceMix ESB provides support for Saxon-based XSLT 2.0 and XQuery transformations. An XSLT/XQuery engine can be deployed as Service Component Architecture (SCA) implementation type or a Java Business Integration (JBI) service engine.

One aspect of data model or data format transformation that can quickly get out of control and become difficult to manage is the mapping specification which says that a field X in message A maps to field Y in message B. In addition, the mapping specification defines business rules. This mapping specification is often produced in Excel spreadsheets format by business analysts and handed over to programmers who then code the transformation script. Now, you need to maintain and synchronize the Excel mapping document, the source XSD, the target XSD, and the XSLT 2.0 or XQuery transformation script. On top of that, if a user interface is involved, you need to ensure that it is also kept in sync with those changes.

One technique that can be useful is to use an xsd:appinfo element to capture and keep metadata close to the XSD declarations:

Data mapping specifications
Business rules using inline ISO Schematron rules for example
Labels, alerts, and appearances of UI components such as XForms controls.

This allows you to use XSLT 2.0 or XQuery to automatically generate data mapping reports in Excel or even generate UI components by transforming the XSD into XForms controls.

Managing Artifacts and Promoting Reuse with an SOA Repository

One of the key aspects of design-time SOA Governance is the management of the lifecycle of service artifacts and the dependencies between them. This is accomplished through a new breed of tools called SOA Repositories. Some of these repositories are being build on top of a JCR compliant repository such as Apache Jackrabbit. JCR supports querying compliant repositories using XPath 2.0. Suppose that I need to add a new element to my XSD. To ensure that I am reusing existing schema constructs, I first query the SOA Repository with XPath 2.0 to find all schema components (types, elements, and attributes) that contain a certain keyword inside their xsd:documentation element. XPath 2.0 can also help in detecting dependencies between artifacts (e.g. WSDL and XSD definitions) for change impact analysis. Open source SOA repositories such as Mule Galaxy, JBoss DNA, and WSO2 Repository have adopted this approach.

Functional Testing of Web Services

Automated testing is a key principle in agile software development. SoapUI is an open source web services functional testing framework that allows testers to not only perform XSD validation of SOAP messages, but also allows them to specify assertions on the structure and content of those messages using XPath 2.0 and XQuery. SoapUI can be easily integrated into a continuous integration process.

Data Integration

XQuery can alleviate performance and scalability issues related to the marshalling/unmarshalling of Java objects to/from XML (databinding) and object-relational mapping (ORM) for persisting XML data in relational databases. XQuery is a natural solution for querying and aggregating data coming from heterogeneous sources such as relational databases, native XML databases, LDAP, file systems, and legacy data formats such as EDI and CSV.

One promising specification in the data integration space is the W3C XQuery Scripting Extension (XQSE). By extending XQuery with imperative features such as state management, XSQE (pronounced "excuse") provide developers with additional XML processing power without the need to embed XQuery in a host language such as Java. At the time of this writing, XQSE is still a W3C working draft but is already supported by the Oracle AquaLogic Data Services Platform (ALDSP 3.0).

The Content Imperative: Unlearning the Relational Model

2008-11-11T22:14:00.028-05:00

The relational data model and the SQL query language are an essential part on any computer science curriculum and are well understood by a large number of developers. On the other hand, the use of markup technologies such as SGML and XML for content management and publishing has remained a niche market for highly specialized vendors and consultants.

Today, the majority of developers use XML for application configuration files (e.g. Spring, Hibernate, JSF), syndication (RSS), and web services. When these developers are asked to design and develop XML content management and publishing applications, they often approach the problem from a relational data paradigm which is what they know and are used to. For example, when migrating content stored in a relational database into an XML format, they will simply dump the relational tables into a flat XML representation. The problem is that content is not relational data. The following are some fundamental differences between content and relational data:

Content is created to be consumed by eyeballs
Content can be rendered in multiple presentation formats such as print, web, and wireless devices. Therefore it is very important to cleanly separate content from presentation
Content can have an inherent deep hierarchical structure. For example, think about the book/part/chapter/section/subsection/paragraph hierarchy
The relationships between content items are expressed through hierarchical containment and hyperlinks
Content is often mixed (in the sense of mixed content in XML). For example, inside a paragraph, some words are italicized, in bold, or underlined to indicate special meaning
Content can have multi-valued properties such as the authors of a document. Multi-valued properties are not supported by SQL.

However, content and data do have one important thing in common. Data and content stay with us for a long time, sometimes forever. APIs, protocols, and programming languages come and go. Therefore, content modeling is by far the most important investment you can make during a content management migration project.

Unstructured Content Modeling

In a typical enterprise, there are two different types of content: unstructured and structured. Unstructured content represents the large majority of content in the enterprise. Examples are Office documents such as Word, PowerPoint, and Excel. Content modeling for unstructured content consists in describing document metadata as well as the relationships between the documents. The metadata is usually stored in a relational database. In a typical CMS, the content model is used to customize the user interface for querying specific documents based on metadata and for customizing the user interface (e.g. for capturing and displaying metadata).

Most content management systems have their own proprietary meta-model. The Java content repository API (JCR) introduced a standardized hierarchical repository model. This article by David Nuescheler (JCR spec Lead and CTO at Day Software) explains the peculiarities of JCR content modeling and the gotchas for people coming from a relational data modeling background.

Apache Jackrabbit (the JCR reference implementation) uses a textual DSL called Compact Namespace and Node Type Definition (CND) for specifying a JCR content model. There is no formal graphical notation like UML or ERDs for specifying JCR content models. Lars Trieloff of Day Software proposed a content modeling notation based on UML and Fundamental Modeling Concepts (FMC).

The Content Management Interoperability Services (CMIS) specification proposed a simplified meta-model based on documents, folders, policies, and relationships. The CMIS query language extends SQL 92 with text search, multi-valued properties, and folder-scoped queries. This choice was made because most existing CMS use a relational database and SQL is well understood by the majority of developers out there.

The Case Against Unstructured Content

The problem with unstructured content is that it cannot be processed and queried like the well-structured relational data stored by the RDBMS on which your ERP and CRM systems sit. XML goes beyond tags (in the web 2.0 sense), taxonomies, full-text search, and content categorization to provide fine-grained content discovery, query, and processing capabilities. With XML, the document becomes the database. If your business is content (you are a media company, a publisher, or the technical documentation department of a manufacturing company), then you should seriously consider the benefits of XML in terms of content longevity, reuse, repurposing, and cross-media publishing.

The question is: how to do it right?

From SGML to XML to the Infoset

Charles Goldfarb invented SGML at IBM in 1969, the same year his colleague Edgar Codd invented the relational model. The SGML (through its very popular subset called XML) and relational models are still rock solid today. To be precise, the abstract data model for XML documents was formally specified in the XML Infoset and subsequently the XQuery 1.0 and XPath 2.0 data model (XDM) specifications.

SGML was originally designed at IBM for the editing, retrieval, and composition of legal documents. The second edition of the Oxford English Dictionary and the US Department of Defense (DoD) adopted SGML in the 80s. The goal of the US DoD CALS initiative was to replace the huge quantities of paper with digital data in the acquisition of weapon systems. Before the adoption of XML by the W3C in February 98, SGML was primarily used in the publishing industry as well as for technical documentation applications in industries such as aerospace, defense, software, and telecommunications. Examples of SGML vocabularies include S1000D (aerospace and defense), Docbook (software), and TIM (telecommunications). The most popular SGML vocabulary, HTML, is the foundation of the web itself.

The XML Infoset describes the content of a well-formed XML document as an abstract tree of information items including document, namespace, element, and attribute information items. These information items have properties such as children and parent. Based on the XML Infoset, the XDM defines the abstract data model for the input to XSLT 2.0 and XQuery 1.0 processors. In addition, the XDM supports XML Schema types, atomic typed values, and ordered heterogeneous sequences.

The relational data model is based on set theory and predicate logic. It was designed originally for accounting and banking systems. Data is represented as n-ary relations and manipulated with relational algebra. CMS vendors and even standard bodies have tried to fork SQL in order to support hierarchies and multi-value properties. It is clear however that XQuery is a superior alternative, specifically designed to address those content-related concerns.

WikiXMLDB is an interesting application that uses XQuery to query Wikipedia content. WikiXMLDB allows you to not only perform database-like queries, but also enables dynamic content assembly or the ability to build compound documents from multiple Wikipedia pages. This opens up new opportunities in terms of content enrichment at a time when publishers are struggling to find new ways to monetize their content assets in the face of declining ad revenues.

Structured Content Modeling

There is a body of specialized knowledge in SGML/XML content analysis and modeling that has been applied successfully to projects such as the Oxford English Dictionary (OED), CALS, and Docbook.

In relational data modeling, the three phases of modeling typically include:

The conceptual or domain model
The logical data model (LDM)
The physical data model (PDM)

The LDM describes entity types, data attributes, and the relationships between the entities. The LDM is normalized to eliminate redundancies. The PDM describes the schema of the database that will be used to store the data and may be denormalized to improve performance. Relational data modeling uses well known modeling notations like Entity-Relationship (ER) diagrams and UML.

The difference between logical model and physical model is even more important in modeling XML documents. The logical model can be expressed in XML Schema or Relax NG. However, the physical model depends on the underlying data persistence mechanism (e.g. relational vs. JCR vs. XQuery-enabled native XML database). Many projects make the mistake of skipping the logical modeling phase. The problem is that the physical storage can and will probably change over time.

There is no formal notation for modeling XML documents. Back in the SGML days, document analysis and modeling was a well understood process. Eve Maler and Jeanne El Andaloussi describes a tree notation for modeling DTDs in their book entitled “Developing SGML DTDs: From Text To Model To Markup”. One of the peculiarities of modeling narrative text is the presence of mixed content which is essential for intelligent processing of content (for example for the automatic extraction of book indexes).

David Carlson proposed a UML Profile for XSD which can be used to auto-generate XML schemas from UML class diagrams. As with any model-driven development tool, care should be taken to ensure that the generated XML Schema complies with XML content modeling principles and satisfies the business and technical requirements such as content reuse and repurposing.

Many people prefer Relax NG to XML Schema for its simplicity. The important think to remember however is that XML Schema is part of a full stack of XML specifications which also includes XForms, XPath 2.0, XQuery, and XSLT 2.0. The ability to load XML Schema types into your XForms, validate XForms submissions against an XML Schema, and create schema-aware XSLT 2.0 transforms and XQuery queries can be a deciding factor.

Instead of starting your content modeling effort from scratch, you can leverage any of the existing proven and well tested document-oriented XML vocabularies such as Docbook, DITA, S1000D, and NewsML.

JCR and XML Content Modeling

JCR supports the import of arbitrary XML documents into a compliant repository. The following is an excerpt from the specification:

On import...

Each XML element E becomes a content repository node of the same name, E.
...
Each child XML element C of XML element E becomes a content repository child node C of node E.
Each XML attribute A within an XML element E becomes a property A of content repository node E. The value of each XML attribute A becomes the value of the corresponding property A.
...
Text within an XML element E becomes a STRING property called jcr:xmlcharacters of a node called jcr:xmltext, which itself becomes a child node of the node E.

This is fine if you're only storing a small quantity of XML documents. Performance will probably degrade quickly if you're storing a large quantity of XML documents with a deep hierarchy. Same name siblings are almost always used in document-oriented XML (e.g. paragraph siblings) and can cause performance to degrade or JCR paths to become brittle if you remove or reorder nodes. JCR allows roundtripping of imported XML. However, JCR adds repository metadata such as jcr:primaryType that must be stripped out at export time. It's possible to derive an optimized JCR content model from the XML document's logical model, although this could prevent you from fully exploiting the original hierarchy of the XML documents in your application. This shows the importance of separating the logical model from the physical model.

You should seriously consider a native XML database when dealing with large quantities of document-oriented XML documents. A simple benchmarking exercise (between Jackrabbit and the Exist database for example) can help you settle down on the right solution.

Business Rules and Content Quality

In addition to specifying the content items, types, and relationships in your schema, you should also specify business rules that are beyond the capabilities of the XML Schema language. ISO Schematron uses XPath 2.0 to declare assertions about arbitrary patterns in XML documents and then reports on the presence or absence of those patterns. In this article, Rick Jelliffe, inventor of Schematron explains how it works.

Assertions and conditional type assignments capabilities have been added to XML Schema 1.1.

Content Capture

XML editors have been around for a long time. However, these XML editors remain complex specialist tools that are often used only by professional technical authors in documentation departments. Using XForms, you can provide a user friendly interface for your end users to contribute XML content by presenting them with a simple XHTML form. The Alfresco web content management (WCM) platform uses XForms for content capture and XSLT/XSL FO for content rendition. The XForms controls can be generated automatically from an XML Schema. Alfresco's implementation is based on the open source Chiba XForms engine.

The Right Tools for the Job

When content is stored in a relational database, typically an object relational mapping (ORM) solution like Hibernate is used to map relational tables into Java objects. The business logic is handled with POJOs such as Spring beans which communicate with the front end through a UI framework or templating technology such as JSP or Freemarker.

When dealing with XML documents however, there are domain specific languages such as XForms, XInclude, XLink, XPointer, XPath 2.0, XSLT 2.0, XQuery, and XSL FO that greatly facilitate the processing of document-oriented XML and provide unmatched processing power when compared to traditional approaches based on SQL, JSP, or Freemarker. These XML-related languages are declarative in nature and require some learning curve. Many software architects and developers are not yet aware of the power of this new paradigm. However, I believe that rather than using only familiar development frameworks, it’s important to always evaluate alternatives and select the best approach even if it involves a learning curve.

XML databases such as Exist can store data natively in XML and provide full XQuery support for sophisticated queries and manipulation of XML content. Exist also provides integrated support for XInclude, XSLT 2.0, and AtomPub. The XRX (XForms, REST, XQuery) architecture with Exist and the Orbeon XForms engine allows you to integrate an XForm front end to an Exist data store through a REST API, therefore bypassing the ORM layer altogether.

Content Modeling for the Semantic Web

The web ontology language (OWL) is used for modeling for the Semantic Web. In the RDF data model, statements are expresses as Subject-Predicate-Object. Specialized RDF stores exist for storing RDF triples. In this blog post, Kingsley Idehen, CEO of OpenLink Software (maker of the Virtuoso RDF triple store), explains why The Time for RDBMS Primacy Downgrade is Nigh!

Conclusion

In summary, content modeling in general and XML content modeling in particular are intrinsically different from relational data modeling. In modeling XML content, the logical model should be separated from the physical model because the latter depends on the persistence storage mechanism which can change during the content's life cycle. XQuery-enabled native XML databases provide a better alternative for storing, querying, and processing large quantities of document-oriented XML. The relational data model is still rock solid today and will be around for many years to come because of its strong foundations in mathematics. However, it is not a panacea for all information management problems. While SQL can be forked to support content characteristics such as hierarchy and multi-valued properties, XQuery was natively designed to address those concerns. Companies can help their developers by providing training on declarative XML processing languages like XForms, XQuery, and XSLT 2.0 which are better suited for handling XML content.

AtomPub Use Cases in the Aviation Industry

2008-10-23T20:20:00.024-04:00

At the XML 2007 Conference in Boston, I introduced my concept of an Integrated Documentation Environment for Aircraft Support (IDEAS) based on AtomPub and OpenSearch. The following are some use cases that illustrate how such an approach could facilitate integration and technical information publishing in the aviation industry:

Notification. Nancy is an aircraft mechanic. When she gets to work in the morning, she opens her feeds aggregator to get new and updated content from all of the following sources: airframer, engine manufacturer, component manufacturers, FAA, or airline policies. Essentially, Nancy doesn't want to login into the support sites of all those content providers to find out what is new and updated. She wants content pushed to her instead. This use case is implemented with the Atom syndication format.

Federated search. While Nancy is repairing the hydraulic tank, she wants to perform a single search against all those content repositories. She wants the results aggregated and returned to her as Atom entries, so that she can subscribe to those items that she is interested in and receive updates via web feeds. This use case is implemented with the OpenSearch specification.

Airline Originated Changes. Judy is an engineer working on a new engineering order (EO) to be performed on the hydraulic tank. The airline's technical documents are hosted by the aircraft manufacturer. Judy uses an XML editor which is also an AtomPub client to post the EO to the remote content repository (an AtomPub server).

Distributed Aircraft Manufacturing. Future Composites Inc. is a supplier of composite aircraft structures to X-Aero, a major airframer (systems integrator). Future Composites is also responsible for providing technical content in S1000D to X-Aero on those composite structures. After a failed attempt to connect to X-Aero's repository using their SOAP and WS-* interface, Future Composites and X-Aero mutually agree to go back to the basics and use AtomPub and its simple and generic RESTfull HTTP interface to CRUD (create, retrieve, update, and delete) documents to X-Aero's content repository.

The main argument in favor of this approach is simplicity and scalability. I am glad to see that the software industry is moving in that direction. Having been involved in complex WS-*-based integration projects in the airline industry, I believe this new approach is a breath of fresh air. The RESTful approach is also more amenable to agile software development as opposed to the waterfall approach which is typical when the big up front purchase of a proprietary ESB is involved.

Integration projects are becoming critical to the success of new aircraft projects. Speaking about the repeated postponement of the 787 maiden flight in an internal memo send to Boeing employees on April 21, 2008 (and obtained by the Seattle Times) Boeing CEO Jim McNerney wrote:
I expect we’ll modify our approach somewhat on future programs—possibly drawing the lines in different places with regard to what we ask our partners to do, but also sharpening our tools for overseeing overall supply chain activities.

Why AtomPub specifically? Because too many people have been putting the "REST" label on their unRESTful chef-d'oeuvre (HTTP APIs) lately. AtomPub is a good embodiment of the principles of the REST architectural style and a good place to start.

So, what are the key principles of RESTful design:

Everything is a URI addressable resource
Representations (media types such as XHTML, JSON, and Atom) describe resources and use links to describe the relationships between those resources.
These links drive changes in application state (hence Representational State Transfer or REST).
The only type that is significant for clients is the representation media type, not any other resource type
URI templates (a la OpenSearch) as opposed to fixed or hard coded resource names
Generic HTTP methods (no RPC-style overloaded POST)
Stalessness (the server keeps no state information)
Cacheability

Adherence to these principles is what drives massive scalability. Security in a RESTful application can be achieved with any of the following existing solutions:

XML Signature and Encryption
OpenID
HTTP Authentication
SSL

How does the aviation industry gets started with this new approach? This will require leadership from aviation IT specialists, particularly from original equipment manufacturers (OEMs). I don't think that another Air Transport Association (ATA) standard committee is needed. Such standard committees are plagued by vendor politics. By the time they finish their work, someone may have invented a better solution than AtomPub.

In the Java space, the recently approved Java API for RESTful Web Services (JAX-RS) specification greatly simplifies REST development with simple annotated POJOs. Jersey is the open source reference implementation of JAX-RS. Apache Abdera is an AtomPub implementation with Spring Framework integration. The latest release of Abdera features a collection of pre-bundled Atom Publishing Protocol adapters for JDBC, JCR, and filesystems.

The following is an excellent technical article from InfoQ that explains how REST and AtomPub facilitate integration: "How to Get a Cup of Coffee".

Recession-Proof Computing

2008-08-11T11:45:00.031-04:00

The past three years have been very exciting at Efasoft. We've delivered value to a number of customers in a variety of industries including automotive, pharmaceutical, homeland security, wireless internet, aerospace, defense, insurance, and customer loyalty management. We've also learned a lot in the process. These projects have allowed us to strengthen our expertise in XML, Java EE, and SOA. In the aerospace vertical where we have a strong expertise, we took the initiative to propose new ideas such as using ISO Schematron to exchange and validate S1000D business rules and the AtomPub protocol and Atom syndication for the efficient exchange of up-to-date aircraft technical data between airlines and aerospace manufacturers.

Going forward, our objective will continue to be the success of our customers. We'll achieve that by researching and implementing best practices as always. We understand that technology must be aligned with strategic business goals, but should also take into consideration the context within which the business operates.

So, key questions that a lot of business leaders are now asking include:

How can we continue to invest in much needed strategic IT initiatives in the current economic downturn under tight budgets?
Given the high rate of IT project failures, how do we minimize risk?
Which software development methodology can help us deliver quality software on time and under budget?
What are the tools that we can use to help our developers in their work and keep them productive?
Is outsourcing and offshoring the right approach? And if we do outsource, how do we keep control, quality, and our intellectual property?

Open source software (OSS) is the answer to some of these questions. OSS lowers the total cost of ownership (TCO). By providing full access to the source code (including unit and functional tests), OSS provides transparency into enterprise software. By supporting standards and open frameworks, OSS allows organizations to avoid vendor lock-in, protect their investments, and find talent in the open job market to maintain and support their software assets in the future (in case the software vendor goes out of business).

SOA and Web 2.0 technologies allow organizations to gain a competitive advantage by supporting business process efficiency and by facilitating collaboration and online communities.

On the SOA front, OSS tools such as Apache CXF (web services framework), Apache Tucsany (SCA implementation), Intallio BPMS (BPMN), Apache Axis2, Mule ESB, Apache ODE (BPEL), and Apache ServiceMix (ESB) have demonstrated their strength in supporting SOA projects in mission critical applications in industries such as banking. Based on carefully researched SOA design principles and patterns, our SOA offering includes the following:

Business process analysis using BPMN
A model driven development (MDD) approach where appropriate
SOA implementation using emerging standards such as BPEL and SCA (Service Component Architecture)
SOA Governance using open source SOA Repositories.

On the Web 2.0 front, we really like the Liferay enterprise portal and the Alfresco document/web content management platforms particularly their built-in social networking features which enable enterprise collaboration and online communities. Document management is one area where we can leverage our expertise in XML and related technologies (XInclude, XSLT, XQuery, XForms, ISO Schematron, and S1000D) to help our customers bring their knowledge assets under control. We'll continue to support the Exist XQuery-enabled native XML database to build dynamic XML content applications. The XRX (XForms, REST, XQuery) architecture with Exist and the Orbeon XForms engine enables what we call "Web 2.0 XML authoring and Publishing". We've acquired a strong expertise in document management for maintenance and operation documentation in the aerospace industry (our traditional forte) and drug related documentation in the pharmaceutical industry.

JBoss Seam is a very compelling application development framework because it not only brings together Java EE frameworks such as Hibernate, JPA, EJB 3, Spring, JSF, Facelets, and Java portlets, but also integrates human workflow capabilities (jBPM), full-text search (Hibernate Search), a business rules engine (Drools), and an integration testing facility. We like the ability to leverage third party AJAX-enabled JSF component libraries such as ICEFaces and Apache MyFaces to quickly create rich internet applications (RIA). However, JBoss Seam is not limited to JSF and can also integrate Flex 3 front-ends.

Going forward, all these open source software will be part of our toolkit as we craft innovative software solutions for our customers.

At Efasoft, we are proponents of agile development methodologies such as Extreme Programming and Scrum. These methodologies are based on practices such as user stories, iteration (sprint) planning, pair programming, unit test first, refactoring, continuous integration, and acceptance tests. Agile programming helps create better software that is also easier to maintain. We've witnessed the success of agile first hand and believe that it can help IT organizations achieve success. For more on Efasoft's approach to quality, see my previous blog Addressing Software Quality Head-On.

Architecting SOA Solutions with a Model Driven Development (MDD) Approach

2008-07-26T07:48:00.014-04:00

How do you architect an SOA solution to ensure that it is driven by the business and can respond rapidly and efficiently to ever changing business requirements? A Model Driven Development (MDD) approach can help provide that level of agility. The goal with the MDD approach to SOA is to auto-generate service artifacts such as WSDL, XSD, SCA composites, and BPEL code from the service model.

First, the business articulates their vision for the SOA project in requirements documents or in the form of use cases. Business analysts (BAs) then model business processes that realize the use cases by leveraging the Business Process Modeling Notation (BPMN). With the help of the right tools, the BAs can specify Key Performance Indicators (KPI) such as those required by service level agreements (SLAs). They can also run simulations to validate the proposed business processes.

SOA is indeed all about reengineering and supporting organizational business processes. Back in 1993, Michael Hammer and James Champy made the case and outlined the management framework for reengineering in their book entitled "Reengineering the Corporation: A Manifesto for Business Revolution". Today, SOA is the software architecture that enables and facilitates the reengineering of business processes.

BPMN is an effective tool for BAs (as opposed to UML) because they should only focus on the business and operational aspects of the business process and shouldn’t have to worry about IT concerns such as service loose coupling, reusability, reliability, security, persistence, and transactions. While direct transformation from BPMN to executable BPEL code (so called BPMN-BPEL round tripping) may be effective for simple business processes, it can not always satisfy those IT concerns. More complex business processes will require advanced modeling and coding by SOA architects and developers.

For example, the SOA architect will have to decompose the proposed business process into task, entity, and utility service layers in order to satisfy the SOA principles of loose coupling, reusability, and composability. That will also give the SOA Architect opportunity to apply SOA design principles and patterns and check the enterprise SOA Repository or Registry to reuse existing services or legacy assets.

After decomposing the proposed business process to identify reuse opportunities and address other IT concerns, the SOA architect can then build an assembly of service components based on the Service Component Architecture (SCA). SCA implementation types can be Spring beans, EJBs, C++, Cobol, WS-BPEL, PHP, Spring, XSLT, XQuery, and OSGi bundles. SCA supports different bindings such as SOAP/HTTP Web services, JMS, RSS, and Atom.

The tool of choice for software architects is UML 2.0. In the case of SOA, UML can help abstract the service model from technology-specific implementation details. Basic UML artifacts such as activity and collaboration diagrams can be auto-generated from the BPMN diagrams produced by the BAs to bootstrap the SOA Architect's modeling effort.

To help SOA architects in crafting service-oriented solution logic, a UML Profile for service-oriented design should be adopted. The profile should define a number of stereotypes that can be applied to UML artifacts in order to refine the transformation from UML artifacts to service artifacts.

The automatic generation of the service artifacts from the UML model should be part of the build and continuous integration process which should also include automated tests (for example to ensure that the generated XSD and WSDL are syntactically correct, WSI-BP compliant, and backward compatible).

The benefits of an MDD approach to crafting SOA solutions include: increased development productivity, traceability to business requirements, responsiveness to changing requirements, quality, and overall agility.

SOA in the Java Space: State of the Union

2008-07-09T18:23:00.012-04:00

In the Java space, an SOA Architect starting a new SOA project will have to make some strategic as well as tactical decisions regarding which approach and technologies are appropriate for their project. Of course, the SOA project should be aligned with the organization’s long term business goals.

Technically, there is a myriad of specifications to choose from:

The Business Process Modeling Notation (BPMN)
The Java API for XML Web Services (JAX-WS)
The WS-* specifications including the WS-I Basic Profile, WS-Addressing, WS-Policy, WS-Reliable Messaging, and WS-Security
The Java API for RESTful Web Services (JAX-RS)
The Java Business Integration (JBI)
The Web Services Business Process Execution Language (WS-BPEL)
The Service Component Architecture (SCA).

Each of these specifications has its raison d’etre and should be part of the architect’s toolkit. However, I find SCA quite intriguing. SCA defines a language neutral programming model for the assembly and deployment of services. SCA implementation types include Java, C++, Cobol, WS-BPEL, PHP, Spring, XSLT, XQuery, and OSGi bundles. SCA supports different bindings such as SOAP/HTTP Web services, JMS, RSS, and Atom. SCA applications can be hosted in web containers, application servers, and OSGi runtimes. SCA is geared toward the developer and can apply policies such as reliability, security, and transactions to services in a declarative manner. SCA has the support of big players including Oracle, SAP, and IBM. At the time of this writing, Sun Microsystems support for SCA is less than clear (to me anyway).

JBI is implemented by a number of Enterprise Service Bus (ESB) products. JBI defines a runtime architecture that allows plugins such as binding components and service engines to interoperate via a Normalized Message Router (NMR). Binding components (BCs) use communication protocols such as JMS, FTP, XMPP, and HTTP/S to connect external services to the JBI environment. Service engines provide application logic in the JBI environment. Examples of SEs are XSLT/XQuery data transformation engines, rules engines, and WS-BPEL engines. BCs and SEs do not communicate directly. They only communicate through the NMR. IBM, BEA (now part of Oracle) and SAP did not vote in favor of JBI Java Specification Request (JSR 208).

When starting a new SOA project, SOA architects will have to look beyond vendors politics and make some judgment calls about the best approach based on their business goals and functional requirements.

My personal take on this is to adopt an agile approach where new functionalities are implemented in an iterative manner. For example, instead of starting with an ESB infrastructure, a project can start by service enabling existing applications (code-first approach) with JAX-WS annotation capabilities or by creating new services with a contract-first approach where JAX-WS annotated service and server stub are generated from a WSDL. Alternatively, the Java API for RESTfulWeb Services (WS-RS) could be used when a RESTful approach seems more appropriate.

An ESB could later come into the picture in a context where you are connecting to multiple services (often across organizational boundaries) and there is a need for mediation services such as business process orchestration, business rules processing, data model transformation, message routing, and protocol bridging. In that context, JBI provides plug-and-play functionality in ESBs for service engines such as business rules engines and BPEL engines. For that reason, JBI can help avoid ESB vendor lockin (perhaps a reason why proprietary ESB vendors are not backing JBI).

However, SOA architects should carefully consider the benefits of the programming language and binding agnostic service assembly model proposed by SCA. This will be essentially a choice between centralized mediation and decentralized assembly. The Open Service Oriented Architecture (OSOA) group believes that JBI and SCA can actually work together for example to allow SCA components to call JBI components or to use JBI runtime containers to deploy SCA composites.

Decentralized assembly is agile and looks a lot more like the way the web itself works. So I believe that while the JBI model is fine for integrating legacy enterprise applications, new and future service-oriented applications will embrace the SCA approach.

So is the state of the union strong? There is certainly a risk of fragmentation. But choice will always drive innovation forward and that’s what attracts me to the Java platform in the first place.

XML Schema Design Strategies for SOA Projects

2008-07-04T23:20:00.023-04:00

The schema modeling and design effort should be an integrated part of an agile approach which implements practices such as user stories, acceptance tests, unit test first, refactoring, short iterations, common code base, and continuous integration.

In the recommended contract-first approach to web services development, the XML Schema and WSDL artifacts are the foundation of an SOA project. For example, with Apache CXF, you can use the WSDL2Java tool to automatically generate a JAX-WS annotated service and server stub from your WSDL.

The first step is to adopt schema naming and design rules (NDR). Industry and government standard bodies like the Universal Business Language (UBL) and the National Information Exchange Model (NIEM) have published such NDRs.

If you’re building a new schema from scratch, then the schema should be designed in an iterative and collaborative manner. During each iteration, add just enough components to your schema to support the specific user stories that are being implemented. As the schema grows, refactor as required.

One option is to start the modeling effort with a domain model in the form of UML class diagrams to facilitate collaboration between non-technical subject matter experts (SMEs), the modeler, and the technical team. An XML schema can then be generated automatically from the UML class diagram with a tool such as Hypermodel. David Carlson, creator of Hypermodel, proposed a UML Profile for XSDs which defined a number of stereotypes that can be added to UML class diagrams to refine the mapping from class diagrams to XML schemas. Alternatively, you could export the UML model to the XML Metadata Interchange (XMI) format and use an XSLT transform to map the XMI into an XML Schema and even an XML instance. This Model Driven Development (MDD) approach to XSD provides agility in the face of constant changes in business requirements.

If you are reusing an industry or government schema such as UBL or NIEM, it is very important to use the right methodology for extending the schema as recommended by the applicable NDR or Information Exchange Package Documentation (IEPD) process in the case of NIEM. Extensions to the standard schema should be clearly defined in a new custom namespace and documented properly. The following are some strategies for extending an XML schema:

Wildcards xs:any and xs:anyAttribute
Element substitution and abstract elements
Type substitution via xsi:type and abstract types
Concrete Extension (creating a new type by extending an existing type to include additional local elements).

The schema should be tested for quality against the NDRs. The US National Institute of Standards and Technology (NIST) has published an XML Schema Quality of Design Tool (or QoD Tool) which combines Schematron and JESS rules (a Java-based open source rule engine) to validate schemas against NDRs.

For unit testing, the XMLUnit framework can be helpful in testing the schema as you refactor and implement new user stories. XMLUnit for Java allows you to make assertions about the validity of an XML document against an XML Schema. The execution of these tests should be part of your build and continuous integration process.

The automatic generation of the XSD code from the UML model should also be part of the build and continuous integration process.

Business rules or modeling requirements that are beyond the capabilities of XML Schema 1.0 should be implemented with an assertion-based language such as ISO Schematron or with the new assertion and conditional type assignment (co-constraints) capabilities in XML Schema 1.1.

Although ISO Schematron and XML Schema 1.1 can be quite powerful when used with XPath 2.0, some complex business rules will be easier to handle with rule engines such as JBoss Drools and JESS. The rule engine can be deployed as a dedicated and reusable utility web service to validate messages.

When modeling XML schemas for an SOA project, careful consideration should be given to the issue of data transformations. When the services don’t share the same data model and XML Schema, there is a need to transform the XML data using technologies such as XQuery and XSLT. This can introduce additional design complexity and runtime performance issues. Data transformations shall be avoided unless absolutely required.

The XML Schema's xsd:appinfo element can be used to capture and keep metadata close to the XSD declarations:

Metadata such as data transformation specifications
Business rules using inline ISO Schematron rules
Labels, alerts, and appearances of UI components such as XForms controls. This provides the opportunity to auto-generate UI components from your XSD using a transformation language like XSLT or XQuery. Keeping UI and XSD components in sync can be a challenge in SOA projects.

Finally, to maximize reuse, an enterprise-wide SOA Repository/Registry should be used to publish, centralize, and discover schema components (see my previous post on SOA Governance tools).

Addressing Software Quality Head-On

2008-06-10T18:48:00.023-04:00

Adopting a test-driven development methodology (TDD) and using the right tools can help deliver quality software.

With TDD, you start with user stories and you write acceptance tests for those stories. To implement the functionality for the stories, you write unit tests first, then just enough code to make the unit tests pass, and then you refactor. You repeat the test-code-refactor cycle until the acceptance tests pass.

If your development team is not already using TDD, the first step is to provide adequate training on the concepts and patterns of TDD. One option is to hire an Agile coach or pair junior developers with experienced practitioners of TDD. Next, you need to use the frameworks and tools that will facilitate adoption, keep your developers productive, and provide transparency into the process. The following are tools and frameworks (all are free and open-source) that we find useful:

Build tool: Maven 2
Continuous integration: Hudson
A tool for configuring, starting, stopping Java containers and deploying applications for continuous integration and functional tests: Cargo
Unit testing frameworks: JUnit 4.4, EasyMock, and JSFUnit (for JSF applications)
Enforcing coding standards: Checkstyle
Detecting bugs, overcomplicated expressions, and suboptimal/dead/duplicate code: PMD and FindBugs
Code review: Jupiter
Test coverage: Cobertura
Web Services functional and load testing: SoapUI
User interface testing: Selenium and Umangite
Integration testing: Fit, DBUnit, DbFit, and ORMUnit
XML unit testing: XMLUnit and Tennison Test
Load and performance testing: JMeter
Profiling and monitoring: JConsole
Analyzing code dependencies: JDepend
Source code documentation: Doxygen
Project tracking and planning: XPlanner

By enforcing Java best practices, tools like Checkstyle, PMD, and FindBugs are very helpful for developing high quality code.

Unit and integration tests should be fast, repeatable, and automated. That's why you need to deploy a build tool and a continuous integration server from the start. All the tools listed above should be executed as part of your continuous integration process. You should use a dedicated integration build machine.

DBUnit helps make your system tests repeatable by using XML to insert a specific data set into the database before each test run. Cargo can configure and starts the web container or application server (AS), deploy your application's WAR or EAR file, and then shutdown the container after each test run.

Keep in mind that test coverage reports such as those provided by Cobertura should be used mainly to isolate code that has not been appropriately tested in order to take corrective measures. These reports shall not be used solely to aim at a magical high coverage percentage. Applications that are built on an inversion of control (IoC) container such as Spring are more amenable to unit testing.

For web applications with a user interface (UI) layer such as Struts or JSF, make sure that you exercise the UI functionalities on real web browsers (IE, Firefox, etc.) with a tool like Selenium. Fit and Fitness (the wiki-based version of Fit) are effective tools for integration testing your application's business logic. By integrating Selenium, TestNG, Spring, and Cargo, the Umangite framework makes it easy to write web tests.

If you intend to service-enable the same application, then SoapUI will help with both integration and load testing of the web services over HTTP. The nice thing about SoapUI is that in addition to XML Schema validation, it allows you to specify response assertions by using XPath 2.0, XQuery, and Groovy scripts.

Sun JDK tools such as jConsole, jmap, and jhat can help with profiling and diagnosing memory leak issues. The JMeter Proxy makes load and performance tests easy my allowing you to record a test case.

Are agile practices applicable to XML development (XML Schema modeling, XSLT, XQuery, and XSL FO programming)? You bet. More on that on my previous blog entitled Extreme XML Programming.

S1000D Content Reuse for Aircraft Documentation

2008-05-24T21:09:00.030-04:00

One of the justifications for moving to an XML-based S1000D content management system (CMS) is the ability to reduce cost and improve quality by reusing content. In the aerospace industry, hundreds of thousands of pages of maintenance and operation documentation are produced and maintained for every new aircraft project. Warnings and cautions are a good example of reuse in aerospace documentation. They describe hazards that may cause injury or death or damage to the aircraft. For product liability reasons, these warnings and cautions are carefully reviewed and approved by qualified personnel. Technical authors may be required to reuse these warnings and cautions verbatim across all documents. In this blog, I will discuss some principles and practices that facilitate S1000D content reuse.

From a technical perspective, the key to successful reuse in S1000D is the W3C XInclude specification. The S1000D specification does not make reference to XInclude. The reason is that earlier versions of S1000D were based on SGML. Some S1000D CMS still rely on the SGML/XML 1.0 external parsed entity mechanism for implementing reuse. This approach has several limitations and should be avoided. The preferred approach in modern XML content applications is to use XInclude which allows the transclusion of not only whole chunks of XML content, but also elements (addressed using XPath/XPointer) within those chunks. The following are some examples:

<xi:include href="dm.xml"/>
<xi:include href="dm.xml" xpointer="warning-001"/>

In the first example a data module file named dm.xml is included. In the second example, an element with ID value "waning-001" within the data module is included.

Using XInclude in an S1000D content application requires some modifications to the XML Schema used for the authoring of data modules to allow the insertion of xi:include elements. However, these modifications will still produce valid S1000D documents since you're not altering the structure of your documents, but rather simply modularizing the content.

While we are on the subject of inclusion, the XLink specification can be used as a simpler alternative to the XML 1.0 unparsed entity and notation mechanism (another concept inherited from SGML) for including illustrations into S1000D documents.

At the DocTrain 2007 conference in Boston, I gave a presentation on how to integrate training and documentation using S1000D and the Shareable Content Object Reference Model (SCORM) specification. One way to reuse S1000D content in SCORM is to assign a unique ID to all elements in S1000D data modules (DMs) that are reusable such as paragraphs, steps, warning, cautions, notes, tables, etc. This can be done automatically using the XSLT generate-id() function. The instructional designer then searches the S1000D common source database (CSDB) to find and display relevant DMs. She can then use XInclude to include reusable elements from S1000D DMs into SCORM shareabe content objects (SCOs). When this is done, the SCOs are automatically updated when the DMs are updated.

Successful S1000D reuse requires adherence to the principle of context-agnostic content. For example, to make it possible to reuse a warning across multiple documents in different contexts, one should avoid formulations such as "refer to the illustration in the next section" inside the warning.

Enforcing the principle of context-agnostic content can be semi-automated using an assertion-based schema language like ISO Schematron to report the occurrence of keywords such as "previous", "next", "below", etc. The warning shall be routed through a comprehensive review and approval workflow provided by the CMS before final publication. The principle of business rules definitions and enforcement ensures that reusable content is of the highest quality. Consider a dual-purpose data module that is written to be reused by both training and publications. A business rule could require the use of a certain language style (e.g. active as opposed to passive voice) for the dual-purpose data module.

Another principle that can help when the content cannot be context-agnostic, is the parameterization of reusable content. With parameterization, you include variable references in the reusable content that are resolved at run time. The Exist XML database has an elegant way of handling this using a combination of XInclude and XQuery as in the following example:

<xi:include href="warning.xq?var1=material&var2=process"/>

Here warning.xq is a stored XQuery witch is compiled and executed by Exist to return the root element of the warning. The content of the warning depends on the material and process used to carry out the maintenance procedure. var1 and var2 are passed as global external variables to the XQuery.

The issue of content granularity is directly related to the principle of context-agnostic content. Although the data module is the basic unit of information in S1000D, content can be managed at a lower level of granularity. An interested feature of some XML editors is the ability to select an element inside an XML document and convert that element into an XIncluded file. So while a technical author is writing a warning inside a data module, she can pull out that warning as an XIncluded XML file if she determines that the warning could be reusable in other publications.

Another area where XQuery facilitates reuse is the dynamic assembly of content based on product attributes such as applicability, security, and skill level. S1000D has a comprehensive metadata facility called IDSTATUS that can be leveraged to filter content. A good example is applicability filtering. In the case of an aircraft, the applicability of an S1000D maintenance or operation procedure can depend on the following attributes and conditions (among others):

Manufacturer serial number
Aircraft registration number
Service bulletin incorporation
Location of maintenance
Aviation regulations
Temperature, wind speed, and sandy conditions.

XInclude and XQuery can be used together to package content into S1000D publication modules by executing queries that filter content based on metadata in the IDSTATUS.

An important condition for content reuse is the principle of discoverability of reusable content. Obviously, you cannot reuse a piece of content if you don't know that it exists and where to find it. A technical author should be able to query or browse the S1000D CSDB (Common Source Data Base) to find relevant reusable content. To facilitate enterprise-wide content reuse, I highly recommend a CSDB based on a native XQuery-compliant XML database and deployed as a web application. That will allow authors to perform both full-text and structured queries on the CSDB. The query should return a list of data modules or reusable chunks. The author should then be able to select the reusable chunk to automatically insert an XInclude targeting that chunk.

In support of the principle of reusable content discoverability, appropriate metadata should be added to the content. The DMs already have comprehensive metadata in the IDSTATUS section. Reusable content at a lower level of granularity (like a warning) should also have appropriate metadata specified.

An XQuery-enabled native XML database can help with the governance of your reuse initiative by providing powerful reporting capabilities. For example, you can easily run an XQuery to find all documents that contain an XInclude to a particular chunk. This is important for understanding the impact of updates to that chunk. Another potential issue that could require some attention is the versionning of reusable content. Some form of notification mechanism can be helpful to alert consumers to changes to reusable content. This can take the form of an Atom feed to which consumers can subscribe.

It is important to select an XML authoring tool that has good support for XInclude. Fortunately, some commercial XML editors now have decent support for XInclude. However, these XML editors remain complex specialist tools that are often used only by professional technical authors in documentation departments. At one of our aerospace customers, manufacturing assembly and functional test procedures were used to create installation and testing procedures for service publications. To allow their engineers to contribute S1000D content, we designed a light XML authoring application based on an XForms front-end with XML data persisted in a native XML database using a RESTful API.

Any data reuse strategy should look beyond training and publication to identify opportunities to reuse data and streamline processes across the entire aircraft lifecycle.

Spring, SCA, and OSGi

2008-05-15T23:06:00.004-04:00

French paleontologist Pierre Teilhard de Chardin once said: “Tout Ce Qui Monte Converge” or “Everything That Rises Must Converge”. I learned this quotation from my father who mentioned to me that it was the topic of his philosophy dissertation at his university entrance exam.

The quotation accurately describes what I see happening in the world of software development with the rise of Spring, OSGi, and SCA.

During the last 30 years, the software industry has evolved from structured design to object-oriented design, POJO programming, and lately service-oriented design. With the rising complexity and costs of software systems, the ultimate goal of this evolution has been the reuse of software assets through loose coupling and service orientation.

Spring is based on the principle of inversion of control (IoC) or dependency injection. With Spring, objects (simple POJOs) are provided with their dependencies as opposed to the objects managing or looking up those dependencies themselves. Spring relies on aspect-oriented programming (AOP) to declaratively manage cross-cutting concerns such as security, transaction, and logging. From a quality and agile development perspective, one big advantage of Spring-based applications is that they are amenable to unit testing using frameworks such as JUnit or EasyMock.

Service-oriented architecture (SOA) exposes application business logic as a set of services that are remotely accessible and reusable across platforms and programming languages. The Service Component Architecture (SCA) has been designed to facilitate service composition. The SCA Assembly Model consists of one or more service components. Service components provide business functions to other components within or outside the module. A composite contains one or more service components and specifies communication bindings and policies such as security and transactions. Like in Spring, the artifacts and the dependencies between them are described using XML.

The Open Services Gateway initiative (OSGi) defines a dynamic service model where components (packaged as bundles) and their dependencies are specified in a service registry. OSGi standardizes the life cycle management of these bundles including deployment, installation/uninstallation, and updates with full versioning capabilities. Bundles can be dynamically started, stopped, or updated without the need for a reboot. OSGi defines a model for publishing, discovering, and binding to services within the same virtual machine (VM).

Spring, SCA, and OSGi are converging to create an environment that facilitates the design and the lifecycle management of software assets that are exposed as reusable services. Software vendors are actively exploring different opportunities to combine these three technologies. The combination of Spring, SCA, and OSGi is already having a transformative impact not only on service-oriented design and application development in general, but also on the application servers and middleware market as well.

SOA and ROA Design Principles and Patterns

2008-05-04T11:44:00.010-04:00

I've been compiling a list of design patterns and anti-patterns on Service Oriented Architecture (SOA) and Resource Oriented Architecture (ROA). I find the following resources quite useful.

If you're looking for design patterns in building RESTful applications, the best way to start is to look at the Atom Publishing Protocol (AtomPub) which is a good embodiment of the principles of the REST architectural style. The Google Data API (GData) is a real world implementation of AtomPub. At the XML 2007 Conference, I've proposed a RESTful approach to aviation technical data management called "Integrated Documentation Environment for Aircraft Support (IDEAS)" (more on that on my previous blog RESTful IDEAS).

Another good resource is the book "RESTful Web Services" by Leonard Richardson and Sam Ruby. Chapter 8 entitled "REST and ROA Best Practices" is a must read and also addresses potential REST implementation issues such as asynchronous operations and transactions. Chapter 10 entitled "The Resource-Oriented Architecture Versus Big Web Services" offers ROA alternatives to WS-* specifications.

For SOA design patterns and anti-patterns, here are some useful resources:

SOA Antipatterns: an article by a group of IBM SOA architects
SOA anti-Patterns: an article by Steve Jones at InfoQ
SOA Patterns: the first draft of Thomas Erl's upcoming book entitled "SOA Design Patterns"
Pete Lacey Criticizes Web Services: an interesting discussion on REST vs. WS-*.

I don't believe that ROA is the answer to all SOA project failures out there. However, I do believe that certain requirements and use cases are more amenable to the REST architectural style (more on that in a future post).

SOA Governance Tools

2008-04-13T14:39:00.017-04:00

First, an important caveat: SOA governance is not a tool or a product. SOA governance is about people and leadership. No tool will deliver good governance out the box if the human factor is not taken into consideration. However the right tool can facilitate and provide transparency into SOA governance.

It’s important to make a distinction between design-time SOA governance and run-time SOA governance. The main objective of run-time SOA governance is the enforcement of QoS and SLAs. Design-time governance focuses on the enforcement of industry-recognized SOA design principles and patterns.

The goal of these patterns is not to kill the creativity of SOA developers or police their work, but instead to avoid SOA anti-patterns that are known to undermine the success of SOA projects. For example, SOA developers cannot reuse services if they are not aware of the existence of these services in the enterprise. Even if they know these services exist and where to find them, they cannot reuse the services if they don't understand them. Therefore the ability to easily discover well specified service metadata is one good design principle that can help deliver on SOA's promise of service reuse across the enterprise.

One of the key aspects of design-time SOA Governance is the management of the lifecycle of service artifacts and the dependencies between them. This is accomplished through a new breed of tools called SOA Repositories/Registries. The following are what I consider important requirements for an SOA Repository/Registry.

Indexing of XML-formatted artifacts such as WSDL, XML Schemas, Schematron rules, XSLT transforms, Spring and Hibernate configuration files, data mapping specifications, WS-Policy documents, etc. Ideally, users should be able to use languages such as XSLT and XQuery to manipulate artifacts and query the Repository/Registry. Requirements and specification documents should be stored in XML (as opposed to MS Word or Excel) if possible so that they can be processed the same way. The SOA Repository/Registry should sit on a native XQuery compliant database. This would provide powerful visualization and reporting capabilities to the registry. I should be able to run an XQuery search against the SOA Repository/Registry to return all artifacts that contain a reference to a certain XML element, so that I can visualize the impact that a change to that element would have. Automatic detection of certain dependencies (e.g. WSDL and XML Schemas) should be supported as well.

Policy enforcement is also important as artifacts are added to the registry. For example, Schematron can be used to enforce XML Schema best practices. WS-I Basic Profile compliance and XML schema backward compatibility may also need to be enforced.

Support for a RESTful API for all CRUD (create, read, update, delete) operations on artifacts. Ideally, I would prefer support for the AtomPub specification and the Atom syndication format for pushing updates to stakeholders. Competing SOA Registry protocols and APIs include: UDDI v3, the Java API for XML Registry (JAXR), JSR 170/283 (Java Content Repository API), and IBM WebSphere Registry and Repository (WSRR). However, Open source vendors such as MuleSource and WSO2 have adopted AtomPub for its simplicity, while RedHat is building its upcoming SOA Repository/Registry (JBoss DNA) on JSR 170. Mule Galaxy sits on the Apache Jackrabbit JCR reposiroty. The JSR 170 repository model could also be adopted in the Java space as a standardized repository model for SOA registries.

Of course authentication, authorization, audit trails, workflow, and versioning should be expected in any SOA Repository/Registry.

Passportgate: what would you do about it?

2008-03-23T17:19:00.009-04:00

The current Passportgate scandal in the US involving the unauthorized access to the passport files of the three presidential candidates got me thinking about information security in enterprise applications particularly records and content management systems.

Ensuring information security requires a multidimensional approach based on technology, process, policy, and governance. Technology alone is not the answer. However, since this is a technology-oriented blog, I will focus only on the state of the art in securing Java EE applications particularly in the open source space.

From a technology standpoint, I see at least four potential issues: authentication, authorization/access control, audit trail, and business process.

Spring Security (formally Acegi) has demonstrated its strength for both authentication and authorization in Spring-based portal and content/record management applications. Spring AOP (Aspect-Oriented Programming) provides an elegant and simple solution for audit trails in such systems.

JBoss jBPM is a robust BPM engine that meets the requirements for workflow and enterprise business process orchestration between applications, services, and people.

The eXtensible Access Control Markup Language (XACML) is an OASIS standard for specifying access control policies in XML. XACML is not currently widely used in content/record management systems. One explanation is that XACML has been designed to provide access control for new services such as web services in service-oriented architectures (SOA). XACML would be challenging to use for document-level security in content repositories that have a hierarchical structure (e.g. JSR 170/283 repository model) and demand sophisticated caching for scalable and rapid access to massive amounts of content.

However, my favorite XML database (eXist) has an elegant implementation of XACML for controlling access to resources such as XQuery modules and Java methods, proving once again that Open Source is ahead in terms of innovation in the software industry.

Boeing 787 Flight Control Software and Composite Fuselage Tested

2008-03-04T20:51:00.012-05:00

Randy Tinseth, Boeing Commercial Airplanes vice-president for marketing announced on his blog that Boeing chief pilot Mike Carriker and 787 systems director Mike Sinnett successfully tested the flight-control Blockpoint 8 software code in the 787 engineering flight simulator. He wrote:

During the test, Mike and Mike demonstrated most of the operational procedures required by a flight crew - pushback and engine start at Sea-Tac airport near Seattle, taxi and takeoff, climb, cruise, simulated engine failure, descent, approach, single-engine go-around, landing, taxi and arrival at the gate at the Portland, Oregon airport.

Boeing also performed a serie of test on the composite fuselage of the B787 including "limit load", "ultimate load", and beyond 2.5 times the normal force of gravity (2.5 G). According to a Boeing press release dated 02/28/2008:

Testers observed audible indications of damage as the test progressed but the piece did not reach the level of destruction that had been anticipated.

This is a significant development because last September, Boeing announced the delay of the maiden flight of the first 787 due to flight control software issues, fastener shortage, and supply chain bottleneck. Also last year, a former Boeing engineer went public with concerns about the survivability of the 787 composite structure in case of a crash.

The maiden flight has been postponed again to June this year. First delivery to launch customer All Nippon Airways is scheduled for early 2009.

UPDATE: On April 9, 2008, Boeing has announced that 787 maiden flight has been postponed to the 4th quarter of 2008 and the first delivery for the 3rd quarter of 2009.

Speaking about the 787 globally distributed aircraft manufacturing model in an internal memo send to Boeing employees on April 21, 2008, (and obtained by the Seattle Times) Boeing CEO Jim McNerney noted:
I expect we’ll modify our approach somewhat on future programs—possibly drawing the lines in different places with regard to what we ask our partners to do, but also sharpening our tools for overseeing overall supply chain activities.

Declarative Programming Recipes

2008-02-29T22:00:00.018-05:00

If you've been using an imperative programming language such as Java, C#, or JavaScript, then you should consider the benefits of declarative programming when designing your next application. In this blog, I will explore the benefits of the following technologies:

XML Schema
ISO Schematron
XForms/XBL (XML Binding Language)
XSLT 2.0
XQuery
Atom Syndication
AtomPub

The basic difference between declarative programming languages and imperative programming languages such as C# and Java is that the former specify the “what” (the intent) as opposed to the “how” (the algorithm). The following are some reasons to consider this new paradigm:

Declarative programming languages are accessible to many non-programmers
It’s possible to create a solution that is completely declarative (no Java, C#, JavaScript, or AJAX code)
It facilitates the Model-Driven Architecture (MDA) software design approach
When the data or content is managed in XML, a declarative programming model is a superior and simpler alternative to a conventional approach based on Java Server Faces (JSF), Spring, and Hibernate. For example, you get a more robust and simpler data and business rules validation mechanism and you can forgo ORM mapping altogether. However, declarative and imperative programming can co-exist in the same solution.

The following are some recipes for taking advantage of this new paradigm:

Define your data structures and types in the XML Schema.
Use XML Schema's xsd:appinfo element to:
- Capture metadata such such as XML to RDBMS mapping information
- Capture business rules using inline ISO Schematron rules
- Control the labels, alerts, and appearances of UI components such as XForms controls.
Generate your UI by transforming the XML Schema into XForms with XSLT.
Alternatively, store your XML schema and other XML-formatted metadata artifacts in a native XML database and use XQuery to extract and manipulate the metadata during development and at run-time.
Use XBL for enabling custom UI controls for your XForms. As an example, you can integrate a rich text editor using XBL and Dojo.
Use ISO Schematron for business rules validation either post form-submission or by using XSLT to inject schematron rules into the XForms directly with the xforms:bind element.
Use the RESTful or AtomPub API to a native XQuery-enabled XML database (such as Exist) to CRUD (create, read, update, delete) the data.
If you're using SOAP-based web services, use XForms to send SOAP requests and display SOAP responses as well.
Leverage Atom syndication for pushing updates to data consumers. In general, when creating your application, think "Syndication Bus".

Many software architects and developers are not yet aware of the power of this new paradigm. However, I believe that rather than using only familiar development frameworks, it’s important to always evaluate emerging alternatives and select the best approach even it involves a learning curve.

Open Source and the Democratization of Knowledge

2008-02-02T20:29:00.000-05:00

I like Open Source Software (OSS) for a number of reasons. It lowers the cost of software and allows a larger number of organizations and individuals worldwide to make use of it. OSS is also where real innovation is currently happening in the software industry. OSS companies focus their resources on the engineering process and use the web itself as a marketing and distribution channel. This allows users to freely download and try the product and decide for themselves if it’s good for them. They can get free support from the community of users or purchase commercial support for mission critical applications. The quality of product can be quickly enhanced because of the feedback that the OSS developer gets from the potentially thousands of users who download the software.

But the most important reason why I like OSS is that it enables what I call the democratization of knowledge. Complex software like operating systems, databases, ERP, CMS, and web portals all exist today in OSS. OSS is a great equalizer because anyone can access the source code to learn or discover how such complex systems are designed. They can also contribute to the code if they wish. This is unprecedented in human history. Traditionally certain groups of people have kept their technological know-how to themselves as a competitive advantage used to dominate and control markets and/or other groups of people.

Learning From My Students

2008-01-23T20:54:00.000-05:00

For the past 10 years, I've taught a variety of technical topics in classroom settings. The subjects I've covered include XML, XML Schema, XPath, XSLT, XSL FO, SAX, DOM, XQuery, and S1000D. The participants are usually professionals who are looking to upgrade their skills. It’s gratifying to read an e-mail from a student saying how a course I’ve taught has helped them be more productive in their work or pass a certification exam.

It’s always a pleasure to share my knowledge and experience with people, but also to learn from them. When you teach a class, you must master the topic because you can't afford to always respond to students questions by saying that you will do some research and get back to them later with an answer. While preparing for the class, you need to find examples and real life scenarios to explain complex concepts to them. During the class itself, by listening to students’ questions and comments and trying to answer them, you actually discover some aspects or applications of the topic that you have not thought about before.

As the saying goes "the best way to learn is to teach".

Applicability in S1000D 3.0

2008-01-19T20:40:00.001-05:00

S1000D has a new and improved applicability mechanism based on the concepts of Applicability Cross Reference Table (ACT), Condition Cross Reference Table (CCT), and Product Cross Reference Table (PCT).

The ACT data module declares attributes of the product that are not likely to change during its life cycle such as model, series, and serial number. Examples of product attributes for a commercial aircraft include the manufacturer serial number and aircraft registration number.

The CCT data module declares technical, operational, and environmental conditions that can affect the applicability of technical content. Examples of these conditions are: service bulletin incorporation, location of maintenance, aviation regulations, temperature, wind speed, and sandy conditions.

The PCT data module lists actual physical product instances. For each product instance, the PCT specifies the values of product attributes and conditions pertaining to the product instance.

Applicability can be specified at the data module level inside the IDSTATUS or within the content of the data module at a more granular level such as a <step1> element. The ACT and the CCT are used as look up tables to lookup the relevant product attribute or condition as well as their allowed possible values. The applicability element then specifies the correct product attribute or condition identifier from the ACT or CCT and the values to test against.

The applicability information itself can be captured in human readable format for simple cases. For more complex cases, one or more assertions are used to specify the product attribute or condition to test and the values to test against. These values can be constrained with a pattern based on regular expressions as defined by the XML Schema specification.

The new S1000D applicability mechanism supports the "effectivity" requirements of civil aviation and provides capabilities that are beyond the ATA 2200 effectivity mechanism. It also facilitates the development of applicability filtering functionalties in Interactive Electronic Publications (IETPs). However, building an authoring front end that hides its complexity (regular expressions and logical operations) to the technical authors creating the content will be the key. This is also an area where well-defined business rules should be specified and enforced using a tool such as ISO Schematron.

On our Radar in 2008

2008-01-04T23:35:00.000-05:00

First, Happy New Year and Best Wishes for a Peaceful 2008!

Since this is my first blog this year, I will talk about what will be on our radar screen. The Java EE platform with JSF and the open source frameworks Spring and Hibernate will continue to be our preferred development platform for robust enterprise applications (ERP, portals and CMS). We like the ability to leverage third party AJAX-enabled JSF component libraries such as ICEFaces and Apache MyFaces to quickly create rich internet applications (RIA). JSF also allows us to target both the mobile and web delivery platforms simultaneously through the use of render kits. Spring brings sanity and simplicity into the development of complex Java EE applications and Hibernate allows us to develop database agnostic applications among other benefits.

Although we still believe Java EE is the way to go for complex enterprise applications, we will learn and embrace Ruby on Rails and Adobe Flex this year. Feel free to recommend your favorite books on these topics.

We'll continue to make innovative uses of ATOM syndication, AtomPub, FeedSync, and OpenSearch to solve our customers problems.

A new version of our favorite XML database (Exist) will be released soon and will improve performance significantly. Exist already has some support for AtomPub. Declarative programming using a combination of XForms, ISO Schematron, XSLT 2.0, XQuery, and Exist's RESTful API will deliver great value for those who are willing to experiment.

Finally on the Web 2.0 front, we'll be exploring the intersection of content management with social computing, mashups, user-generated content, and rich internet applications (RIA).

We look forward to another collaborative and productive year.

Extreme XML Programming

2007-11-17T21:59:00.004-05:00

I am a proponent of agile development methodologies such as Extreme Programming and Scrum. These methodologies are based on practices such as user stories, iteration (sprint) planning, pair programming, unit test first, refactoring, continuous integration, and acceptance test. Agile programming helps create better software that is also easier to maintain.

Test-Driven Development (TDD) is a well known practice in the Java EE world with unit testing frameworks such as JUnit, EasyMock, and JMock. With TDD, you always write the test code first before you write the functional code itself. This simple principle can also be used in developing applications with XML related languages such as XSLT 2.0, XSL FO, and XQuery. Both XSLT 2.0 and XQuery 1.0 are strongly typed languages with a rich built-in function library. They can also be used to create libraries of custom functions that perform very complex business logic. Adopting a TDD methodology can increase the quality of your XSLT 2.0 or XQuery code and make it easy for you to maintain the code as you refactor and implement new requirements.

The following are my favorite unit testing frameworks:

Tennison-test is an XSLT Unit Test framework that allows you to write your unit tests in XML, but also allows you to automate the execution of the test as part of a build and continuous integration process based on Ant.
XMLUnit 1.1 for Java allows you to make assertions about the differences between two XML documents, the result of an XSLT transformation, the evaluation of an XPath expression, and the validity of an XML document.
Apache FOP provides a LayoutEngineTestSuite which can be used to check against the "Area Tree XML" generated by FOP's XMLRenderer.
Schema-aware XSLT 2.0 and XQuery can help root out errors early. With XML Schema 1.1's assertions and conditional type assignments, this will get even more interesting.
ISO Schematron can be used to write XPath 2.0-based assertions about the output of your XSLT or XQuery program.
xchecker is an interesting testing framework based on XML Schema, XPath 2.0, XSLT 2.0, XQuery, and Relax NG.
xSpec is a Behavior Driven Development (BDD) framework by Jeni Tennison for XSLT testing.

XML IDEs today provide the ability to write XSLT and XQuery code in debug mode my setting breakpoints, stepping through code one line at the time, and inspecting the value of variables. It would be nice if they could also integrate an XSLT/XQuery unit test framework a la Eclipse IDE.

Aviation Data Management in the Web 2.0 Era

2007-11-03T18:08:00.001-04:00

I will make a presentation at the XML 2007 Conference in Boston on December 5. The title of my presentation is: RESTful IDEAS. IDEAS stands for Integrated Documentation Environment for Aircraft Support. Based on AtomPub and OpenSearch, the IDEAS Framework enables federated searches of technical content and updates via web feeds. This presentation is essentially about how Web 2.0 innovations can be leveraged to create a cost-effective, efficient, and massively scalable environment for aggregating and publishing up-to-date technical documentation to end users in the aviation industry.

So what exactly is Web 2.0? Does it mean anything at all? For me Web 2.0 represents new web functionalities such as:

Content aggregation and syndication using technologies like RSS and Atom
Social networking (e.g. Linkedin and Facebook)
User generated content with videos, pictures, blogs, wikis, and podcasts (e.g. YouTube and Flickr)
Mashups or the ability to merge data from different sources (e.g. Google Maps and Yahoo Pipes)
Rich Internet Applications (RIA) using user interface technologies such as Flex and AJAX
Web Services and web-based APIs particularly those using RESTful protocols such as AtomPub

Those are the types of functionalities that today's web users expect from web applications, and the consumers of aviation technical documents are no exception.

Atom and AtomPub are already playing a fundamental role in the Web 2.0 world. The Google Data API which also includes Google Documents is based on AtomPub and OpenSearch. This week, a group of social networking sites including hi5, LinkedIn, MySpace, Ning, Orkut, and XING released OpenSocial which is a common set of APIs for developing social applications across multiple websites. OpenSocial is also based on AtomPub.

So how can the aviation industry embrace and extend Web 2.0 innovations to solve the challenge of aggregating and presenting up-to-date technical data from multiple content sources? I will be sharing my thoughts.

See you in Boston!

S1000D and SCORM Integration

2007-10-20T19:06:00.001-04:00

I gave a presentation yesterday on S1000D and SCORM integration at the Doctrain 2007 conference in Lowell, MA. The main goal of integrating these two specifications is to reduce product life cycle costs and eliminate redundancies by streamlining business processes across the documentation and training functions. I noted that there are many opportunities for data reuse at every phase of the product life cycle including: concept, design, manufacturing, assembly, testing, delivery, and support. Documentation and training belong to the support phase and are often the only departments where content is captured in XML. This is due to the complexity and cost of current specialist XML authoring tools. XForms will allow knowledge workers to contribute knowledge assets in XML at every phase of the product lifecycle with a simple web form. I also believe that Office Open XML (OOXML) offers the opportunity to extract some value out of MS Office documents by exposing their contents to XML processing languages and tools such as XSLT 2.0 and XQuery.

Engineering data should be the trusted source of data for both publications and training. For example, product model data and engineering drawings can be used to create simulation for training and manufacturing assembly instructions can be used to create installation procedures for publications. Any data reuse strategy should look beyond training and publication to identify ways to reuse data and streamline processes across the entire product lifecycle.

The integration of SCORM and S1000D presents management and technical challenges. Since training and documentation are often two separate functions within the enterprise, the integration can have an impact on budget, processes, roles, and the organizational structure. For example, the cross-functional integrated project team is a good approach. The success of the integration will also require top leadership commitment and support. It is also important to address technical challenges such as the integration of existing content management systems (CMS) and learning management systems (LMS).

The first technical approach is to create dual purpose data modules (DMs). I highlighted the importance of clearly defining and documenting business rules, particularly for dual purpose data modules. The business rules should specify among other things the appropriate level of granularity and language style (e.g. Simplified English). These business rules should be validated with technologies such as ISO Schematron and Simplified English Checkers. Since S1000D is weak on learning content metadata, the IEEE Learning Object Metadata (LOM) specification should be used to add learning object metadata to the S1000D dual purpose DMs. It is also possible to package all training data modules as a training publication module (PM). XSLT is then used to transform the S1000D DM into SCORM sharable content objects (SCOs) and learning assets. The S1000D metadata (IDSTATUS) shall be retained in the result SCOs to facilitate product applicability filtering when the SCOs are presented to the learners with an LMS. The S1000D PM can be used to generate the SCORM manifest as well. The dual purpose S1000D DM approach does not always support complex learning interactions and good instructional design principles.

The second approach is to give complete freedom to the instructional designer to design an effective learning experience. All elements in the S1000D data modules that are reusable in SCOs are assigned a unique ID. Examples are: paragraphs, steps, warning, cautions, notes, tables, etc. This can be done automatically using the XSLT generate-id() function. The instructional designer then searches the CSDB to find and display relevant DMs. She can then use XInclude/XPointer to include reusable elements from the DM into the SCO. When this is done, the SCO is automatically updated when the DM is updated.

Future versions of the S1000D specification will incorporate change proposal forms (CPFs) that will facilitate the integration of SCORM and S1000D content.

A copy of my presentation is available here.

Guidance for the Paperless Cockpit

2007-09-23T18:52:00.002-04:00

One of the interesting applications of the Electronic Flight Bag (EFB) is electronic documents. Electronic documents allow aircraft operators to amend manufacturer’s flight operations manuals based on operator's policies and procedures and publish these manuals in electronic formats such as Adobe® Portable Document Format (PDF) and XML. Examples of these manuals are:

Flight Crew Operating Manual (FCOM)

Quick Reference Handbook (QRH)

Flight Crew Training Manual (FCTM)

Minimum Equipment List (MEL)

Fault Reporting Manuals (FRM)

Weight and Balance Manual

Dispatch Deviations Guide

US Federal Aviation Administration (FAA) Advisory Circular (AC) 120 76A “Guidelines for the Certification, Airworthiness, and Operational Approval of Electronic Flight Bag Computing Devices” specifies the design and technical criteria for the approval of the human/machine interface of EFB systems. The following is an excerpt of the EFB Operational Evaluation and Approval Job Aid used by FAA inspectors for electronic documents functionalities:

Is there a training program on how to display and interact with electronic documents? Is it adequate?
Can the crews find the material they are looking for?
Is the information organized in a way that makes sense to the crews?
Is the information arranged in a consistent way on the screen so that the crews know where to look for specific types of information?
Is it obvious when text is out of view? Is it easy to bring that text into view?
Can the crew tell where they are in relation to the full document?
Can the crew tell where they are in relation to the section of the document they are currently viewing?
Is the text of the document easy to read on the screen?
Is white space used to separate short main sections of text?
Is high priority information especially easy to read?
Are tables readable and usable?
How are especially long and complex tables handled?
Are figures readable and usable?
Can the entire figure be viewed at one time?
Can the crew zoom in to read details on the figure?
Is it easy to move quickly to specific locations (e.g., to the beginning of a section, or to recently visited locations)?
Are active regions (e.g., hyperlinks) clearly indicated?
Is it easy to move between documents quickly?
Is it easy to tell what document is currently in view?
Is there a list of available documents to choose from?
Can crews search the document electronically?
Is the search technique adequate?
If animation is supported, does the crew have adequate control over it?
Can the crew start and stop the animation as needed?
Is there a text description of the animation that describes its contents (so the crews know its contents without running the segment)?
Is printing supported? If so, is it adequate?
Can crews select a portion of a document to be printed?
Is the hard copy usable?
Can the crew terminate a print job immediately, if necessary?
These criteria have been developed as the result of research into human factors in the use of electronic documents in EFBs by the Human Factors Division of the Office of Aviation Programs at the Volpe National Transportation Systems Center. Knowing these criteria in advance can help an aircraft operator in preparing for approval. However, I believe that operators can benefit from a more detailed set of specifications in regard to the interface to electronic documents. Section 6.3.1 of the S1000D standard provides rules and guidance for the look and feel, and printed output from an Interactive Electronic Technical Publication (IETP). Section 6.4.1 defines a functionality matrix for IETPs to be used as an aid for defining requirements for S1000D projects. The functionality matrix leverages the US Department of Defense (DoD) long experience in defining class 1 to 5 IETMs with military specifications MIL-PRF-87268 and MIL-PRF-87269. For example, in the area of searching, the S1000D functionality matrix provides very detailed guidelines that go beyond the simple criteria "Can crews search the document electronically?" and "Is the search technique adequate?". The matrix breaks down searching functionalities into:

Full-text search
User-defined Boolean search
Search across multiple databases and files
Context search
Keyword search

Publishing EFB electronic documents in XML provides many benefits over the Adobe® PDF format. Key enabling technologies for XML-based EFB electronic documents are: ISO Schematron, XSLT, XSL FO, XLink, XPointer, XInclude, and XQuery. For quality assurance, the electronic documents application should be subjected to rigorous unit testing and functional testing before its release to flight crews. A content management system can help an operator by providing features such as workflow routing, versioning, document locking, access control, and full audit trail of modifications made to documents.

The Air Transport Association (ATA) has adopted S1000D as the next generation aircraft digital data standard and there is already a very close collaboration between the ATA and the S1000D TPSMG to harmonize commercial aviation technical data requirements with S1000D. That collaboration should be extended to electronic documents for EFBs to allow aircraft operators to leverage and influence the development of the S1000D IETP functionality matrix for better guidance on creating the paperless cockpit.

The Business Value of XML

2007-09-02T16:59:00.000-04:00

What exactly is the business value of XML and its related technologies such as XML Schema, ISO Schematron, XForms, XSLT, and XQuery? Let's review three use cases where XML adds real value.

The first use case is software configuration and metadata. Java EE Frameworks such as Hibernate, Spring, Struts, and JSF use XML for configuration and metadata. Unfortunately, not all Java EE developers understand the value of configuring their application with XML as opposed to Java annotations. There is currently a backlash against XML with some developers complaining about what they call "XML hell". These developers prefer to keep configuration and metadata closer to the code itself using Java annotations. Newer Java EE frameworks such as the JPA (Java Persistence API) and Struts 2 provide annotation capabilities which are very popular with developers. I personally believe that this is a trend in the wrong direction. Annotations are certainly convenient for developers, but not necessarily to end users of your application. With XML configuration, end users of your software who are not programmers can achieve a certain level of customization on their own by simply editing an XML configuration file in a text editor without importing your SDK in an IDE and compiling Java code or hiring a Java EE developer to do it for them. As a software buyer, when deciding between two competing products, I would choose (everything else being equal) the one that allows me to do some customizations with simple XML files. Developers can reduce the verbosity of their XML configuration files by using techniques such as inheritance, overridable default values, and preferring the use of attributes over child elements for example. A compromise could be to make the annotations overridable with XML configuration.

The next use case is data exchange across organizations. Two example applications that are currently delivering real value are UBL and NIEM. XML vocabularies such as UBL and NIEM define common semantics and data structure trough data dictionaries and XML schemas respectively. In addition, they can specify certain business rules that can be enforced with the use of an assertion-based schema language such as ISO Schematron.

Developed by OASIS, UBL (Universal Business Language) is an XML vocabulary for the exchange of business documents such as invoices, purchase orders, and receipts. In Denmark, the government has mandated the use of UBL invoices for all public-sector billing. The result is over 100 millions euros in savings every year. The Swedish government estimates that it can save 440 millions euros with the adoption of UBL for electronic commerce. Please note that these initiatives involve not only big government and Fortune 500 companies, but hundreds of thousands of SMEs (small and medium size enterprises) as well.

NIEM (National Information Exchange Model) developed by the U.S. Department of Justice and the Department of Homeland Security is an XML vocabulary for the exchange of information between government agencies. For example, it allows law enforcement agencies to quickly exchange information. Law enforcement agencies use heterogeneous applications called RMS (Record Management Systems) and XML data is the bridge between them because it is vendor neutral, cross-platform, and supports structured data of arbitrary complexity. XSLT 2.0 as a generic XML transformation language can play an important role here as well. As an example, an RMS system can export raw XML data which can then be mapped to a NIEM compliant XML Schema by performing an XSLT transformation. If a legacy RMS system can only export CSV (comma-separated values)text files, XSLT 2.0 can up-convert the CSV into a NIEM compliant XML document. It is possible to process XML with a traditional programming language such as C# or Java. However, the problem is the "impedance mismatch" between the type system of these programming languages and a type system based on XML (such as the XML schema type system). Some developers will find XQuery easier to use than XSLT (probably because of its SQL-like syntax) for processing XML data. In addition, XSLT 2.0 and XQuery are declarative processing languages (they describe the "what" as opposed to the "how") and are therefore accessible to many non-programmers.

The last use case is knowledge management in general and content management in particular. In the new global knowledge economy, the most important asset of an organization is its intellectual capital which is acquired and developed by its knowledge workers. That intellectual capital is often captured in documents such as blogs, wikis, emails, PowerPoint presentations, podcasts, engineering drawings, architecture diagrams, ISO 9000 quality manuals, installation and troubleshooting procedures, Microsoft Word documents containing requirements and design specifications, various corporate forms, etc. These mission-critical documents are often dumped into shared network drives. They are not managed with the same rigor and cannot be queried as the data contained in your CRM and ERP systems. The main reason is that these documents represent unstructured data as opposed to the well-structured relational data stored by the RDBMS on which your ERP and CRM systems sit. In some industries, the need to bring content under control is driven by regulatory compliance. In any case, organizations shouldn't wait until their most valuable employees leave before they start thinking about managing their knowledge assets. That's where an enterprise content management system (CMS) and an enterprise portal come into play.

XML goes beyond tags, taxonomy, and content categorization to provide fine-grained content discovery, query, and processing capabilities. With XML, the document becomes the database. First, using XML schema, you can constrain and validate the structure and data types of the content of your business documents just like you do with a relational database schema. Using XForms, you can provide a user friendly interface for your end users to contribute XML content by presenting them with a regular HTML form. Once the content is captured as XML, it can be stored in a native XML database. With XQuery, the native XML database allows you to perform structured queries on the content (as opposed to just full-text or metadata search). XQuery also allows you to assemble content dynamically (for example, build two distinct training manuals for two different configurations of your product from a single source). You can use XQuery to query both relational (from your ERP and CRM systems) and XML data sources and aggregate the results. With XSLT, you enable content adaptation for cross-media publishing (print, web, and wireless) from a single source.

If you decide to manage your product technical documentation with XML, there are standards that can help. The DITA (Darwin Information Typing Architecture) specification is very popular with computer software and hardware documentation. The S1000D standard is designed to support mission critical maintenance and operation documentation in industries such as aerospace, defense, automotive, oil and gas, heavy equipments and machinery, and power generation.

Organizations that have a strategy for managing their knowledge assets using XML and related technologies will have a definitive competitive advantage in today’s economy.

Canada welcomes the C-17

2007-08-10T20:13:00.001-04:00

The Canadian Air Force will receive this weekend the first of four strategic lift aircraft-the C-17 Globemaster III.

This is a special occasion for the Canadian Air Force for the following reasons:

The men and women of our armed forces who are deployed in dangerous peace keeping operations worldwide deserve the best equipment and logistics support we can afford, and they need them now.

Canada needs its own airlift capability to provide quick humanitarian relief and piece keeping operations anywhere in the world when needed.

The C-17 was competing against the EADS A400M. The C-17 was a good choice for Canada because:

The C-17 is a proven airlifter and Canada can leverage that experience immediately.

EADS revealed last week that the A400M's first flight has been delayed until "the summer of 2008", and that "the consequence on deliveries and cost is under assessment".

Finally, I must confess that I've always been in love with the C-17 for its design and record breaking capabilities.

My russian aviation adventures

2007-08-04T20:52:00.002-04:00

Fifteen years ago, in the summer of 1992, I had the opportunity to fly the Tupolev 154 as a flight engineer. I was flying with an Aeroflot crew based in St-Petersburg, Russia. This was part of a flight training program that also included theoretical studies at the Russian Academy of Civil Aviation in Aviagorodok (near St-Petersburg) and practical exercises on a flight simulator.

The Tu-154 did not have a glass cockpit like modern aircrafts, so the flight crew had to be skilled in aerodynamics as well as aicraft systems and engines. For someone who wanted to learn about airplanes, this was the right airplane to fly. The Tu-154 was powered by three Kuznetsov NK-82 turbofans and was a very reliable airplane at the time. The Tu-154 seats about 160 passengers and flight crew of three or four. I was seated on the flight engineer deck. My role was to control and monitor the aircraft's engines and systems (hydraulic, electrical, pneumatic, air conditioning, and APU) during normal, abnormal, and emergency situations. Each flight started with a visual inspection of the exterior of the airplane followed by some systems and engines checks inside the cabin.

The Tu-154 flight manual was a good guide and I was fortunate to have a very professional and experienced instructor during the simulator training. The flights took me to the following cities:

Anapa
Mourmansk
Dnepr
Kiev
Moscow
Simferopol
Volgograd
Krasnadar
Sotchi

All these flights were uneventful and gave me the chance to discover the russian countryside. It was a pleasure to fly with the rest of the crew. They were interested in learning few english and french words from me. My first 6 months in Russia were spent learning to speak, read, and write in the russian language. The total immersion helped learn the language very quickly and I was able to take all the aviation courses and read the aircraft technical documentation in russian.

I wanted to revive my memories of flying the Tu-154 with a PC-based flight simulator. While surfing the web, I found a web site for Project Tupolev Tu-154B2 which is a flight simulator for the Tu-154. In the simulator, the flight engineer panel is very detailed and realistic. It is divided in sub-panels for engine parameters, hydraulic system, electrical system, etc.

Instructions are provided on how to get the engines and systems started, but clearly, my previous knowledge about the aircraft was very helpful. The simulator comes with an english user guide, but the flight deck is completely in russian.

The simulator can be downloaded from the Project Tupolev Support Site.