Sunday, April 28, 2013

How I Make Technology Decisions

The open source community has responded to the increasing complexity of software systems by creating many frameworks which are supposed to facilitate the work of developing software. Software developers spend a considerable amount of time researching, learning, and integrating these frameworks to build new software products. Selecting the wrong technology can cost an organization millions of dollars. In this post, I describe my approach to selecting these frameworks. I also discuss the frameworks that have made it to my software development toolbox.

Understanding the Business


The first step is to build a strong understanding of the following:

  • The business goals and challenges of the organization. For example, the healthcare industry is currently shifting to a value-based payment model in an increasingly tightening regulatory environment. Healthcare organizations are looking for a computing infrastructure that support new demands such as the Accountable Care Organization (ACO) model, patient-centered outcomes, patient engagement, care coordination, quality measures, bundled payments, and Patient-Centered Medical Homes (PCMH).

  • The intended buyers and users of the system and their concerns. For example, what are their pain points? which devices are they using? and what are their security and privacy concerns?

  • The standards and regulations of the industry.

  • The competitive landscape in the industry. To build a system that is relevant, it is important to have some ideas about the following: what is the competition? what are the current capabilities of their systems? what is on their road map? and what are customers saying about their products. This knowledge can help shape a Blue Ocean Strategy.

  • Emerging trends in technologies.

This type of knowledge comes with industry experience and a habit of continuously paying attention to these issues. For example, on a daily basis, I read industry news as well as scientific and technical publications. As a member of the American Medical Informatics Association (AMIA), I receive the latest issue of the Journal of the American Medical Informatics Association (JAMIA) which allows me to access cutting-edge research in medical informatics. I speak at industry conferences when possible and this allows me not only to hone my presentation skills, but also attend all sessions for free or at a discounted price. For the latest in software development, I turn to publications like InfoQ, DZone, and TechCrunch.

To better understand the users and their needs and concerns, I perform early usability testing (using sketches, wireframes, or mockups) to test design ideas and obtain feedback before actual development starts. For generating innovative design ideas, I recommend the following book: Universal Methods of Design: 100 Ways to Research Complex Problems, Develop Innovative Ideas, and Design Effective Solutions by Bruce Hanington and Bella Martin.

 

Architecting the Solution


Armed with a solid understanding of the business and technological landscape as well as the domain, I can start creating a solution architecture. Software development projects can be chaotic. Based on my experience working on many software development projects across industries, I found that Domain Driven Design (DDD) can help foster a disciplined approach to software development. For more on my experience with DDD, see my previous post entitled How Not to Build A Big Ball of Mud, Part 2.

Frameworks evolve over time. So, I make sure that the architecture is framework-agnostic and focused on supporting the domain. This allows me to retrofit the system in the future with new frameworks as they emerge.


 

Due Diligence


Software development is a rapidly evolving field. I keep my eyes on the radar and try not to drink the vendors Kool-Aid. For example, not all vendors have a good track record in supporting standards, interoperability, and cross-platform solutions.

The ThoughtWorks Technology Radar is an excellent source of information and analysis on emerging trends in software. Its contributors include software thought leaders like Martin Fowler and Rebecca Parson. I also look at surveys of the developers community to determine the popularity, community size, and usage statistics of competing frameworks and tools. Sites like InfoQ often conduct these types of surveys like the recent InfoQ survey on Top JavaScript MVC Frameworks. I also like Matt Raible's Comparing JVM Web Frameworks.

I value the opinion of recognized experts in the field of interest. I read their books, blogs, and watch their presentations. Before formulating my own position, I make sure that I read expert opinions on opposing sides of the argument. For example, in deciding on a pure Java EE vs. Spring Framework approach, I read arguments by experts on both sides (experts like Arun Gupta, Java EE Evangelist at Oracle and Adrian Colyer, CTO at SpringSource).

Finally, consider a peer review of the architecture using a methodology like the Architecture Tradeoff Analysis Method (ATAM). Simply going through the exercise of explaining the architecture to stakeholders and receiving feedback can significantly help in improving it.


Rapid Prototyping 

 

It's generally a good idea to create a rapid prototype to quickly learn and demonstrate the capabilities and value of the framework to the business. This can also generate excitement in the development team, particularly if the framework can enhance the productivity of developers and make their life easier.

 

The Frameworks I've Selected


The Spring Framework

I am a big fan of the Spring Framework. I believe it is really designed to support the need of developers from a productivity standpoint. In addition to dependency injection (DI), Aspect Oriented Programming (AOP), and Spring MVC, I like the Spring Data repository abstraction for JPA, MongoDB, Neo4J, and Hadoop. Spring supports Polyglot Persistence and Big Data today. I use Spring Roo for rapid application development and this allows me to focus on modeling the domain. I use the Roo scaffolding feature to generate a lot of Spring configuration and Java code for the domain, repository (Roo supports JPA and MongDB), service, and web layers (Roo supports Spring MVC, JSF, and GWT). Spring also support for unit and integration testing with the recent release of Spring MVC Test.

I use Spring Security which allows me to use AOP and annotations to secure methods and supports advanced features like Remenber Me and regular expressions for URLs. I think that JAAS is too low-level. Spring Security allows me to meet all OWASP Top Ten requirements (see my previous post entitled  Application-Level Security in Health IT Systems: A Roadmap).

Spring Social makes it easy to connect a Spring application to social network sites like Facebook, Twitter, and LinkedIn using the OAuth2 protocol. From a tooling standpoint, Spring STS supports many Spring features and I can deploy directly to Cloud Foundry from Spring STS. I look forward to evaluating Grails and the Play Framework which use convention over configuration and are built on Groovy and Scala respectively.

Thymeleaf, Twitter Boostrap, and JQuery

I use Twitter Boostrap because it is based on HTML5, CSS3, JQuery, LESS, and also supports a Responsive Web Design (RWD) approach. The size of the components library and the community is quite impressive.

Thymeleaf is an HTML5 templating engine and a replacement for traditional JSP. It is well integrated with Spring MVC and supports a clear division of labor between back-end and front-end developers. Twitter Boostrap and Thymeleaf work well together.


AngularJS

For Single Page Applications (SPA) my definitive choice is AngularJS. It provides everything I need including a clean MVC pattern implementation, directives, view routing, Deep Linking (for bookmarking), dependency injection, two-way databinding, and BDD-style unit testing with Jasmine. AngularJS has its own dedicated debugging tool called Batarang. There are also several learning resources (including books) on AngularJS.

Check this page comparing the performance of AngulaJS vs. KnockoutJS. This is a survey of the popularity of  Top JavaScript MVC Frameworks.

 

D3.js 

D3.js is my favorite for data visualization in data-intensive applications. It is based on HTML5, SVG, and Javascript. For simple charting and plotting, I use jqPlot which is based on JQuery. See my previous post entitled Visual Analytics for Clinical Decision Making.

 

I use R for statistical computing, data analysis, and predictive analytics. See my previous post entitled Statistical Computing and Data Mining with R.


Development Tools


My development tools include: Git (Distributed Version Control), Maven or Gradle (build), Jenkins (Continuous Integration), Artifactory (Repository Manager), and Sonar (source code quality management). My testing toolkit includes Mockito, DBUnit, Cucumber JVM, JMeter, and Selenium.

Sunday, April 14, 2013

Addressing Challenges to the Adoption of Clinical Decision Support (CDS) Systems: A Practical Approach

Laptop and stethoscope by jfcherry is licensed under CC BY-SA 2.0
This post has been updated on February 15, 2015.

Despite its potential to improve the quality of care, CDS is not widely used in health care delivery today. In technology marketing parlance, CDS has not crossed the chasm. There are several issues that need to be addressed including:

  • Clinicians' acceptance of the concept of automated execution of evidence-based clinical practice guidelines.

  • Integration into clinical workflows and care protocols.

  • Usability and human factors issues including alert fatigue and the human factors that influence alert acceptance.

  • With the expanding use of clinical prediction models for diagnosis and prognosis, there is a need for a better understanding of the probabilistic approach to clinical decision making under uncertainty.

  • Standardization to enable the interoperability and reuse of CDS knowledge artifacts and executable clinical guidelines.

  • The challenges associated with the automatic concurrent execution of multiple clinical practice guidelines for patients with co-morbidities.

  • Integration with modern information retrieval capabilities which allow clinicians to access up-to-date biomedical literature. These capabilities includes text mining, Natural Language Processing (NLP), and more advanced Clinical Question Answering (CQA) tools.  CQA allows clinicians to ask clinical questions in natural language and extracts answers from very large amounts of unstructured sources of medical knowledge. PubMed has more than 23 millions citations for biomedical literature from MEDLINE, life science journals, and online books. The typical Clinical Practice Guideline (CPG) is 50 to 150 pages long. It is impossible for the human brain to keep up with that amount of knowledge.

  • The use of mathematical simulations in CDS to explore and compare the outcomes of various treatment alternatives.

  • Integration of genomics to enable personalized medicine as the cost of whole-genome sequencing (WGS) continues to fall.

  • Integration of outcomes research in the context of a shift to a value-based healthcare delivery model. This can be achieved by incorporating the results of Comparative Effectiveness Research (CER) and Patient-Centered Outcome Research (PCOR) into CDS systems. Increasingly, outcomes research will be performed using observational studies (based on real world clinical data) which are recognized as complementary to randomized control trials (RCTs) for discovering what works and what doesn't work in practice. This is a form of Practice-Based Evidence (PBE) that is necessary to close the evidence loop.

  • Support for a shared decision making process which takes into account the values, goals, and wishes of the patient.

  • The use of Visual Analytics in CDS to facilitate analytical reasoning over very large amounts of structured and unstructured data sources.

  • Finally, the challenges associated with developing hybrid decision support systems which seamlessly integrate all the technologies mentioned above including: machine learning predictive algorithms, real-time data stream mining, visual analytics, ontology reasoning, and text mining.

In response to a paper titled Grand challenges in clinical decision support by Sittig et al. [1], Fox et al. [2] outlined four theoretical foundations for the design and implementation of CDS systems: decision theory, theories of knowledge representation, process design, and organizational modeling. The practical approach discussed in this post is grounded in those four theoretical foundations.


CDS Interoperability


The complexity and cost inherent in capturing the medical knowledge in clinical guidelines and translating that knowledge into executable code remain a major impediment to the widespread adoption of CDS software. Therefore, there is a need for standards for the interchange and reuse of CDS knowledge artifacts and executable clinical guidelines.

Different formalisms, methodologies, and architectures have been proposed over the years for representing the medical knowledge in clinical guidelines. Examples include but are not limited to the following:

  • The Arden Syntax
  • GLIF (Guideline Interchange Format)
  • GELLO (Guideline Expression Language Object-Oriented)
  • GEM (Guidelines Element Model)
  • The Web Ontology Language (OWL)
  • PROforma
  • EON
  • PRODIGY
  • Asbru
  • GUIDE
  • SAGE.
More recently, HL7 has published the Clinical Decision Support (CDS) Knowledge Artifact Specification which provides guidance on shareable CDS knowledge artifacts including event-condition-action rules, order sets, and documentation templates.

The HL7 Context-Aware Knowledge Retrieval (Infobutton) specifications provide a standard mechanism for clinical information systems to request context-specific clinical knowledge to satisfy clinicians and patients' information needs at the point of care.

Enabling the interoperability of executable clinical guidelines requires a standardized domain model for representing the medical information of patients and other contextual clinical information. The HL7 Virtual Medical Record (vMR) is a standardized domain model for representing the inputs and outputs of CDS systems. The ability to transform an HL7 CCDA document into an HL7 vMR document means that EHR systems that are Meaningful Use Stage 2 certified can consume these standard-compliant decision support services.

Because of the complexity and cost of developing CDS software, CDS software capabilities can be exposed as a set of services (part of a service-oriented architecture [16]) which can be consumed by other clinical systems such as EHR and Computerized Physician Order Entry (CPOE) systems. When deployed in the cloud, these CDS software services can be shared by several health care providers to reduce costs. The HL7 Decision Support Service (DSS) specification defines a REST and a SOAP web service interfaces using the vMR as message payload for accessing interoperable decision support services.

In practice, executable CDS rules (like other complex types of business rules) can be implemented with a production rule system using forward chaining. This is the approach taken by OpenCDS and some other large scale CDS implementations in real-world healthcare delivery settings. This allows CDS software developers to externalize the medical knowledge contained in clinical practice guidelines in the form of declarative rules as opposed to embedding that knowledge in procedural code. Many viable open source business rule management systems (BRMS) are available today and provide capabilities such as a rule authoring user interface, a rules repository, and a testing environment. Furthermore, a rule execution environment can be integrated with business processes, ontologies, and predictive analytics models (more on that later).

The W3C Rule Interchange Format (RIF) specification is a possible solution to the interchange of executable CDS rules. The RIF Production Rule Dialect (RIF PRD) is designed as a common XML serialization syntax for multiple rule languages to enable rule interchange between different BRMS. For example, RIF-PRD would allow the exchange of executable rules between existing BRMS like JBoss Drools, IBM ILOG JRules, and Jess. RIF is currently a W3C Recommendation and is backed by several BRMS vendors. Consentino et al. [3] described a model-driven approach for interoperability between IBM's proprietary ILOG rule language and RIF.


Seamless Integration into Clinical Workflows and Care Pathways


One of the main complaints against CDS systems is that they are not well integrated into clinical workflows and care protocols. Existing business process management standards like the Business Process Modeling Notation (BPMN) can provide a proven, practical, and adaptable approach to the integration of CDS rules and clinical pathways and protocols. Some existing open source and commercial BRMS already provide an integration of business rules and business processes out-of-the box and there are well-known patterns for integrating rules and processes [4, 5, 6] in business applications.

In 2014, the Object Management Group (OMG) released the Decision Model and Notation (DMN) specification which defines various constructs for modeling decision logic. The combination of BPMN and DMN [7, 8] provides a practical approach for modeling the decisions in clinical practice guidelines while integrating these decisions with clinical workflows. BPMN and DMN also support the modeling of decisions and processes that span functional and organizational boundaries.


Human Factors in the Use of Clinical Decision Support Systems


We need to do a better job in understanding the human factors that influence alert acceptance by clinicians in CDS. We also need clear and proven usability guidelines (backed by scientific research) that can be implemented by CDS software developers. Several research projects have sought to understand why clinicians accept or ignore alerts in medication-related CDS [9, 10]. Zacharia et al. [11] developed and validated an Instrument for Evaluating Human-Factors Principles in Medication-Related Decision Support Alerts (I-MeDeSA). I-MeDeSA measures CDS alerts on the following nine human factors principles: alarm philosophy, placement, visibility, prioritization, color learnability and confusability, text-based information, proximity of task components being displayed, and corrective actions.

The British National Health Service (NHS) Common User Interface (CUI) Program has created standards and guidance in support of the usability of clinical applications with inputs from user interface design specialists, usability experts, and hundreds of clinicians with a diversity of background in using health information technology. The program is based on a rigorous development process which includes: research, design, prototyping, review, usability testing, and patient safety assessment by clinicians. In the US, the National Institute of Standards and Technology (NIST) has also published some guidance on the usability of clinical applications.

Studies have also shown that like in aviation, checklists can provide cognitive support to clinicians in the decision making process.


Integrating Genomic Data with CDS


The costs of whole-genome sequencing (WGS) and whole-exome sequencing (WES) continue to fall. Increasingly, both WGS and WES will be used in clinical practice for inherited disease risk assessment and pharmacogenomic findings [21]. There is a need for a modern CDS architecture that can support and facilitate the introduction and use of WGS and WES in clinical practice.

In a paper titled Technical desiderata for the integration of genomic data with clinical decision support [14], Welch et al. proposed technical requirements for the integration of genomic data with clinical decision support.  In another paper titled A proposed clinical decision support architecture capable of supporting whole genome sequence information [15], Welch et al. proposed the following clinical decision support architecture for supporting whole genome sequence information (click to enlarge):

Proposed service-oriented architecture (SOA) for whole genome sequence (WGS)-enabled CDS by Brandon M. Welch, Salvador Rodriguez Loya, Karen Eilbeck, and Kensaku Kawamoto is licensed under CC BY 3.0

The proposed architecture includes the following components: the genome variant knowledge base, the genome database, the CDS knowledge base, a CDS controller and the electronic health record (EHR). The authors suggest separating the genome data from the EHR data.


Practice-Based Evidence (PBE) needed for closing the evidence loop


Prospective randomized controlled trials (RCTs) are still considered the gold standard in evidence-based medicine. Although they can control for biases, RCTs are costly, time consuming, and must be performed under carefully controlled conditions.

The retrospective analysis of existing clinical data sources is increasingly recognized as complementary to RCTs for discovering what works and what doesn't work in real world clinical practice [23]. These retrospective studies will allow the creation of clinical prediction models which can provide personalized absolute risk and treatment outcome predictions for patients. They also facilitate what has been referred to as "rapid learning health care." [24]

Toward Data-Driven Clinical Decision Support (CDS)


Williams Osler (1849-1919) [19] famously said that "Medicine is a science of uncertainty and an art of probability."

The use of clinical prediction models for diagnosis and prognosis is becoming common practice in clinical care. These models can predict the health risks of patients based on their individual health data. Clinical Prediction Models provide absolute risk and treatment outcome prediction for conditions such as diabetes, kidney disease, cancer, cardiovascular disease, and depression.  These models are built with statistical learning techniques and introduce new challenges related to their probabilistic approach to clinical decision making under uncertainty [20]. In his book titled Super Crunchers: Why Thinking-By-Numbers Is The New Way To be Smart, Ian Ayres wrote:

Traditional experts make better decisions when they are provided with the results of statistical prediction. Those who cling to the authority of traditional experts tend to embrace the idea of combining the two forms of knowledge by giving the experts 'statistical support'. The purveyors of diagnostic software are careful to underscore that its purpose is only to provide support and suggestions. They want the ultimate decision and discretion to lie with the doctor. [12]

Furthermore, in order to leverage existing clinical domain knowledge from clinical practice guidelines and biomedical ontologies [22], machine learning algorithms' probabilistic approach to decision making under uncertainty must be complemented by technologies like production rule systems and ontology reasoners. Sesen et al. [18] designed a hybrid CDS for lung cancer care based on probabilistic reasoning with a Bayesian Network model and guideline-based recommendations using a domain ontology and an ontology reasoner.

Fox et al. [2] proposed an argumentation approach based on the construction, summarization, and prioritization of arguments for and against each generated candidate decision. These arguments can be both qualitative or quantitative in nature. On the importance of presenting evidence-based rationale in CDS systems, Fox et al. wrote:

In short, to improve usability of clinical user interfaces we advocate basing design around a firm theoretical understanding of the clinician’s perspective on the medical logic in a decision, the qualitative as well as quantitative aspects of the decision, and providing an evidence-based rationale for all recommendations offered by a CDS. [2]
In a paper titled A canonical theory of dynamic decision-making [13], Fox et al. proposed a canonical theory of dynamic decision-making and presented the PROforma clinical guideline modeling language as an instance of the canonical theory.

Clinical predictive model presentation techniques include traditional score charts, nomograms, and clinical rules [17]. However Clinical Prediction Models are easier to use and maintain when deployed as scoring services (part of a service-oriented software architecture) and integrated into CDS systems. The scoring service can be deployed in the cloud to allow integration with multiple client clinical systems [20]. The Predictive Model Markup Language (PMML) specification published by the Data Mining Group (DMG) supports the interoperable deployment of predictive models in heterogeneous software environments.

Visual Analytics or data visualization techniques can also play an important role in the effective presentation of Clinical Prediction Models to nonstatisticians particularly in the context of shared decision making.


Concurrent execution of multiple guidelines for patients with co-morbidities


According to the Medicare 2012 chart book, "over two-thirds of beneficiaries having two or more chronic conditions and 14% having 6 or more chronic conditions." [25]

In Grand Challenges in Clinical Decision Support [1], Sittig et al. wrote:
The challenge is to create mechanisms to identify and eliminate redundant, contraindicated, potentially discordant, or mutually exclusive guideline-based recommendations for patients presenting with co-morbid conditions.
Michalowski et al. [26] proposed a mitigation framework based on a first-order logic (FOL) approach.


A CDS Architecture for the era of Precision Medicine


I proposed a scalable CDS architecture for Precision Medicine in another post titled Toward a Reference Architecture for Intelligent Systems in Clinical Care.

 

References


[1] Sittig D, Wright A, Osheroff JA, Middletone B, Jteich J, Ash J, et al. Grand challenges in clinical Decision support. J Biomed Inform 2008;41(2):387–92.

[2] Fox, J., Glasspool, D.W., Patkar, V., Austin, M., Black, L., South, M., et al. (2010). Delivering clinical decision support services: there is nothing as practical as a good theory. J. Biomed. Inform. 43, 831–843

[3] Valerio Cosentino, Marcos Didonet del Fabro, Adil El Ghali. A model driven approach for bridging ILOG Rule Language and RIF. RuleML, Aug 2012, Montpellier, France.

[4] Mauricio Salatino. (Processes & Rules) OR (Rules & Processes) 1/X. http://salaboy.com/2012/07/19/processes-rules-or-rules-processes-1x/. Retrieved February 15, 2015.

[5] Mauricio Salatino. (Processes & Rules) OR (Rules & Processes) 2/X. http://salaboy.com/2012/07/28/processes-rules-or-rules-processes-2x/. Retrieved February 15, 2015.

[6] Mauricio Salatino. (Processes & Rules) OR (Rules & Processes) 3/X. http://salaboy.com/2012/07/29/processes-rules-or-rules-processes-3x/. Retrieved February 15, 2015.

[7] Sylvie Dan. Modeling Clinical Rules with the Decision Modeling and Notation (DMN) Specification. http://sylviedanba.blogspot.com/2014/05/modeling-clinical-rules-with-decision.html. Retrieved February 15, 2015.

[8] Dennis Andrzejewski, Eberhard Beck, Eberhard Beck, Laura Tetzlaff. The transparent representation of medical decision structures based on the example of breast cancer treatment. 9th International Conference on Health Informatics.

[9] Phansalkar S, Zachariah M, Seidling HM, Mendes C, Volk L, Bates DW. Evaluation of medication alerts in electronic health records for compliance with human factors principles. J Am Med Inform Assoc. 2014 Oct;21(e2):e332-40. doi: 10.1136/amiajnl-2013-002279.

[10] Seidling HM, Phansalkar S, Seger DL, et al. Factors influencing alert acceptance: a novel approach for predicting the success of clinical decision support. J Am Med Inform Assoc 2011;18:479–84.

[11] Zachariah M, Phansalkar S, Seidling HM, et al. Development and preliminary evidence for the validity of an instrument assessing implementation of human-factors principles in medication-related decision-support systems--I-MeDeSA. J Am Med Inform Assoc 2011;18(Suppl 1):i62–72.

[12] Ayres I. Super Crunchers: Why Thinking-By-Numbers Is The New Way To be Smart. Bantam.

[13] Fox J., Cooper R. P., Glasspool D. W. (2013). A canonical theory of dynamic decision-making. Front. Psychol. 4:150 10.3389/fpsyg.2013.00150.

[14] Welch, B.M.; Eilbeck, K.; Del Fiol, G.; Meyer, L.; Kawamoto, K. Technical desiderata for the integration of genomic data with clinical decision support. 2014,

[15] Welch BM, Loya SR, Eilbeck K, Kawamoto K. A proposed clinical decision support architecture capable of supporting whole genome sequence information. J Pers Med. 2014 Apr 4;4(2):176-99. doi: 10.3390/jpm4020176.

[16] Loya SR, Kawamoto K, Chatwin C, Huser V. Service oriented architecture for clinical decision support: a systematic review and future directions. J Med Syst. 2014 Dec;38(12):140. doi: 10.1007/s10916-014-0140-z.

[17] Ewout W. Steyerberg. Clinical Prediction Models. A Practical Approach to Development, Validation, and Updating. New York: Springer, 2010.

[18] Sesen MB, Peake MD, Banares-Alcantara R, Tse D, Kadir T, Stanley R, Gleeson F, Brady M. 2014 Lung Cancer Assistant: a hybrid clinical decision support application for lung cancer care. J. R. Soc. Interface 11: 20140534. http://dx.doi.org/10.1098/rsif.2014.0534

[19] Bean RB, Bean, BW. Sir William Osler: aphorisms from his bedside teachings and writings. New York; 1950.

[20] Joel Amoussou. How good is your crystal ball?: Utility, Methodology, and Validity of Clinical Prediction Models. http://efasoft.blogspot.com/2015/01/how-good-is-your-crystal-ball-utility.html. Retrieved February 15, 2015.

[21] Dewey FE, Grove ME, Pan C, et al. Clinical Interpretation and Implications of Whole-Genome Sequencing. JAMA. 2014;311(10):1035-1045. doi:10.1001/jama.2014.1717.

[22] Joel Amoussou. Ontologies for Addiction and Mental Disease: Enabling Translational Research and Clinical Decision Support. http://efasoft.blogspot.com/2014/08/ontologies-for-addiction-and-mental.html. Retrieved February 2015.

[23] Future radiotherapy practice will be based on evidence from retrospective interrogation of linked clinical data sources rather than prospective randomized controlled clinical trials. Dekker, Andre L. A. J. and Gulliford, Sarah L. and Ebert, Martin A. and Orton, Colin G., Medical Physics, 41, 030601 (2014), DOI:http://dx.doi.org/10.1118/1.4832139

[24] Lambin, Philippe et al. 'Rapid Learning health care in oncology' – An approach towards decision support systems enabling customised radiotherapy. Radiotherapy and Oncology , Volume 109 , Issue 1 , 159 - 164.

[25] Centers for Medicare & Medicaid Services. Chronic Conditions Among Medicare Beneficiaries, Chartbook: 2012 Edition. http://www.cms.gov/Research-Statistics-Data-and-Systems/Statistics-Trends-and-Reports/Chronic-Conditions/Downloads/2012Chartbook.pdf. Accessed Feb. 15, 2015.

[26] Szymon Wilk, Martin Michalowski, Xing Tan, Wojtek Michalowski: Using First-Order Logic to Represent Clinical Practice Guidelines and to Mitigate Adverse Interactions. KR4HC@VSL 2014: 45-61.