The S Curve

I will be speaking at SemTech 2010 - June 21-25 in San Francisco

Orthopaedia - A Semantic Platform for Collaborative Knowledgebase Development in Orthopaedics

Semantic technologies have been aggressively applied in biomedical and healthcare domains to address a number of problems related to the representation, interpretation, integration and access of knowledge. These technologies include the development of machine readable thesauri and vocabularies, taxonomies, formal ontologies as well as NLP tools and techniques. And of course, there are emerging applications that are being developed to deal with these underlying metamodels.  

This case study discusses a semantic platform that incorporates these technologies and metamodels towards the collaborative development of a knowledgebase in Orthopaedics. The platform enables users to easily enhance local content with publications from various sources, blogs, discussions, annotations, and automatically tag them with domain-specific terms, as well as make this information accessible across the enterprise to other tools and applications via a semantic repository.

We will also describe the current challenges in achieving coherency and integration across medical subdomains, and describe some the design decisions taken to overcome some of these challenges.

We will address the following points:

• Overview of semantic technologies in healthcare
• Challenges in semantic information integration across resources and applications
• Objectives of the project
• Discussion of technologies implemented
• Review of strengths and weaknesses of the current state of technologies in addressing knowledge management issues in healthcare
• Demo

 

Achieving Requirements Traceability with Confluence & JIRA, the rational (!Rational) way

Requirements Traceability refers to the ability to link requirements to projects, change requests, development tasks, test cases and results, checkins, builds, releases and stakeholders.   In other words - linking requirements with all relevant concepts associated with a software engineering life cycle. Given that Requirements drive the activities across the software engineering projects, maintaining such interrelationships can provide answers to some very critical and business-relevant questions relative to each requirement, whether they are related to business justification, auditing/compliance, delivery management or quality assessment.  

These can be grouped by stages and constituents (some examples below). 

Business Justification:

Who are the Stakeholders of each requirement?

Which requirements are being delivered in a release and for which stakeholders?

What is the business justification and cost of delivering a feature or requirement?

What is the ROI (once the costs are determined)?

Project Management:

Which release will the requirement be delivered in?

What are the risks and dependencies related to the implementation of a requirement?

What is the current state of a requirement with respect to development and QA?

Who signed off on it, who accepted it? 

Which checkins correspond to code complete for a requirement or feature?

Which build includes a particular requirement implementation?

Development & QA 

What are the design goals and motivations of a requirement?

What are the associated test cases and quality goals?

What are the associated use cases or user stories?

In typical engineering environments, some of these questions are difficult or impossible to determine, while others require a significant amount of tedious manual effort.  The reasons can be mostly attributed to the heterogeneity of ALM tools and the lack of integration between them.  Whatever the cause of this fragmentation of information, requirements traceability isn't always optional for organizations. Many expend significant human effort and co$t towards collecting sufficient traceability data to maintain their audit and compliance mandates.  

Traditional ALM vendors provide their own integrated solution set to help their customers achieve requirements traceability.  While they are able to address most, if not all, questions above, you have to invest into their entire portfolio of products and at significant co$ts.  I have rarely seen an engineering environment that has successfully adopted a single vendor solution across teams and departments.  For most part, such solutions end up as shelf-ware, reflecting wasteful and irRational decisions.

It is with this in mind that zAgile delivers its tool-agnostic and open source zCALM platform that is capable of integrating concepts across software engineering life cycle with the tools that you are using in YOUR environment. 

And to take that even further, you can achieve significant amount of requirements management and traceability by simply integrating your Confluence wiki and JIRA issue tracking application via zTeamwork.

  zTeamwork allows you to: 

1. Capture features, requirements, use cases, test cases, test suites, etc. in Confluence in a semi-structured way using zAgile's smart templates.  
2. Link them to each other using pre-defined relationships.
3. Create JIRA tasks, link them to various Confluence artifacts created above, so that you can now associated requirements with use cases, test cases, tasks, and bugs
4. View any and all of these links and relationships (including attributes such as status, completion, etc.) in Confluence or JIRA using simple queries.  There are JIRA properties tabs that may be configured to provide 'dashboard' type contextual views of this information.
5. Search for these specific concepts from within Confluence or JIRA, regardless of where they are created.  The search not only returns the matching result set but also all of the contextual relationships.

Teamwork's integration can be further extended with connectors for salesforce (link customers and cases to requirements and issues within your environment), Subversion (associate checkins with requirements) and CruiseControl (link requirements to builds).  

 With these connectors in place, you can answers ALL of the questions above with respect to requirements traceability -- without having to buy into expensive, complex and painful single-vendor solutions.  

And since zAgile's solutions are tool agnostic, you can also plug other tools on its platform to achieve the same integration capability - via custom connectors.

Somehow that seems more rational than Rational :)

-Sanjiva

Test drive these features in the sandbox

Machine-based semantic annotation of content in Confluence Enterprise Wiki

Annotation of content in any CMS, including traditional wikis, to provide contextual reference to any word or phrase is a common practice.  I am doing that in my blogs as I write.  When I reference specific topics or items, I go through the steps of creating links to them. 

Wiki annotation, therefore, also typically involves highlighting words or phrases and associating each to some reference - an external link or another page within the wiki.   It is a tedious process that involves marking up each word or phrase using specific wiki tags.  It not only requires considerable effort but also a meticulous nature and some dedication towards creating meaningful content.

What if it were automatic?  All you had to do is to add a macro at the beginning of your paragraph or page and everything meaningful within that text would automatically be annotated.

What if it were dynamic, so that the wiki content is still in its original form but all annotations are done when the page is rendered.  The annotation would automatically reflect any changes to the reference.  It would be more reliable and not prone to going stale.

What if the annotation was more contextual to your specific domain, to your business processes and functions, rather than highlighting of people, countries, organizations -- which may have less relevance when the text if referring to some technical requirements for a software component.

And finally, what if the annotation not only provided a link to some reference, but also some information about its type (is it a reference to a Project, a Requirement, a Task, a Bug, a Customer, a Software Component, ...)

Well, that's the capability that we will soon release for Atlassian's Confluence Enterprise Wiki in zAgile Teamwork and Wikidsmart products.

A simple, yet powerful capability in our upcoming release of Wikidsmart (Release 2.0) is the machine-based semantic annotation of wiki content.  It is easier to play with this feature than to describe it to really understand the possibilities that it opens up in making content more meaningful and accessible in enterprise wikis.  But here is an explanation:

By machine-based, I mean all you need to do to enable this capability is to surround the wiki paragraph with a macro (zannotate).

By semantic annotation, I mean that words or phrases in that paragraph that match any reference in our ontology-based semantic repository will be highlighted and linkable to additional information about the references.  Since zAgile Teamwork ships with ontologies for software engineering, you will see annotations of words and phrases that reference instances of Requirements, Projects (JIRA Projects), JIRA Issues, Tasks, Bugs, Software Components, Test Cases, etc.

You can test drive this capability in the zTeamwork sandbox.

Here is what it would look like:

WRITE A PARAGRAPH IN A WIKI AND BRACKET IT WITH ANNOTATION MACRO

SAVE THE PAGE AND THE PARAGRAPH WILL BE AUTOMATICALLY ANNOTATED.  CLICKING ON ANY TAGGED PHRASE WILL POPUP ITS REFERENCE CONTEXT

FOR EXAMPLE, REFERENCES TO JIRA ISSUES NOW ARE AUTOMATICALLY ANNOTATED WITH TYPE, DESCRIPTIONS AND LINKS TO THE INDIVIDUAL ISSUES

AND FINALLY, THE SAME APPLIED TO MEDICAL TEXT - APPLIED AGAINST AN  ONTOLOGY DERIVED FROM MeSH (Medical Subject Headings) THESAURUS, ALSO IN CONFLUENCE - SHOWING HIGHLIGHTED TERM AND ITS SYNONYMS

- Sanjiva

To test drive this feature, visit our sandbox

Creating JIRA Tasks from within Confluence Requirements Pages

One of the recent features delivered in zAgile Teamwork is the ability to create a JIRA Task from within a Confluence page and have the Task automatically inherit the context of the Requirement that is driving it.  The use case description below explains this 'context inheritance'.

This is also a great examples of the interoperability that can be achieved between heterogeneous applications (JIRA, Confluence in example but almost any app) using semantic technologies.  In this case, it is our zCALM semantic application server, that is enabling this seemingly simple scenario.

The use case is as follows:

1. Using one of the example semantic templates in Teamwork, you define a list of Requirements in a Confluence page 

2. Each Requirement on that page may be linked to one or many Design Goals that is the motivation for it, as well as a specific JIRA Project in which it is planned for delivery. 

Note: It is also possible to modify the template to associate all of the requirements on the page to a single project and map the individual requirement to a specific version. 

3. Once a certain number of requirements have been assigned to corresponding JIRA projects, you can use the Create Issue link to launch a JIRA Issue Creation page.  Assuming you also have a JIRA account and issue creation privilege for the project, you will notice a number of things:

1. The JIRA Issue Creation page is automatically associated with the project to which the Requirement was linked.
2. The issue is of type Task
3. The Issue Creation page is pre-filled with Summary and Description fields from the Requirement itself
4. The Issue Reporter is initialized to the person logged in
5. And more significantly, a custom field in JIRA showing links to "Related Requirements" has the originating requirement flagged. 

At this point, all you have to do after clicking on the Create Issue link is to press the Create button on the issue creation page.  The saved Task in JIRA will be automatically linked to the Requirement for which it was created.

And finally, the Requirements Document from where all this began will also automatically show the created JIRA Tasks,  their corresponding Requirements, status, etc.

A simple use case but a very powerful paradigm for Enterprise Application Interoperability.  All of this integration is driven by an underlying semantic infrastructure that maintains the relationship across the concepts that various tools represent.

You can Test-Drive this functionality in the sandbox for Teamwork.

-Sanjiva

PS - To test drive this feature, visit our sandbox

The screenshots below show the flow from Confluence Requirements Document page to JIRA Issue Creation page and back, reflecting the context that carries across the two applications automatically.

CAPTURE REQUIREMENTS AND ASSOCIATE WITH JIRA PROJECT(S)

 

AUTO-GENERATE PRE-FILLED JIRA ISSUE CREATION PAGE (WITH LINK TO INITIATING REQUIREMENT)

 

SAVE JIRA TASK (LINK TO INITIATING REQUIREMENT AUTOMATICALLY CAPTURED)

THE ORIGINATING REQUIREMENTS "DOCUMENT" IN CONFLUENCE AUTOMATICALLY UPDATES WITH NEWLY ASSIGNED TASKS

zAgile's Semantic Search - an OpenSocial gadget for Confluence and JIRA

The support for OpenSocial gadgets in the latest releases of Atlassian's products (specifically JIRA 4.0 and Confluence 3.1) provides some interesting opportunities for creating dashboards as well as aggregating information into the application from external sources to enrich the overall information context.

This capability is even more relevant to us at zAgile because of the core premise of our platform of semantically integrating information from heterogeneous sources (tools) in a contextual manner to provide a more cohesive view.  Hence, we have been exploring this capability within our solutions.  One of the easiest and most intuitive is our implementation of Semantic Search.  This functionality has been available in zAgile Teamwork from the very beginning, but now to implement it as an OpenSocial gadget made it portable, both within the applications as well as outside.  Within JIRA and Confluence, gadgets can be placed anywhere, particularly for building various dashboards or views.  Outside of JIRA and Confluence, OpenSocial gadgets can be incorporated into any compliant container.   Once you have a unified information repository, then it is more compelling to be able to access that information from a variety of sources rather than resorting to specific container applications.  This is where our platform comes in.

zAgile's Semantic Search has the following characteristics:

1. It accesses a single composite information repository which comprises of information from multiple sources, such as JIRA, Confluence (and soon) Subversion, CruiseControl, SalesForce, etc.  
2. It is domain independent, driven solely by underlying ontologies and metamodels implemented as OWL-DL triple stores comprising of inferred and indexed models.  Therefore,
3. 1. It can search based on taxonomic structures (ex: searching for an Issue or a specific Issue type such as Bug, Task, etc.).  Furthermore, you can also filter on specific properties (ex: search for a Jira Project that implements a specific Requirement).
   2. It can search for inferred categories (a Person or an Author; an Issue or a Resolved Issue)
   3. The results returned are of a specific category (specified in the search parameter) so you are not going to get anything that matches a particular search string.  Search here is much more precise.
   4. It also returns semantic properties of each instance in the result set (ex: project related to an issue, issue status, related requirement, related test cases, etc.)
   5. It provides semantic (contextual) navigation to related items (from the issue navigate to its related project and project properties, from the project, navigate to project overview document, etc.) -- that without leaving the page or gadget. 
   6. Direct access to the resource - at any point during the search, you can go to the specific resource (issue screen, requirements page, etc.)

All of this is a manifestation of the semantically captured information from various sources.  How powerful is this that you can view information from within Confluence that may actually be created by JIRA or SVN or any other tool, without needing to explicitly specify it on the page or even be aware of it.   This begins to shift the pradigm from an application-centric environment to one that is more information-centric.

Finally, what if there are multiple Confluence or JIRA instances?  Well, in this case, they all show up in a unified manner, and references to specific entities within these applications are captured so that navigation directly to the source is always available.

You can test drive the capabilities of zAgile's Smart Search (or Semantic Search) in the sandbox.  A login is required with email validation and here you will find both Confluence and JIRA hosting this feature. 

-Sanjiva

The screenshot below shows the results of a search for JIRA Projects in an OpenSocial gadget in a Confluence page.  Project properties including related entities (such as Requirements, etc) are also shown along with drill down into one of the related entities.

Semantic Enablement of Wikis for Enterprise Information Integration

Yesterday, I gave a "lightning talk" (3 min prezo) at SemanticWiki mini-series Session-2 hosted by Ontolog.  It was interesting to hear the evolution of semantic wikis to date and the varied focus and unique applications offered by various organizations.  Its even more interesting that with all that these wikis promise, they have not picked up much attention in an enterprise.

Here is the rough text that accompanied my presentation:

Our focus at zAgile is towards information integration within the enterprise -- towards which, the wiki is only one of many potential sources of information or knowledge.  Rather than treat the wiki as THE knowledge repository, we have developed technology and extensions which allow the wiki (as well as other applications) to contribute to a central information repository, although the wiki may also functions as a 'portal' or 'dashboard' for integrated information.  In this regard, we have focused on "semantically enabling" commercially deployed wikis such as Atlassian's Confluence. 

 

The semantic enablement consists of three major components:

1. Within the wiki -- it is Templates, Forms and Query support. 
   1. Templates are based on pre-defined ontologies, each one potentially representing a single concept, its attributes and relations, such as Artifact, Person, Task, etc.  Templates support pre-population of instance-specific data from the repository via embedded SPARQL queries
   2. Forms are based on templates, to allow users to create hybrid pages -- consisting of semantically structured as well as free-form content -- in an implementation similar to the work done for MediaWiki.
   3. And of course, you can write SPARQL/RDQL type queries and integrate and enrich wiki content with information from external sources.     
2. zAgile's ontology-based repository consisting of domain-specific ontologies (our current focus is on ontologies for software engineering)
3. zAgile's semantic framework which provides a consistent mechanism for all applications, including wikis to interact with the repository

 

The central ontology-based repository provides for unification of information across the enterprise, incorporating what so far has been relatively "unstructured content" that lives in the wikis and offers little contextual value. 

 

-Sanjiva, Prague, Nov '08

Confluence is now Wiki'd Sem'art ... and it SPARQLs

Of course, there is a lot of pun intended here.  We (zAgile) recently added semantic capabilities to Atlassian's Confluence wiki and are quite excited about its impact to a very popular yet traditional wiki.

In my earlier blogs, I had been talking about my quest for semantic wikis.  Wikis provide an easy way for teams to produce and organize content--creating very effective central repositories, often used for cross-departmental collaboration and information exchange.  The problems arise when all this statically organized content becomes unwieldy in terms of maintenance, and more importantly access.  You end up spending lots of time in wildcard searches across all this content, repetitively navigating complex hierarchical page structures and constantly maintaining links across wiki pages to keep the content credible and current.

 

Semantic capabilities make wikis a dynamic application.  Addressing the problems noted above, they allow for content to be 'semantically' annotated with the annotations stored in  semantic repositories, therefore, the meaning is captured in a machine-processable way and is more directly searchable, shareable and accessible across applications not just from within the wiki itself.

 

To 'semantically' enable Confluence, we have taken the following steps:

1. Provided support for SPARQL in Confluence.  Analogous to SQL, you can embed SPARQL queries anywhere in a page and the data may be retrieved from semantic repositories anywhere on the web.  This allows for auto integration/incorporation of data from external systems into the wiki.  These queries can also provide a mechanism for dynamically organizing more readily addressable wiki content. 

2. Provide Semantic Forms in the wiki which allow users to create 'structured' or semantically annotated content.  The data and object attributes associated with a page created using such a form are captured in the semantic repository while the page maintains a composite of both semantic and unstructured content.  We have used such forms to create Feature/Requirements Documents on a wiki page.  Since the page is annotated using Requirements ontology, this provides a very effective approach to link requirements to projects, tasks, bugs, test cases, etc. in a software engineering lifecycle -- all data that exists in tools external to the wiki. 

3. Provide macros that allow users to annotate any existing page on a wiki based on the available set of ontologies (Project, Person, Document, Process, etc.).  Again, semantic annotations provide a deeper and more structured mechanism than just metadata tags for capturing relevant information from each page and organizing it within the wiki in a more meaningful way as well as integrating it with external applications.

                                          

And lastly, this semantic capability can now be added to any extensible wiki -- to make it wiki'd semart and sparql'ing :)

                                                

-Sanjiva, Orinda, October '08