The S Curve

I will be speaking at SemTech 2010 - June 21-25 in San Francisco

Orthopaedia - A Semantic Platform for Collaborative Knowledgebase Development in Orthopaedics

Semantic technologies have been aggressively applied in biomedical and healthcare domains to address a number of problems related to the representation, interpretation, integration and access of knowledge. These technologies include the development of machine readable thesauri and vocabularies, taxonomies, formal ontologies as well as NLP tools and techniques. And of course, there are emerging applications that are being developed to deal with these underlying metamodels.  

This case study discusses a semantic platform that incorporates these technologies and metamodels towards the collaborative development of a knowledgebase in Orthopaedics. The platform enables users to easily enhance local content with publications from various sources, blogs, discussions, annotations, and automatically tag them with domain-specific terms, as well as make this information accessible across the enterprise to other tools and applications via a semantic repository.

We will also describe the current challenges in achieving coherency and integration across medical subdomains, and describe some the design decisions taken to overcome some of these challenges.

We will address the following points:

• Overview of semantic technologies in healthcare
• Challenges in semantic information integration across resources and applications
• Objectives of the project
• Discussion of technologies implemented
• Review of strengths and weaknesses of the current state of technologies in addressing knowledge management issues in healthcare
• Demo

 

Achieving Requirements Traceability with Confluence & JIRA, the rational (!Rational) way

Requirements Traceability refers to the ability to link requirements to projects, change requests, development tasks, test cases and results, checkins, builds, releases and stakeholders.   In other words - linking requirements with all relevant concepts associated with a software engineering life cycle. Given that Requirements drive the activities across the software engineering projects, maintaining such interrelationships can provide answers to some very critical and business-relevant questions relative to each requirement, whether they are related to business justification, auditing/compliance, delivery management or quality assessment.  

These can be grouped by stages and constituents (some examples below). 

Business Justification:

Who are the Stakeholders of each requirement?

Which requirements are being delivered in a release and for which stakeholders?

What is the business justification and cost of delivering a feature or requirement?

What is the ROI (once the costs are determined)?

Project Management:

Which release will the requirement be delivered in?

What are the risks and dependencies related to the implementation of a requirement?

What is the current state of a requirement with respect to development and QA?

Who signed off on it, who accepted it? 

Which checkins correspond to code complete for a requirement or feature?

Which build includes a particular requirement implementation?

Development & QA 

What are the design goals and motivations of a requirement?

What are the associated test cases and quality goals?

What are the associated use cases or user stories?

In typical engineering environments, some of these questions are difficult or impossible to determine, while others require a significant amount of tedious manual effort.  The reasons can be mostly attributed to the heterogeneity of ALM tools and the lack of integration between them.  Whatever the cause of this fragmentation of information, requirements traceability isn't always optional for organizations. Many expend significant human effort and co$t towards collecting sufficient traceability data to maintain their audit and compliance mandates.  

Traditional ALM vendors provide their own integrated solution set to help their customers achieve requirements traceability.  While they are able to address most, if not all, questions above, you have to invest into their entire portfolio of products and at significant co$ts.  I have rarely seen an engineering environment that has successfully adopted a single vendor solution across teams and departments.  For most part, such solutions end up as shelf-ware, reflecting wasteful and irRational decisions.

It is with this in mind that zAgile delivers its tool-agnostic and open source zCALM platform that is capable of integrating concepts across software engineering life cycle with the tools that you are using in YOUR environment. 

And to take that even further, you can achieve significant amount of requirements management and traceability by simply integrating your Confluence wiki and JIRA issue tracking application via zTeamwork.

  zTeamwork allows you to: 

1. Capture features, requirements, use cases, test cases, test suites, etc. in Confluence in a semi-structured way using zAgile's smart templates.  
2. Link them to each other using pre-defined relationships.
3. Create JIRA tasks, link them to various Confluence artifacts created above, so that you can now associated requirements with use cases, test cases, tasks, and bugs
4. View any and all of these links and relationships (including attributes such as status, completion, etc.) in Confluence or JIRA using simple queries.  There are JIRA properties tabs that may be configured to provide 'dashboard' type contextual views of this information.
5. Search for these specific concepts from within Confluence or JIRA, regardless of where they are created.  The search not only returns the matching result set but also all of the contextual relationships.

Teamwork's integration can be further extended with connectors for salesforce (link customers and cases to requirements and issues within your environment), Subversion (associate checkins with requirements) and CruiseControl (link requirements to builds).  

 With these connectors in place, you can answers ALL of the questions above with respect to requirements traceability -- without having to buy into expensive, complex and painful single-vendor solutions.  

And since zAgile's solutions are tool agnostic, you can also plug other tools on its platform to achieve the same integration capability - via custom connectors.

Somehow that seems more rational than Rational :)

-Sanjiva

Test drive these features in the sandbox

Machine-based semantic annotation of content in Confluence Enterprise Wiki

Annotation of content in any CMS, including traditional wikis, to provide contextual reference to any word or phrase is a common practice.  I am doing that in my blogs as I write.  When I reference specific topics or items, I go through the steps of creating links to them. 

Wiki annotation, therefore, also typically involves highlighting words or phrases and associating each to some reference - an external link or another page within the wiki.   It is a tedious process that involves marking up each word or phrase using specific wiki tags.  It not only requires considerable effort but also a meticulous nature and some dedication towards creating meaningful content.

What if it were automatic?  All you had to do is to add a macro at the beginning of your paragraph or page and everything meaningful within that text would automatically be annotated.

What if it were dynamic, so that the wiki content is still in its original form but all annotations are done when the page is rendered.  The annotation would automatically reflect any changes to the reference.  It would be more reliable and not prone to going stale.

What if the annotation was more contextual to your specific domain, to your business processes and functions, rather than highlighting of people, countries, organizations -- which may have less relevance when the text if referring to some technical requirements for a software component.

And finally, what if the annotation not only provided a link to some reference, but also some information about its type (is it a reference to a Project, a Requirement, a Task, a Bug, a Customer, a Software Component, ...)

Well, that's the capability that we will soon release for Atlassian's Confluence Enterprise Wiki in zAgile Teamwork and Wikidsmart products.

A simple, yet powerful capability in our upcoming release of Wikidsmart (Release 2.0) is the machine-based semantic annotation of wiki content.  It is easier to play with this feature than to describe it to really understand the possibilities that it opens up in making content more meaningful and accessible in enterprise wikis.  But here is an explanation:

By machine-based, I mean all you need to do to enable this capability is to surround the wiki paragraph with a macro (zannotate).

By semantic annotation, I mean that words or phrases in that paragraph that match any reference in our ontology-based semantic repository will be highlighted and linkable to additional information about the references.  Since zAgile Teamwork ships with ontologies for software engineering, you will see annotations of words and phrases that reference instances of Requirements, Projects (JIRA Projects), JIRA Issues, Tasks, Bugs, Software Components, Test Cases, etc.

You can test drive this capability in the zTeamwork sandbox.

Here is what it would look like:

WRITE A PARAGRAPH IN A WIKI AND BRACKET IT WITH ANNOTATION MACRO

SAVE THE PAGE AND THE PARAGRAPH WILL BE AUTOMATICALLY ANNOTATED.  CLICKING ON ANY TAGGED PHRASE WILL POPUP ITS REFERENCE CONTEXT

FOR EXAMPLE, REFERENCES TO JIRA ISSUES NOW ARE AUTOMATICALLY ANNOTATED WITH TYPE, DESCRIPTIONS AND LINKS TO THE INDIVIDUAL ISSUES

AND FINALLY, THE SAME APPLIED TO MEDICAL TEXT - APPLIED AGAINST AN  ONTOLOGY DERIVED FROM MeSH (Medical Subject Headings) THESAURUS, ALSO IN CONFLUENCE - SHOWING HIGHLIGHTED TERM AND ITS SYNONYMS

- Sanjiva

To test drive this feature, visit our sandbox

Creating JIRA Tasks from within Confluence Requirements Pages

One of the recent features delivered in zAgile Teamwork is the ability to create a JIRA Task from within a Confluence page and have the Task automatically inherit the context of the Requirement that is driving it.  The use case description below explains this 'context inheritance'.

This is also a great examples of the interoperability that can be achieved between heterogeneous applications (JIRA, Confluence in example but almost any app) using semantic technologies.  In this case, it is our zCALM semantic application server, that is enabling this seemingly simple scenario.

The use case is as follows:

1. Using one of the example semantic templates in Teamwork, you define a list of Requirements in a Confluence page 

2. Each Requirement on that page may be linked to one or many Design Goals that is the motivation for it, as well as a specific JIRA Project in which it is planned for delivery. 

Note: It is also possible to modify the template to associate all of the requirements on the page to a single project and map the individual requirement to a specific version. 

3. Once a certain number of requirements have been assigned to corresponding JIRA projects, you can use the Create Issue link to launch a JIRA Issue Creation page.  Assuming you also have a JIRA account and issue creation privilege for the project, you will notice a number of things:

1. The JIRA Issue Creation page is automatically associated with the project to which the Requirement was linked.
2. The issue is of type Task
3. The Issue Creation page is pre-filled with Summary and Description fields from the Requirement itself
4. The Issue Reporter is initialized to the person logged in
5. And more significantly, a custom field in JIRA showing links to "Related Requirements" has the originating requirement flagged. 

At this point, all you have to do after clicking on the Create Issue link is to press the Create button on the issue creation page.  The saved Task in JIRA will be automatically linked to the Requirement for which it was created.

And finally, the Requirements Document from where all this began will also automatically show the created JIRA Tasks,  their corresponding Requirements, status, etc.

A simple use case but a very powerful paradigm for Enterprise Application Interoperability.  All of this integration is driven by an underlying semantic infrastructure that maintains the relationship across the concepts that various tools represent.

You can Test-Drive this functionality in the sandbox for Teamwork.

-Sanjiva

PS - To test drive this feature, visit our sandbox

The screenshots below show the flow from Confluence Requirements Document page to JIRA Issue Creation page and back, reflecting the context that carries across the two applications automatically.

CAPTURE REQUIREMENTS AND ASSOCIATE WITH JIRA PROJECT(S)

 

AUTO-GENERATE PRE-FILLED JIRA ISSUE CREATION PAGE (WITH LINK TO INITIATING REQUIREMENT)

 

SAVE JIRA TASK (LINK TO INITIATING REQUIREMENT AUTOMATICALLY CAPTURED)

THE ORIGINATING REQUIREMENTS "DOCUMENT" IN CONFLUENCE AUTOMATICALLY UPDATES WITH NEWLY ASSIGNED TASKS

zAgile's Semantic Search - an OpenSocial gadget for Confluence and JIRA

The support for OpenSocial gadgets in the latest releases of Atlassian's products (specifically JIRA 4.0 and Confluence 3.1) provides some interesting opportunities for creating dashboards as well as aggregating information into the application from external sources to enrich the overall information context.

This capability is even more relevant to us at zAgile because of the core premise of our platform of semantically integrating information from heterogeneous sources (tools) in a contextual manner to provide a more cohesive view.  Hence, we have been exploring this capability within our solutions.  One of the easiest and most intuitive is our implementation of Semantic Search.  This functionality has been available in zAgile Teamwork from the very beginning, but now to implement it as an OpenSocial gadget made it portable, both within the applications as well as outside.  Within JIRA and Confluence, gadgets can be placed anywhere, particularly for building various dashboards or views.  Outside of JIRA and Confluence, OpenSocial gadgets can be incorporated into any compliant container.   Once you have a unified information repository, then it is more compelling to be able to access that information from a variety of sources rather than resorting to specific container applications.  This is where our platform comes in.

zAgile's Semantic Search has the following characteristics:

1. It accesses a single composite information repository which comprises of information from multiple sources, such as JIRA, Confluence (and soon) Subversion, CruiseControl, SalesForce, etc.  
2. It is domain independent, driven solely by underlying ontologies and metamodels implemented as OWL-DL triple stores comprising of inferred and indexed models.  Therefore,
3. 1. It can search based on taxonomic structures (ex: searching for an Issue or a specific Issue type such as Bug, Task, etc.).  Furthermore, you can also filter on specific properties (ex: search for a Jira Project that implements a specific Requirement).
   2. It can search for inferred categories (a Person or an Author; an Issue or a Resolved Issue)
   3. The results returned are of a specific category (specified in the search parameter) so you are not going to get anything that matches a particular search string.  Search here is much more precise.
   4. It also returns semantic properties of each instance in the result set (ex: project related to an issue, issue status, related requirement, related test cases, etc.)
   5. It provides semantic (contextual) navigation to related items (from the issue navigate to its related project and project properties, from the project, navigate to project overview document, etc.) -- that without leaving the page or gadget. 
   6. Direct access to the resource - at any point during the search, you can go to the specific resource (issue screen, requirements page, etc.)

All of this is a manifestation of the semantically captured information from various sources.  How powerful is this that you can view information from within Confluence that may actually be created by JIRA or SVN or any other tool, without needing to explicitly specify it on the page or even be aware of it.   This begins to shift the pradigm from an application-centric environment to one that is more information-centric.

Finally, what if there are multiple Confluence or JIRA instances?  Well, in this case, they all show up in a unified manner, and references to specific entities within these applications are captured so that navigation directly to the source is always available.

You can test drive the capabilities of zAgile's Smart Search (or Semantic Search) in the sandbox.  A login is required with email validation and here you will find both Confluence and JIRA hosting this feature. 

-Sanjiva

The screenshot below shows the results of a search for JIRA Projects in an OpenSocial gadget in a Confluence page.  Project properties including related entities (such as Requirements, etc) are also shown along with drill down into one of the related entities.

zAgile @ '09 Semantic Technology Conference

This year's Sem Tech brings exciting opportunities for us.  I will be participating in two sessions. 

1. Semantic Wikis -- which includes participants from Vulcan, SRI, ontoprise, University of Karlsruhe and Stanford University

2. Semantic Framework for Model Driven Process Deployment and eXtreme Traceability -- a presentation that I will share with Dan Pattyn who will provide real-world use cases for eXtreme Traceability towards which our proposed solutions are most applicable.

The Sem Tech topic list this year continues to be very exciting and demonstrative of the rapid evolution and mainstream adoption of the technology.  

Can't wait for Sunday.

Semantic Technologies within Enterprise Data Management

Today, I had the opportunity to be on the panel at Enterprise Data World to discuss the role of semantic technologies in enterprise data management.  The panel followed a working session consisting of roundtable discussions exploring a number of specific topics related to this theme.  All-in-all, a very interesting and revealing set of discussions.

The two most revealing points that stuck were:

1. There is significant confusion between semantic technologies and semantic web.  In some respect, this is a bit embarrassing for the community that is trying to define semantics of other things.  The consensus was that we need to keep the two separate and define each clearly.  The challenges and solutions are not the same between the two.  I mostly agree with this.

2. Semantic technologies seem to threaten the status quo in the enterprise data community -- in much the same way as object-oriented programming did to the C-development community in the late 80's (so it seemed to me, anyway).  I didn't see this coming.  "we can model anything using our current techniques and technologies -- so why do we need semantic xxx".   The reluctance to adopt this recently popularized technology (I won't call it new) was quite significant. 

I thought it was important to note some of the thoughts that came out of this group (without any perspectives, endorsements or editorials):

Semantic technologies-

1. do the same job as data modeling but better/faster.

2. add web-centric web aspect to the table-centric world of relational databases.  Are easily extensible (rather than expanding tables after tables)

3. Ontologies provide significant flexibility in modeling

4. means that a term is defined by its relationship to other concepts instead of text paragraphs

5. and definitions can capture logical discrepancies

6. provide for more precisely defined taxonomic terms

We will try to take these discussions and perspectives to Semantic Technology conference in June.  It will be interesting to see how these perspectives evolve in the near-term.

-Sanjiva, Tampa, April 8, 2009   

Information and Knowledge Integration in an Enterprise

 Recently I came across an article by Ricardo Falbo, et al. from UFES (Brazil) that identifies various dimensions of information and knowledge integration that I wanted to share.  These are:

 

• Data Integration
• Presentation Integration (enterprise portals and mashups)
• Control Integration (what we often think of as application integration)
• Process Integration (see my earlier blog on this)
• Platform Integration
• Knowledge Integration (emphasizing knowledge capture through rich semantic definitions)

 

The context of the discussion is ODE (Ontology-based Software Development Environment), developed at UFES, but it is quite applicable to the needs of any enterprise and not just Software Engineering.  Also, while most enterprise solutions focus primarily on data integration or application integration, none consider all these in a single composite offering.  In that regard, through the application of semantic technologies (particularly a rich set of ontologies), ODE is one of the first systems that I have seen which demonstrates how one could offer such a complete and seamless integration.  While ODE attempts to do this via a full blown application, I believe that soon we will be able to achieve this level of integration across heterogeneous applications in an enterprise by combining some of the recently evolving technologies, including those related to the semantic web.  An example of one of these was recently mentioned in Bob Zureks' blog   

 

Sanjiva, Orinda, Nov '08.  

Making Minimum Ontological Commitments

In my last blog, I cited some examples of inconsistent semantics that are prevalent in applications as well as in their usability. Applications tend to be siloed in nature and their schema often corresponds more tightly to specific dialects. The end users make it worse by often using it for multiple purposes, thereby muddying the semantics even more. Also as I mentioned in the blog, this becomes a nightmare for those striving for an Information Integration strategy (a cohesive platform that unifies information across tools, departments and processes). Such strategies and solutions have been championed by some of the biggest vendors in the applications and infrastructure space (IBM, SAP, Oracle) -- via their MDM (Master Data Management) or EIM (Enterprise Information Management) strategies. 

But how does one effectively address this EIM challenge? 

To us, the answer is relatively simple. By making "minimum ontological commitments". Of course the devil is in the implementation details. 

• First of all, it involves a set of interconnected ontologies (or sub-ontologies) which together provide a specification for a domain (in our case, Software Engineering). Examples of these sub-ontologies would be SCM, Process, Person, Document, Test, Requirement, Collaboration, etc. These ontologies provide a consistent and coherent vocabulary that can be shared amongst applications, processes and people within a domain [Gruber]. The focus here is on consistent sharable vocabulary and not necessarily a mechanical mapping of schemas across applications. Each application may still have its own dialect that is not relevant beyond its sub domain -- therefore the emphasis on making minimum ontological commitments.
• Second, an infrastructure that maps and captures this semantically relevant information from each application (or tool) into the semantic repository (aka Information Integration Framework).
• Third, semantically enabling the applications and the overall platform so that the same information is accessible to all participants and agents using the same vocabulary and taking it a notch higher -- creating composite views or dashboards from this integrated information to alter the traditional metaphors of how we see things.
• Fourth, leverage reasoning capabilities from the ontological specifications to get beyond the data that is empirically captured and enrich the knowledge base.

  So it is quite straightforward :)

 

    In concrete terms, such commitments can answer simple questions, such as:

Who broke the build, what tests were conducted for a particular set of checkins, who reviewed them, what requirements or bugs initiated the actions, who authorized the final move to production of a particular set of changes, what customers are impacted, which components are most vulnerable with respect to defect density at a particular development stage, etc. 

  Of course, we can get these answers now but it either requires a lot of walking the hallways and late night phone conversations with remote teams to gather the information across teams and silos, or making some costly commitment to an integrated toolset from one particular vendor.

   

  Or, the questions above can be more directly and easily (and automatically) answered in your own environment with your own processes and tools -- by integrating the zAgile semantic infrastructure.

  Its about Information Integration in your world.

   

  Its about building relevant knowledge through this integration that will allow you to effectively collaborate, coordinate and manage your software development and delivery cycles.

-Sanjiva, Orinda, Oct '08

The value of semantics in applications

Commercial software applications often tend to be quite loose and non-binding of any conventional semantics when it comes to implementation of their metamodels.  This not only impairs their usability but also has significant impact to their ability to integrate with other applications and systems in an environment. 

   

A case in point -- Atlassian's Jira issue management application -- which has become quite popular with organizations of all sizes as well as a number of open source communities.  With over 10,000 customers for its products, it is no wonder that we (zAgile) have been taking a closer look at the Atlassian application suite with respect to integrating them into our semantic framework for our customers.   

   

One of the most curious aspects of Jira that has often perplexed potential adopters is its metamodel.  We see people often confused with some of its key concepts or use them  arbitrarily thus potentially contributing to a different set of usability problems.         

 

Most notably in this context is the Jira 'Project'.  This defies any coherence with the common or conventional notions of a Project.  For starters, people tend to associate projects with a begin and end date.  Not so in Jira.  Projects go on forever with interim deliveries packaged as releases.  So what is a Jira project and how do you plan one? 

   

Then there is Component.  It seemed intuitive at first since semantics of the word 'component' are also not too controversial--that is until you begin to see how people have applied it in practice in Jira.   

To understand the usages of some of these fields, you can look at Atlassian's JIRA site.  Another popular Jira site for us has been JBoss.

   

In both usage models, the concept of Project more closely resembles a software product or component (example -- Jboss Enterprise Platform, JEMS, Jboss Shotoku, Bamboo, Clover, JIRA, etc).  There is no hierarchy or crossover relationship between software components so the issues or tasks within each have to be fairly isolated from those belonging to another Project.  Of course, there are Project Categories (example: Atlassian Products, JIRA Plugins, etc) -- to help organize the Projects.  But how do those fit into our mental model of Projects? 

   

Similarly, I have seen examples of Components such as QA, Performance, Test Suite and Documentation.  Its tough to figure out the definition (or semantics) of Component from such usage. 

 

And finally, if we are going to be arbitrary in terms of how we use these fields, then how do we integrate them with similar concepts in other tools?   Is a Jira Project the same as an MS Project instance?  Are all values in Component representing software components? 

      

This emphasizes the value of common and shared semantics in applications that would facilitate both usability and potentially integration.  While Jira has a lot of third-party support to make it functionally rich, a lot of people are still spending significant amount of effort towards integrating it with other applications and processes within their environment.  And many of them are struggling with the lack of consistent and coherent semantics.

      

   -Sanjiva, Orinda, October '08