The S Curve

zAgile @ '09 Semantic Technology Conference

This year's Sem Tech brings exciting opportunities for us.  I will be participating in two sessions. 

1. Semantic Wikis -- which includes participants from Vulcan, SRI, ontoprise, University of Karlsruhe and Stanford University

2. Semantic Framework for Model Driven Process Deployment and eXtreme Traceability -- a presentation that I will share with Dan Pattyn who will provide real-world use cases for eXtreme Traceability towards which our proposed solutions are most applicable.

The Sem Tech topic list this year continues to be very exciting and demonstrative of the rapid evolution and mainstream adoption of the technology.  

Can't wait for Sunday.

Semantic Technologies within Enterprise Data Management

Today, I had the opportunity to be on the panel at Enterprise Data World to discuss the role of semantic technologies in enterprise data management.  The panel followed a working session consisting of roundtable discussions exploring a number of specific topics related to this theme.  All-in-all, a very interesting and revealing set of discussions.

The two most revealing points that stuck were:

1. There is significant confusion between semantic technologies and semantic web.  In some respect, this is a bit embarrassing for the community that is trying to define semantics of other things.  The consensus was that we need to keep the two separate and define each clearly.  The challenges and solutions are not the same between the two.  I mostly agree with this.

2. Semantic technologies seem to threaten the status quo in the enterprise data community -- in much the same way as object-oriented programming did to the C-development community in the late 80's (so it seemed to me, anyway).  I didn't see this coming.  "we can model anything using our current techniques and technologies -- so why do we need semantic xxx".   The reluctance to adopt this recently popularized technology (I won't call it new) was quite significant. 

I thought it was important to note some of the thoughts that came out of this group (without any perspectives, endorsements or editorials):

Semantic technologies-

1. do the same job as data modeling but better/faster.

2. add web-centric web aspect to the table-centric world of relational databases.  Are easily extensible (rather than expanding tables after tables)

3. Ontologies provide significant flexibility in modeling

4. means that a term is defined by its relationship to other concepts instead of text paragraphs

5. and definitions can capture logical discrepancies

6. provide for more precisely defined taxonomic terms

We will try to take these discussions and perspectives to Semantic Technology conference in June.  It will be interesting to see how these perspectives evolve in the near-term.

-Sanjiva, Tampa, April 8, 2009   

Information and Knowledge Integration in an Enterprise

 Recently I came across an article by Ricardo Falbo, et al. from UFES (Brazil) that identifies various dimensions of information and knowledge integration that I wanted to share.  These are:

 

• Data Integration
• Presentation Integration (enterprise portals and mashups)
• Control Integration (what we often think of as application integration)
• Process Integration (see my earlier blog on this)
• Platform Integration
• Knowledge Integration (emphasizing knowledge capture through rich semantic definitions)

 

The context of the discussion is ODE (Ontology-based Software Development Environment), developed at UFES, but it is quite applicable to the needs of any enterprise and not just Software Engineering.  Also, while most enterprise solutions focus primarily on data integration or application integration, none consider all these in a single composite offering.  In that regard, through the application of semantic technologies (particularly a rich set of ontologies), ODE is one of the first systems that I have seen which demonstrates how one could offer such a complete and seamless integration.  While ODE attempts to do this via a full blown application, I believe that soon we will be able to achieve this level of integration across heterogeneous applications in an enterprise by combining some of the recently evolving technologies, including those related to the semantic web.  An example of one of these was recently mentioned in Bob Zureks' blog   

 

Sanjiva, Orinda, Nov '08.  

Making Minimum Ontological Commitments

In my last blog, I cited some examples of inconsistent semantics that are prevalent in applications as well as in their usability. Applications tend to be siloed in nature and their schema often corresponds more tightly to specific dialects. The end users make it worse by often using it for multiple purposes, thereby muddying the semantics even more. Also as I mentioned in the blog, this becomes a nightmare for those striving for an Information Integration strategy (a cohesive platform that unifies information across tools, departments and processes). Such strategies and solutions have been championed by some of the biggest vendors in the applications and infrastructure space (IBM, SAP, Oracle) -- via their MDM (Master Data Management) or EIM (Enterprise Information Management) strategies. 

But how does one effectively address this EIM challenge? 

To us, the answer is relatively simple. By making "minimum ontological commitments". Of course the devil is in the implementation details. 

• First of all, it involves a set of interconnected ontologies (or sub-ontologies) which together provide a specification for a domain (in our case, Software Engineering). Examples of these sub-ontologies would be SCM, Process, Person, Document, Test, Requirement, Collaboration, etc. These ontologies provide a consistent and coherent vocabulary that can be shared amongst applications, processes and people within a domain [Gruber]. The focus here is on consistent sharable vocabulary and not necessarily a mechanical mapping of schemas across applications. Each application may still have its own dialect that is not relevant beyond its sub domain -- therefore the emphasis on making minimum ontological commitments.
• Second, an infrastructure that maps and captures this semantically relevant information from each application (or tool) into the semantic repository (aka Information Integration Framework).
• Third, semantically enabling the applications and the overall platform so that the same information is accessible to all participants and agents using the same vocabulary and taking it a notch higher -- creating composite views or dashboards from this integrated information to alter the traditional metaphors of how we see things.
• Fourth, leverage reasoning capabilities from the ontological specifications to get beyond the data that is empirically captured and enrich the knowledge base.

  So it is quite straightforward :)

 

    In concrete terms, such commitments can answer simple questions, such as:

Who broke the build, what tests were conducted for a particular set of checkins, who reviewed them, what requirements or bugs initiated the actions, who authorized the final move to production of a particular set of changes, what customers are impacted, which components are most vulnerable with respect to defect density at a particular development stage, etc. 

  Of course, we can get these answers now but it either requires a lot of walking the hallways and late night phone conversations with remote teams to gather the information across teams and silos, or making some costly commitment to an integrated toolset from one particular vendor.

   

  Or, the questions above can be more directly and easily (and automatically) answered in your own environment with your own processes and tools -- by integrating the zAgile semantic infrastructure.

  Its about Information Integration in your world.

   

  Its about building relevant knowledge through this integration that will allow you to effectively collaborate, coordinate and manage your software development and delivery cycles.

-Sanjiva, Orinda, Oct '08

The value of semantics in applications

Commercial software applications often tend to be quite loose and non-binding of any conventional semantics when it comes to implementation of their metamodels.  This not only impairs their usability but also has significant impact to their ability to integrate with other applications and systems in an environment. 

   

A case in point -- Atlassian's Jira issue management application -- which has become quite popular with organizations of all sizes as well as a number of open source communities.  With over 10,000 customers for its products, it is no wonder that we (zAgile) have been taking a closer look at the Atlassian application suite with respect to integrating them into our semantic framework for our customers.   

   

One of the most curious aspects of Jira that has often perplexed potential adopters is its metamodel.  We see people often confused with some of its key concepts or use them  arbitrarily thus potentially contributing to a different set of usability problems.         

 

Most notably in this context is the Jira 'Project'.  This defies any coherence with the common or conventional notions of a Project.  For starters, people tend to associate projects with a begin and end date.  Not so in Jira.  Projects go on forever with interim deliveries packaged as releases.  So what is a Jira project and how do you plan one? 

   

Then there is Component.  It seemed intuitive at first since semantics of the word 'component' are also not too controversial--that is until you begin to see how people have applied it in practice in Jira.   

To understand the usages of some of these fields, you can look at Atlassian's JIRA site.  Another popular Jira site for us has been JBoss.

   

In both usage models, the concept of Project more closely resembles a software product or component (example -- Jboss Enterprise Platform, JEMS, Jboss Shotoku, Bamboo, Clover, JIRA, etc).  There is no hierarchy or crossover relationship between software components so the issues or tasks within each have to be fairly isolated from those belonging to another Project.  Of course, there are Project Categories (example: Atlassian Products, JIRA Plugins, etc) -- to help organize the Projects.  But how do those fit into our mental model of Projects? 

   

Similarly, I have seen examples of Components such as QA, Performance, Test Suite and Documentation.  Its tough to figure out the definition (or semantics) of Component from such usage. 

 

And finally, if we are going to be arbitrary in terms of how we use these fields, then how do we integrate them with similar concepts in other tools?   Is a Jira Project the same as an MS Project instance?  Are all values in Component representing software components? 

      

This emphasizes the value of common and shared semantics in applications that would facilitate both usability and potentially integration.  While Jira has a lot of third-party support to make it functionally rich, a lot of people are still spending significant amount of effort towards integrating it with other applications and processes within their environment.  And many of them are struggling with the lack of consistent and coherent semantics.

      

   -Sanjiva, Orinda, October '08

Learning from Rachel Dawes

"But it's not who you are underneath, it's what you do that defines you"  -- Rachel Dawes in Batman Begins      

         

This catchy quote from Batman Begins had me a bit puzzled when I first saw the movie.  What does she mean by that, I thought?  But it stuck.  If you have seen the movie, you will understand why.      

                  

And then in a totally different context, while trying to submit zAgile to various search engines and business directories, it became somewhat clearer.  The problem I encountered had to do with figuring out where to categorize zAgile.  Yahoo's taxonomy seemed pretty dated and perhaps severely constraining.  It shouldn't be so difficult, I thought.  But it was.  Why do I have to fit zAgile into some pre-defined category?  Why can't I just say 'what it does' and have 'what it is' inferred?  The problem with Categories is that an individual may belong to more than one category and the categorizations are not necessarily permanent.  Categories definitely cannot be used to define what something 'is'. 

          

So it seems that we are constantly constraining ourselves by putting 'things' into specific categories.  When you have to put a book on a shelf in a library then obviously you need to make some hard choices.  But in life, what something 'is' ought to be defined (and inferred) based on its attributes and relations to other things.   

                            

In a somewhat obtuse, indirect and 'Heideggerian' sort-of way, maybe Rachel was trying to say the same thing.  Her quote curiously made me realize the key difference between taxonomy and ontology.  We are so used to dealing with taxonomies when developing applications that the power and flexibility of ontologies are difficult to grasp and leverage.  While ontologies often take the form of taxonomic class hierarchies, they are not constrained by them.  Therefore, rather than struggle with categorizing something, we use an ontology to define it using its characteristics (or attributes) and relationships to other things.  And all that it 'IS' is simply inferred.      

      

And that is all what Rachel was trying to tell Bruce Wayne (so I believe, anyway).         

         

-Sanjiva

On The Power of Semantic Wikis

While the popularity of wikis in the corporate world continues to grow due to the ease with which they allow anyone to publish web-based, shareable and easily accessible content, they also have a number of serious limitations which have been difficult to overcome mostly due to the inherent design.  The most significant of these are:   

                                             

1.     Hierarchical structure of pages -- unless you maintain multiple hierarchical relationships (or references) of your pages, you have to always remember where a specific page content (or topic) may exist.  Not only that but the consumers of this content must also always be aware of the structure or taxonomy.  In the absence of this, endless navigation through the trees or wildcard searches are the only other way to find what you (or they) seek.  In the corporate environment, people seldom have the time or opportunity to construct such taxonomies unless it is a dedicated activity or the wiki package comes with some pre-defined and ready-to-use templates. 

2.     Content updates -- Once a wiki grows to hundreds or thousands of pages, it is much more difficult to find content and keep it up-to-date since it is a manual process.  However, if you can't keep it current then the repository loses its credibility and the wikis quickly become shelf-ware.

3.     Contextual relevance -- While people create content on the wikis at will, its organization and hierarchy are often pretty arbitrary.  Furthermore, there may or may not be a context associated with the content, or perhaps if there is one, it is implied by the author.  So you end up with lots of content without any seemingly contextual relevance.  Once again, something that becomes difficult to maintain and leverage over time.

      

With all the right spirit of generating and organizing a lot of content with ease, I have seen wikis become cumbersome, unwieldy and useless fairly quickly.   

   

With this in mind, I envisioned (and began researching two years ago) the idea behind wikis that could support semantic annotations, thereby adding some needed structure to a naturally unstructured content store.  Semantic annotations, I believed, would allow content to become much more self-organized, accessible using contextual searches and queries and updatable using software agents (where relevant) -- all adding up to some powerful capabilities.   

                

I found a number of projects exploring this concept during this time.  I think the closest one to the vision that I had and the one that we at zAgile have chosen to leverage in our platform -- is the result of efforts by Zdenko (Denny) Vrandečić  & the team at Institute AIFB, Universität Karlsruhe, in the form of an extension to MediaWiki called Semantic MediaWiki (or SMW).  We believe it to be a very efficient implementation of semantic technology on top of a wiki engine.

 

Supporting a very popular wiki (the engine behind Wikipedia), this extension allows the user to create annotations to any page in a free-form manner.   It’s the easiest and most user-friendly approach to creating some structure to a vastly unstructured pool of content.  Once annotated, pages can now be organized in any way using templates and queries and the content of any or all of these pages can easily be reflected elsewhere through simple semantic queries, rearranged, summarized, tabulated, accessed by external applications, etc. etc.  The idea is not much different from the queries and result sets that we are used to getting from relational tables.  However, the depth and expressivity of semantic annotations goes beyond what we are normally able to express in relational models (of course!! another topic) and the interfacing with external applications (both in and out; using RDF exports) is much easier.    

                     

This capability, in my opinion, makes the wiki a very dynamic and valuable application.   

             

For our purpose, we have taken this base foundation of MediaWiki + SMW and extended it in the zAgile Platform to provide some interesting and useful functionality for the Software Engineering domain, although the functionality would be equally relevant anywhere else (our focus is only SE, for now :)).  We have integrated applications with it to both create content into the wiki as well as access content from it.   

       

We have enabled the wiki to become a valuable 'Knowledge Repository' rather than a 'Content Repository'  .

    

·         First, we have extended (or constrained, depending upon your point-of-view) the general free-form annotation capability of SMW with well-defined (and expressive) set of ontologies for the Software Engineering domain.  Thus, the wiki now has the 'semantic awareness' of typical SE concepts, such as Artifact, Project, Person, Role, Team, Process, Work Product, Task, etc.  This set will continue to extend with richer and broader set of ontologies as we move forward.  Rather than allowing the users to arbitrarily create annotations, we constrain the wiki's semantic 'openess' by importing (or referencing) very specific set of ontologies.  This is mostly to ensure semantic integrity of the knowledge domain, as well as interoperability between tools (the core value of the zAgile platform).  If you continue to create arbitrary 'semantics' in your content, it is unlikely that anyone else will understand the meaning.  So constraining the annotations has been a critical part of our efforts.

·         Second, we have developed tools that generate 'annotated' content into the wiki.  Using zAgile's Method Composer, for example, you can define (or derive from) a methodology that may be relevant for your organization or project -- and publish it on the wiki with appropriate annotations that describe Lifecycles, Milestones, Tasks, Documents, Roles, etc. Using our Doc2Wiki converter, you can take MS Office, OpenOffice or PDF files and load them into the wiki in a WYSIWYG form but with annotations that describe basic properties such as Creation/Modification Dates, Author, Subject, Title, etc. but also allow you to extend the semantics of the documents based on the type being uploaded (an Invoice, a Status Report, a Project Plan, etc.).   

    With these applications and widgets generating 'annotated' content on the wiki, you can now begin to organize this content using simple queries.  For example, a set of pre-defined templates (with embedded queries) can now automatically generate My ProjectX page on the wiki.  A page that includes sections containing My ProjectX Milestones, Processes, Phases, Teams, Roles, Status Reports, Invoices, Artifacts (and types), etc.   None of this needing to either be organized or maintained manually.  Based on the templates and queries, any combination or composition of pages is possible, thus allowing you to create virtual binders of projects, products, people, processes -- at will -- and share it across the organization.  This has been done before by many others using wikis, but only manually, and most of the time, such content has been incomplete and out-of-date.  Always a good start but not sustainable enough through the project lifecycle, therefore, generating lots of useless content.   

• And finally, we have extended the typical search capability across this content using our Semantic Search engine.  This is a slight paradigm shift to the way we normally use search.  Rather than searching on keywords which can return any type of content match, now you can search for a specific concept.  In this model, for example, one is either looking for a Person or Document or Project or Product-related information.  Thus the search-string provided will return results that match the concept being searched on.  Performing a search  on a document with the string "Lucas" returns documents authored by Lucas.  Similarly, a search on a project with the string "Lucas" will return Projects in which Lucas may have played some role.  Rather than returning free-form result set, the search not only returns the concept being searched but also relevant properties and relations to other concepts (Document X -- authored by Lucas; created on Jan 19, 2008; copyrighted by zAgile Inc.; associated with Project Y; created in Inception Phase, etc. …) .

 

And of course, this search engine exists outside of the wiki (as a portlet) -- another example of the ease of interaction of semantic data with external apps.

 

-Sanjiva

Prague, Jan ‘08