Complexity shows up almost immediately–get the tools to manage it

Listening to the business requirements is really the only way to help the business with information management.

But listening to the business will take you places you would never had thought of yourself.

It always starts out small and neat; can you help us with our product development projects? We only need to keep track on what articles the project contains and their state of development. It will be fun and easy! I promise! We know almost exactly what we want and we are more or less done…

As a software developer you know the drill.

These last years I have said to myself; be a pro business developer – do what they ask – do not let practical technical issues stand in the way of progress.

Well. Where did that take me?

In this particular view – one of 500 in the system – the requirements has kept trickling in one by one over the years. Everything has gone smooth and no big issues. Since we do everything with UML and declarative ViewModels and as we model everything in the same lingo the business use – there is no issue to add a few new things now and then.

Building things this way really lets the complexity sneak up on you.

The view:

image

It has four main levels – the project, the articles within, the packaging components for the selected article, the logistic solution and barcodes for the selected article.

The main grid has columns to show some details for each article, same for the grid for the used components. Nothing fancy and all very relaxed and moderate.

The users at the innovation department love this view – they should – they designed it.

So why do I use this simple sample to talk about complexity? I will tell you why.

Using the “Analyze expressions” – button in the ViewModel dialog I get this auto generated diagram:

image

A quick calculation shows that I have accessed 6×5=30 classes in one view. 30 CLASSES IN ONE VIEW!

And look at all those relations!

And this is not even the most complex view in the system – is is actually pretty typical.

How did I end up here?!

My conclusion is that if you listen to the business (you know the guys that pay your bills and that you are on a mission to help ) then you end up here.

Of course you can listen to best practice and DO NOT DO COMPLEX THINGS – but then again you might as well stay home with your mother your whole life.

If my conclusion is right – how can we deal with this? It would be hard work to optimize a way to fetch objects for 30 classes in a good and speedy fashion. Even harder to update that strategy whenever something is added or removed. This would probably require one developer becoming an expert on just this single view… But hey – we have 500 views like this… and 2 developers… So 250 views like this each… Did I mention that we release to production every 4 weeks… And we have never missed a release for the last 4 years… And that we do about 50 different updates on each sprint…

We have(had) a problem

MDriven use the declarative ViewModel and the OCL expressions it consists of to look ahead to fetch data efficiently. But recently we found that it did not do it efficiently enough. It turned out that it got tricked by derivations in attributes and associations. Derivations was skipped and they resulted in lazy fetch as the UI was drawn and that resulted in a lot of single queries send to the database. This is the same basic problem that usually kills high abstraction framework ambitions.

In our case it got obvious we had a problem once we got users further away from the server. As long as the user had a latency of 10ms it was fine – but 100ms it sucked. Since each single query sent to the db got the latency penalty of 100ms – the number of queries hit the user experience straight in the face.

I started testing the app with a simulated latency of 400ms – that is what you can expect from Sweden to Australia – not at all our use case – but it made the problem very obvious to see.

We needed to rethink how the MDriven Server can help the application.

Many things were researched – but the solution we kept we call query plan.

We have a new thing to speed up load – Query Plan

MDriven use the declarative ViewModel to find out what primary keys and what foreign keys that are needed for a particular view. It resolved OCL based derived attributes and associations to find out persistent things that much be fetched to get the information on to the screen. It also resolved all the opt-in constraints in the classes shown and also the DefaultStringRepresentation of each class used, and also the ReadOnly, Visible and Style expressions.

From this MDriven retrieves all the persistent single, multi and inner links (links to association classes). It sends this information to MDriven Server along with the root of the part of the ViewModel being fetched. It treats the structure building expressions (the nesting creators) separately from the column expressions.

The MDriven server uses the information to request the set of objects needed – and sends back all that data in 1 go.

This approach reduces the load on the SQL-Server – since now we can fetch multiple foreign keys for multiple multi links in 1 query (not possible before). It also reduces the round trips needed to the server and by doing that effectively addresses the issue we had with latency. Tests confirm a 20 time improvement in time when having 400ms latency.

Phu! We HAD a problem. Now we have an engine that does it better than what you can expect from a developer on his or her best day. And it works every time – for every view – even if I changed it to use 5 more classes tomorrow.

The view above has this in its query  plan:

ProjectPackagingView
AllInstances on:
Members on this level per type:
  ArticleChange.Project                                                                  
  ArticleChange.AffectedArticleRevision                                                  
  ArticleChange.Article                                                                  
  ArticleRevision.Article                                                                
  ArticleRevision.ComponentUsageConstellations                                           
  ArticleRevision.ActiveLogisticSolution                                                 
  ArticleRevision.Changers                                                               
  LogisticSolutionRev.LogisticSolution                                                   
  Article.ArticleCollection                                                              
  Article.PartOf                                                                         
  ArticleCollection.ConsistsOf                                                           
  ArticleCollection.RepresentedAs                                                        

ArticleChange
AllInstances on:
  User
Members on this level per type:
  ArticleChange.Project                                                                  
  ArticleChange.AffectedArticleRevision                                                  
  ArticleChange.Article                                                                  
  ArticleChange.Responsible                                                              
  ArticleRevision.Article                                                                
  ArticleRevision.Filler                                                                 
  ArticleRevision.Bulks                                                                  
  ArticleRevision.ShelfLife                                                              
  ArticleRevision.LogisticsVote                                                          
  ArticleRevision.SafetysVote                                                            
  ArticleRevision.FormulationsVote                                                       
  ArticleRevision.PackagingsVote                                                         
  ArticleRevision.ComponentUsages                                                        
  SalesArticle.MarketingCompany                                                          
  Article.SalesArticles                                                                  
  Article.ArticleRevisions                                                               
  Article.RegulatoryDomain                                                               
  Article.ArticleCollection                                                              
  ArticleRevisionApprovalVote.ArticleRevisionLogisticsVote                               
  ArticleRevisionApprovalVote.ArticleRevision4                                           
  ArticleRevisionApprovalVote.ArticleRevision3                                           
  ArticleRevisionApprovalVote.ArticleRevision2                                           
  ComponentSpecificationRev.ComponentSpecification                                       
  ComponentUsage.UsedRevision                                                            
  ComponentUsage.ArticleRevision                                                         
  ComponentUsage.ComponentSpecification                                                  
  ComponentSpecification.PackagingSupplier                                               
  ProjectRole.Members                                                                    
Structure to reach this level from ProjectPackagingView
  Project.ArticleChanges                                                                 
  ArticleChange.AffectedArticleRevision                                                  
  ArticleRevision.Filler                                                                 
  ArticleRevision.ComponentUsages                                                        
  ArticleRevision.Article                                                                
  ArticleRevision.PackagingsVote                                                         
  ArticleRevision.LogisticsVote                                                          
  ArticleRevision.SafetysVote                                                            
  ArticleRevision.FormulationsVote                                                       
  ComponentUsage.ComponentSpecification                                                  
  ComponentSpecification.PackagingSupplier                                               
  User.FavoriteArticles                                                                  
  ArticleRevisionApprovalVote.ArticleRevisionLogisticsVote                               
  ArticleRevisionApprovalVote.ArticleRevision4                                           
  ArticleRevisionApprovalVote.ArticleRevision3                                           
  ArticleRevisionApprovalVote.ArticleRevision2                                           
  Article.ArticleRevisions                                                               

Components
AllInstances on:
  User
Members on this level per type:
  ComponentSpecificationRev.ComponentSpecification                                       
  ComponentUsage.UsedRevision                                                            
  ComponentUsage.ArticleRevision                                                         
  ComponentUsage.ComponentSpecification                                                  
  ComponentUsage.ComponentUsageConstellations                                            
  ArticleRevision.Article                                                                
  ComponentSpecification.PackagingSupplier                                               
  ComponentSpecification.Replaces                                                        
Structure to reach this level from ArticleChange
  ArticleChange.AffectedArticleRevision                                                  
  ArticleRevision.ComponentUsages                                                        
  ComponentUsage.ComponentSpecification                                                  

ContractManufacturer
AllInstances on:
Members on this level per type:
Structure to reach this level from ProjectPackagingView
  Project.ArticleChanges                                                                 
  ArticleChange.AffectedArticleRevision                                                  
  ArticleRevision.Filler                                                                 

PackagingSupplier
AllInstances on:
Members on this level per type:
Structure to reach this level from ProjectPackagingView
  Project.ArticleChanges                                                                 
  ArticleChange.AffectedArticleRevision                                                  
  ArticleRevision.ComponentSpecifications                                                
  ComponentSpecification.PackagingSupplier                                               

LogisticSolutionRev
AllInstances on:
Members on this level per type:
  LogisticSolutionRev.LogisticSolution                                                   
  LogisticSolutionRev.PalletLogSpec                                                      
  LogisticSolutionRev.TransportLogSpec                                                   
  LogisticSolutionRev.ExtraRetailLogSpec                                                 
  LogisticSolutionRev.RetailLogSpec                                                      
Structure to reach this level from ArticleChange
  ArticleChange.AffectedArticleRevision                                                  
  ArticleRevision.ActiveLogisticSolution                                                 

AffectedArticleRevision
AllInstances on:
  User
Members on this level per type:
  ArticleRevision.Article                                                                
  ArticleRevision.Bulks                                                                  
  ArticleRevision.ActiveLogisticSolution                                                 
  ArticleRevision.FormulationsVote                                                       
  ArticleRevision.PackagingsVote                                                         
  ArticleRevision.ArticleOwner                                                           
  ArticleRevision.LogisticsVote                                                          
  ArticleRevision.SafetysVote                                                            
  ArticleRevision.Filler                                                                 
  ArticleRevision.ShelfLife                                                              
  ArticleRevision.SalesArticles                                                          
  ArticleRevision.Changers                                                               
  ArticleRevision.DeclaredUnit                                                           
  LogisticSolutionRev.LogisticSolution                                                   
  LogisticSolutionRev.ExtraRetailLogSpec                                                 
  LogisticSolutionRev.TransportLogSpec                                                   
  LogisticSolutionRev.RetailLogSpec                                                      
  LogisticSolutionRev.PalletLogSpec                                                      
  Article.RegulatoryDomain                                                               
  Article.SalesArticles                                                                  
  SalesArticle.MarketingCompany                                                          
  MarketingCompany.Country                                                               
  ContractManufacturer.CMValuesPerRegulatoryDomains                                      
  CMValuesPerRegulatoryDomain.AppliesTo                                                  
  CMValuesPerRegulatoryDomain.OnlyInCountries                                            
  CMValuesPerRegulatoryDomain.ExtraRetailBarcodeFormat                                   
  CMValuesPerRegulatoryDomain.RetailBarcodeFormat                                        
  CMValuesPerRegulatoryDomain.TransportBarcodeFormat                                     
  CMValuesPerRegulatoryDomain.PalletBarcodeFormat                                        
  ShelfLife.ShelfLifeMarking                                                             
Structure to reach this level from ArticleChange
  ArticleChange.AffectedArticleRevision                                                  

ComponentDocRefs
AllInstances on:
Members on this level per type:
  ComponentSpecificationRev.ComponentSpecification                                       
  ComponentDocRef.ComponentSpecificationRev                                              
  DocumentReference.DocumentType                                                         
Structure to reach this level from Components
  ComponentUsage.UsedRevision                                                            
  ComponentSpecificationRev.ComponentDocRefs                                             

self
AllInstances on:
  User
Members on this level per type:
  ComponentSpecificationRev.ComponentSpecification                                       
  ComponentUsage.UsedRevision                                                            
  ComponentUsage.ArticleRevision                                                         
  ComponentUsage.ComponentSpecification                                                  
  ArticleRevision.Article                                                                
  ArticleRevision.ComponentUsageConstellations                                           
  ArticleChange.Project                                                                  
  ArticleChange.AffectedArticleRevision                                                  
Structure to reach this level from Components

ComponentUsageConstallations_PickListPresentation
AllInstances on:
Members on this level per type:
Structure to reach this level from self
  ArticleRevision.ComponentUsageConstellations                                           

Members_PickListPresentation
AllInstances on:
Members on this level per type:

Members
AllInstances on:
Members on this level per type:

ArticleChangeDetails
AllInstances on:
  User
Members on this level per type:
  ArticleChange.Project                                                                  
  ArticleChange.AffectedArticleRevision                                                  
  ArticleChange.Article                                                                  
  ArticleRevision.Article                                                                
  ArticleRevision.ActiveLogisticSolution                                                 
  ArticleRevision.Changers                                                               
  ArticleRevision.BatchCodePosition                                                      
  ArticleRevision.ExpiryDatePosition                                                     
  LogisticSolutionRev.LogisticSolution                                                   
Structure to reach this level from ArticleChange

DocumentTypesForFilter
AllInstances on:
Members on this level per type:

ActiveLogisticSolution_PickListPresentation
AllInstances on:
Members on this level per type:
Structure to reach this level from ProjectPackagingView
  ArticleChange.AffectedArticleRevision                                                  
  ArticleRevision.LogisticSolutionRevs                                                   

BatchCodePosition
AllInstances on:
  BatchCodePosition
Members on this level per type:
Structure to reach this level from ArticleChangeDetails

ExpiryDatePosition
AllInstances on:
  ExpiryDatePosition
Members on this level per type:
Structure to reach this level from ArticleChangeDetails

DeclaredUnitPickList
AllInstances on:
  DeclaredUnit
Members on this level per type:
Structure to reach this level from AffectedArticleRevision

This entry was posted in CO-Unique, Declarative, MDrivenDesigner, MDrivenServer, UML, ViewModel and tagged . Bookmark the permalink.

Leave a Reply

Your email address will not be published. Required fields are marked *

*