Skip to end of metadata
Go to start of metadata

Draft

This page is a draft and may contain incomplete or inaccurate information

I spent part of the holiday clearing up a long standing architectural problem that developed in SSDT Common.  It started as an attempt to upgrade to Groovy 2.5.+ but lead to a split in Common and cluster of other upgrades.

Common has become a catch-all for any shared code and contains an odd collection of low-level utility and language functions and higher level infrastructure and business model/logic code.  For quite a while I've been thinking about splitting it into lower-level functions (along the lines of Apache Common Lang).  We spent a lot of time building and versioning project in Common which rarely change.  

One thorn in Common was ModelGroovyHacks which would be better implemented as a formal Groovy Extension Module rather than a meta-class hack.   But modules need to be built into a jar outside of the projects that use it.  Also, Groovy 2.5 contained some breaking changes which I was able to overcome by using a extension to restore the removed methods and behavioral changes.  Another advantage of a module is that IntelliJ can code-complete for modules but can't for methods from ModelGroovyHacks.

The first version of SSDT-Lang will only contain "ssdt.lang.groovyextension" and "ssdt.lang.ast" (replaces ssdt.common.ast) and  will hopefully be used to replace other low churn, but important, modules like ssdt.common.annotations.  Some parts of the ssdt.common.model (in particular) are low level utility classes that could be migrated to ssdt.lang.

Projects depending on ssdt.lang will use "latest.release" instead of "latest.integration" by default.  So we can also start to get used to treating these modules as true internal releases.

I've introduced a new HG repository (http://hg.ssdt-ohio.org/hg/ssdt/SSDT-Lang), JIRA Project and build. 

Groovy 2.5 update

This section contains a log of things I encountered or changed while upgrading groovy.  I tried to list every breaking change in Groovy's API and behavior. 

Significant (potentially breaking) changes:

  • Calendar.format() , clearTime() etc is no longer a thing.  (or I  couldn't  get compatibility groovy-datetime jar to work correctly).  I restored these methods by re-implementing them in the extension module from ssdt-lang.
  • Converted GroovyHacks to standard extension module in ssdt-lang. 
  • Groovy's Set.asImmutable() now returns a copy of the Set instead of a view of the Set.  With the previous behavior, changes to the underlying set would be reflected in the "immutable" reference.  This affected ModelCollections (et al) because changes by the parent would not be reflected in any immutable references.  The solution is to replace Set.asImmutable() with Java's Collections.unmodifiableSet() or unmodifiableSortedSet() which returns an unmodifiable view of the referenced set.  As a convenience, I added asReadOnlyView() to the extensions which does  Collections.unmodifableXXX() but that I think is more expressive than either the Groovy or Java alternatives.
  • List.pop() and push() were changed in 2.5 to remove the head of the list instead of the end.  replaced by List.removeLast() and add().
  • Changes in generic handling require some code changes, especially when @CompileStatic is used.
  • @AutoClone now validates the 'excludes' values and fails to compile if a property is missing. 
  • Had to upgrade to IntellIJ 2018.3.2 to get groovy/gradle project to work.  Could not find classes in same project without it.
  • Random new occurrences of lazy loading problems: In USAS-core had to switch to Eager loading in some places (like, BankTransaction on Disbursement and GLAccount on DistributionItem).  Or use PersistenceUtils.isInstanceOf() to replace "isAssignableFrom".  It's unclear what changed in Groovy that changed the nature of the lazing loading...  Some lazy loading problems resolved by using @CompileStatic.  Note:  PersistenceUtils.ensureLoaded() does not seem to work (in every case) with hibernate 5.3.  In this cases, I resorted to EAGER loading. 
  • I've added @CompileStatic to most classes that I've touched, especially if I'm having trouble with them.  But classes with QueryBuilder query closures couldn't be statically compiled.  So i wrote a simple type checking extension which allows such classes to be compiled static.  Just use:  
        @CompileStatic(extensions = "org.ssdt_ohio.util.query.QueryBuilderTypeChecking")
    at the class to enable QueryBuilder aware type checking.
  • spock (or groovy 2.5) doesn't like "final" for variables defined in a "when:" block.  Had to change "final" to "def".
  • In tests, maybe elsewhere, overriding instance methods using instance.metaClass does work as in the past.  e.g. this code no longer overrides the getPostingPeriods() method:
def fiscalPeriod = new FiscalPeriod()
fiscalPeriod.metaClass.getPostingPeriods = { -> [pp1,pp2,pp3] }

This is particularly ignoring since concrete classes can't be mock by Spock.  In most cases, I had to write subclasses of the class to use in the test, or use a Mock registered with ObjectFactory to supply whatever the underlying method (e.g. PostPeriodService in the above example). 

Other upgrades

The groovy rabbit hole turned in to whack-a-mole problem.  Upgrading to Hibernate 5.2.17 caused problems with some repositories.  For example, AppropriationRepository could no longer delete Appropriation because it reported a foreign key violation (was unable to delete Appropriate because AppropriationCode referenced it).  That is, it appeared that cascade delete stopped working.  But upgrading to 5.3.+ resolved this problem.

But then, the Hibernate 5.3 upgrade had a number of changes to the caching configuration, which started an upgrade of ehcache, which required a completely new configuration.  So I switched to using the new Java Cache API, which required a different configuration, which lead to needing to upgrade dropwizard metrics..  

yada yada.  I've lost track of exactly what I've updated.

During changing of Caching configuration, things to note:

  • Previously, creating a new cache with Spring's @Cachable annotation caused  the cache to be created automatically using the defaults.  This is no longer  true, you must define the cache explicitly in ehcache-spring-jsr107.xml.   This is partly to force you to think about what the settings should be for the cache.
  • Previously, Hibernate and Spring shared a singleton  ehcache CacheManager.  The configurations are now split so Spring caching and Hibernate have their own cache managers.  But this means that Spring can not access the hibernate caches.  Therefore, it's not possible to use Spring's @CacheEvict annotation to clear the hibernate cache (this occurred in USAS-core a few times, but was probably a bad idea).  As a work around, the Common QueryRepository interface now exposes evict() methods to clear parts or or all of an entity cache. 

Random Notes:

These are things you might already know, but I discovered/re-discovered them while in the above rabbit-hole:

I always had trouble getting IntelliJ to run individual tests that have a spring context with it's own test-runner.  I  changed IntelliJ to use the "Gradle Test Runner" (in Settings > "Build, Execution..." > Gradle > Runner" and it started working.   This is mostly, I think, because the gradle test tasks invokes the "processTestResources" correctly.

I added code to our gradle init40.gradle file to automatically configure a "copyright" configuration in everyone's intelliJ.  If you commit using IntelliJ's interface, please turn on "update copyright" so that it will add the copyright notice to the top of every file.

I re-factored DateConverter in common.model. This ancient thing performed really badly and created excessive objects.  Previously, it used non-thread safe parser's so it had to create  new ones for every parse.  The new version uses statically created thread-safe parser's (and regex patterns) and preforms about 50% better (in micro benchmark tests).  Since we are more focused on performance lately, this is good thing to watch for.  It's really easy to create a new parser, but if you'r doing in a context where the format is exactly the same and it may get all lots of times, then it creates performance defects.  For one thing, creating all those temporary objects puts a (pointless) burden on the garbage collector. 

Other Thoughts

This project was a real disaster.  Among other things, (which I've mentioned to you before), I'm questioning my commitment to Groovy.  The backwards compatibility problems caused by Groovy 2.5 seems troublesome.   I'm not ready to give up Groovy, but it deserves some discussion since Groovy is not as valuable given Java lamdas, et al. 

But in any case, I certainly over duck-typing and I'm putting @CompileStatic on most any class I touch.  I suggest you do the same.  There is some performance benefit to it, but it also (seems) to improve generic handling and lazy loading issues.   Relatively few of our classes really need dynamic typing.   Many of those can be modified slightly (usually with casting) to allow them to compile statically  Class that really need duck-typing, often only need it for a method or two.  So if you mark a class statically compiled but there are a few methods that really are duck-typed, then mark those methods as @CompileDynamc and move on.




  • No labels