Dogen v1.0.13, "Náutico Club"
Esplanada do Náutico Club, Moçamedes, Namibe Province, Angola. (C) 2019 Nautico Club Site.
Introduction
The sprint cadence seems to finally be establishing itself, with sprint 13 offering yet another solid 2-week effort. The main emphasis was on solving the unit testing of generated code. If you recall, we had some sparse manual tests for these, delightfully called the "canned tests". These weren't exactly brilliant, but did provide some kind of coverage. Sadly, we ended up having to disable them due to weird and wonderful failures on OSX and Windows, which we could not reproduce on Linux and which were rather difficult to get to the bottom of via CI because of the way the tests were designed.
As the next few sprints are all about very (hard-)core changes, we had to make sure a strong testing base is in place before we can proceed with the refactoring. As usual, the work was much harder than expected, taking us the best part of two sprints to get to a good place: firsy, sprint 12 was all about the system test story, and now sprint 13 is all about the unit tests story. Fortunately, we still managed to sneak in one useful feature.
The below chart breaks down the cost of each story, percentage-wise, in terms of the total sprint time.
The next sections provide a summary of the most significant stories. As usual, for more details of the work carried out this sprint, see the sprint log.
Internal Changes
In this section we cover stories that consumed significant resources but are only visible internally.
Sprint and product backlog grooming
In this sprint we spent quite a lot of time grooming the backlogs. This is something which never gets much of a mention, but which I believe is one of the most important aspects of Agile: you need to keep your product backlog in good shape. Perhaps spending 15% of the total time of a sprint grooming backlogs may sound a tad excessive, but in our defence we do have a hefty product backlog, with over 550 user stories at various levels of detail. Also, given that we have just finished a massive rewrite of the theoretical basis for Dogen, it is no surprise that a lot of the stories started to bit-rot. This clean up was mainly to look for low hanging fruit and remove all stories which are completely deprecated; subsequent clean-ups will delve more into the detail of the stories.
User visible changes
This section covers stories that affect end users. The sprint demo provides a quick demonstration on the user visible changes, whereas the below sections provide more detail.
Code generation of tests for dogen models
This story had started on the previous sprint, but, as always, proved to be much more complicated than anticipated. Whilst the story is user facing - in that users can enable it for their own models - its purpose is very much just to test the code generator, so its not really that helpful to end users outside of Dogen.
In theory, adding tests seems just like adding another facet. Since we already have quite a number of these, we were pretty confident this would be a "quick effort". In practice, there were many subtle differences with the test facet which, cumulatively, caused very large problems and forced some difficult changes to the core of Dogen. On the plus side, the pain seems to be worth it, as we are now testing pretty much all facets for all generated code, across both Dogen itself and the Reference Implementation on all supported platforms. Even better, they are all green:
Whilst coverage is extensive, unfortunately we do not yet cover ODB (C++ ORM mapping) nor C# (which still relies on canned tests). In addition, build time has gone up quite considerably, given that we now need to compile the test data facet for all of these types, plus the tests too. The following chart demonstrates this problem:
As a result of this increase, MSVC is no longer able to complete the builds within the allotted time. Fortunately our clang-cl builds are deemed good enough (only one test failure across some 2.7k tests) so we'll be shipping that to users from now on. In the future we will need to look into ways of decreasing build time, as we are very close to the edge on OSX and clang-cl.
Delete empty directories
In the past we used to generate all facets for all models, Dogen and Reference Implementation. However, over time we ended up having to disable most facets as the build time was getting out of control. Dogen correctly deleted all of the generated files when the facets were disabled, but left behind a number of empty directories. Worse: because git does not care about empty directories, we weren't even aware of their existence until some speculative filesystem browsing revealed them. This sprint adds a new knob to delete any empty directory under the project: delete_empty_directories. Together with delete_extra_files, this should mean that most generated lint is taken care of now.
#DOGEN masd.extraction.delete_extra_files=true
#DOGEN masd.extraction.delete_empty_directories=true
Rename of extraction fields
One user facing change was actually a bug. Some generation fields had been placed incorrectly in extraction. This was spotted and fixed in this release. The change is not backwards compatible. As an example, a model with the following fields:
#DOGEN masd.extraction.cpp.enabled=true
#DOGEN masd.extraction.cpp.standard=c++-17
#DOGEN masd.extraction.cpp.msbuild.enabled=false
#DOGEN masd.extraction.cpp.visual_studio.project.enabled=false
#DOGEN masd.extraction.cpp.visual_studio.solution.enabled=false
#DOGEN masd.extraction.csharp.enabled=false
Now becomes:
#DOGEN masd.generation.cpp.enabled=true
#DOGEN masd.generation.cpp.standard=c++-17
#DOGEN masd.generation.cpp.msbuild.enabled=false
#DOGEN masd.generation.cpp.visual_studio.project.enabled=false
#DOGEN masd.generation.cpp.visual_studio.solution.enabled=false
#DOGEN masd.generation.csharp.enabled=false
This change affects all facets in C# and C++, so if you are configuring these directly you will need to manually update your models.
Next Sprint
Though we risk repeating Sprint 12's parting words, we are still of the of opinion that next sprint will finally be all about core metamodel changes. Now we have solid testing in place, our key objective for next sprint is to move all of the decoration related code into the meta-model. We started work on this in the previous sprint but sadly ran out of time. In addition, we hope to finally make some inroads against moving annotations to the metamodel. This will be a significant major feature, at long last.
Binaries
You can download binaries from Bintray for OSX, Linux and Windows (all 64-bit):
Note: Windows builds are now generated using clang-cl rather than MSVC.
For all other architectures and/or operative systems, you will need to build Dogen from source. Source downloads are available below.