Hi All,

Wow! So, I’ve decided to make this pet project of mine open source … I’m still figuring out how to do that properly, but in the process, I’ve found myself benefiting from other open source projects as I try to publish it, which is inspiring me to keep going with the idea of opening this project up to the world.

What’s it all about, well, put simply, the primary goal is to implement a distributed source control system for JADE using the open source libgit2 library.

But this isn’t the only goal, with the system architecture I’ve got in mind, we’ll be able to implement a suite of tools that will radically change how we can develop/test changes for JADE based solutions before merging, with common bottlenecks or time-consuming tasks being removed (depending on current approaches used). Ultimately, it’ll provide the JADE community a way of publishing/sharing code, which would facilitate starting other open source projects for JADE based solutions.

Future posts will go into more detail about these wider goals, but for now, let’s discuss the fundamentals of why and how.

Why should we do this?

Because we have the skills, are the main stakeholders and understand our needs best. Like I’m sure others have, I’ve spent quite a bit of time preparing some very complicated deployments that reorganize/transform our systems without losing any data, so we’re quite familiar with the types of scenarios that need be handled, where we’d normally treat JADE with some TLC preparing command files and separating changes into different schema files, all of which would ideally be handled more elegantly and efficiently with a smarter source control system.

There’s been some discussion with JADE over the past year or so about the need for source control improvements. Implementing a new file format has been proposed (confirmed by the roadmap recently published), which will make it easier to plug-in different source control systems (like Git or TFS).

In the meantime, we need to be thinking about how we’re going to plug-in source control systems in a way that’ll complement our day to day development in the best way possible. In this case, I’m looking to implement a Git based solution, whereas others may well prefer TFS or another source control system, but aspects of what’s implemented as part of this project may help.

While we’d initially be working with own our XML extract format and conversion for loading, this project will help to illustrate how we can/want to implement source control for JADE, and help to highlight where we need JADE to make platform changes to solve problems that can’t or are difficult to work-around (or perhaps they’d prefer we don’t work-around).

Ultimately, if we implement and maintain our own source control tools, we can all contribute to building solution(s) that work for everyone, and can benefit immediately as we’ll be able to release updates as frequently as we want, independently of the JADE platform release cycle, allowing us to adapt more quickly as our needs evolve (along with our suite of tools getting more sophisticated!).

What’s wrong with the current approach?

In my experience, we generally have a centralised approach to source control where we all develop in the same master environment. We’ve heard about some who work in isolated environments, but have a very manual process for copying changes back into the master copy, which is subject to human error. Some have a system which locks anything touched during development, which is held until changes are signed off. In an environment where this may take months, this can then block other development, which leads to situations where changes must be reversed manually to give way to other pieces of work when priorities change. As a result, as the volume of changes we’re working on simultaneously increases, developer productivity decreases because we end up needing to spend a lot of time managing code conflicts.

Overall, there’s various frustrations here that’ve been making the JADE development role increasingly difficult. Hence, the motivation behind starting this project a year or so ago in my spare time, as I believe our lives will be a lot better with the right tooling to support development and testing of JADE applications.

Why use the libgit2 library and how’s it going to work?

JADE source code isn’t stored in files, rather the database. This is a fundamental challenge for any solution as we need to normalize the source code, so it’s stored in a file format that’s source control friendly, which can be automatically merged and can be easily understood when we’re reviewing changes.

Once we’ve extracted the source into a friendly format, we also need to be able to do the reverse to build a set of schema files to load changes back into a JADE environment, along with commands for special functions (delete/rename/move) we need to perform the schema load without error or data loss.

The libgit2 library provides an ODB backend interface, using which we can make JADE the database for it to store all the changes being staged/committed to Git.

The basis of this solution is to combine the use of existing patch control hooks provided by JADE, with the provision of the ODB backend for libgit2. Using the hooks, we can identify when changes are being made and extract/stage them to libgit2 as changes occur, with all changes being committed in the same overall database transaction, so everything is kept in sync atomically. Any other approach which attempts to identify/extract changes after the fact is at risk of missing critical changes (like renames), which need to be handled in the moment, so we can understand what is happening in an accurate manner.

Once changes have been extracted/committed to Git, we use libgit2 to compare one version/branch to another, which provides the basis for building deployments that are tailored to transition from one specific version to another. Ultimately, we’ll be able to implement a build service that’s just backed by a Git server, to construct updates for test & production environments, without needing to use a central/master JADE environment.

What’s next?

I’m aiming to publish the project (once stable)! With or followed by a demo video to show what’s been built so far.

Thanks for reading :)

Kevin