In one of our previous articles (specifically my last article “Public Cloud Vs Private Cloud“) we discussed the drawbacks of developing a private cloud against a public one. Today we will address one of the issues that come as direct consequences of this choice: LiveBox updates and releases, customer installations and fix bugs.
When you have a public installation you own the servers and the source code is centralized, so any update can be released and it will have instant effect on the final users.
On the other hand with the private cloud approach you need to deliver software updates to installations virtually spread all over the world, this creates a new set of possible problems. Each customer has his own needs and environments resulting in a multitude of update scenarios.
Let’s start by imagining the evolution of an update system from scratch and take steps until we reach the current stage of the LiveBox update scheme.
UpdateDummy 1.0 is not really an automatic update system. It boils down to physically going to the customer deployment site with a bunch of files containing the differences between his current installation and the new update one, replacing outdated files. This can be extended to database update scripts and some kind of code versioning system could help in downloading the updated release on a customer’s servers. But this system makes it very hard to modify current data if needed, such as changing specific values for specific records and stuff like this.
UpdateDummy 2.0 evolves bringing one shot scripts to be executed on the updating server into the picture, making sure that only a minimum of tasks is still manual. At this point we still have the problem of how to perform technical updates, such as installing new Unix modules, reconfiguring networks or LDAP systems, and so on. But there really is no easy way to solve this, so we will not take it into consideration here and now.
Physically reaching every single customer every time an update is needed is impractical, time-wasting and generally inefficient. UpdateDummy 3.0 frees us from this burden. We simply compile a list of files and instructions, some automatic and some manual, to be performed on the installed server to update it. Of course this can have drawbacks, because while it makes the update process faster and easier, there can be customers unable to execute systemistic operations due to lack of dedicated staff. In practice LiveBox still needs to make occasional hard updates on the servers when new modules or new configurations are necessary.
Now, why send the update scripts when you can just make them available and notify the customer that they are present? UpdateDummy 4.0 does exactly that. The updates are released on a server and a mail is sent notifying the update’s availability. Customers download the update and execute it just as they did in the previous step.
Anyway we are still delegating the update process to human operations, that can be error and delay prone. This is why UpdateDummy 5.0 suddenly shows up. This is a big step: server installations have a callable script that calls the centralized hub server and downloads the updates, executing what it receives. We can set the right repository versioning to synchronize, run database queries and also optional custom scripts to adapt and migrate existing data or create new ones. The update script must perform lots of checks and authenticate against the central hub to confirm the customer identity. Moreover the update script is part of the software repository, so it gets updated too, allowing improvements on the update sequence. What can go wrong here? Ever thought about what can happen if your updated update script introduces a bug? You may never update again. And you may never fix it.
This is were UpdateDummy 6.0 kicks in, bringing a smarter system to the update procedure. The server script is reduced to a simple downloader, which calls the hub to download the latest updater. In this way the sensible updater is so simple that it will never be changed anymore, so that it cannot be broken, while the current updater is centralized and can be fixed any time as the need arises.
The last step toward a LiveBox like updater is UpdateDummy 7.0, which brings forth a new concept: why manually launch the update script when a crontab can do it for us everyday? The update script is launched daily and logs the result to let us know whether the update was performed or if it was already up to date. This also ensures that the customer server is constantly in contact with the LiveBox hub and LiveBox staff can check whether the updates are failing for some obscure reasons, or if the remote server cannot contact the LiveBox hub correctly.
The situation just shown is, more or less, similar to how the current LiveBox updater is. And with very few exceptional cases requiring manual updates (for security reasons etc) this method allows LiveBox to distribute updates without problems.
There is (as always) space for improvement and the LiveBox development team is always looking for ways to improve any script that can be further improved.
(my daily quote: “I don’t think you’ll ever have a perfect world because we humans are prone to error, and so we’re always in search of an upgrade.” – Henry Rollins)