How to handle external dependencies (Suggestion)

METAMOD uses a large number of external libraries and this number has increased greatly during the conversion to Catalyst. This document describes the how dependencies will be handled and why we have chosen to handlen them in this way.

Underlying this discussion is a guiding principle that external dependecies is the reponsible of each project. This should increase developer productivity as external libraries can be used when they are available and new features and bug fixes will be available sooner. The downside is that dependency handling takes time. With the solution proposed below we expect that the time required for dependency handling will be minimal.

1. Access to any relevant module on CPAN. We don't want to be restricted to only the modules or module version available in operating system package repositories.
2. Installation of specific module versions and not just any version.
3. Installation on different platforms. For instance installation on both 32 and 64 bit Ubuntu Lucid.
4. Automatic installation of necessary packages and all dependencies.
5. Automatic installation also outside of met.no environment. This is necessary for possible adoption by other organisations.
6. Installed modules should never conflict with other systems installed on the same server.
7. Installation and upgrading should be simple when inside the met.no system.

In general handling dependencies is hard! There is no perfect solution to all the problems that can arise when it comes to dependencies as there are often conflicting requirements and the underlying infrastructure is not good enough (e.g. different Perl module installers cause problems).

Some of the problems we come across when using CPAN modules:

1. If we should only base the application on packages available in the Ubuntu package repositories we would be quite restricted as:
   1. We are currently only using the LTS version of Ubuntu, meaning that we will slowly be out-dated when it comes to new version.
   2. Not all of CPAN is available in the Ubuntu package repository meaning that we would have to build packages our selves any way. This would mean that we do not save that much time on using Ubuntu packages.
2. CPAN clients will always install the newest version it can find of a specific module. In this scenario this is not what you want as you want to test METAMOD for specific set of module with a specific version number.
3. Old version of a module or an entire module can be deleted from CPAN.
4. Installing modules to the standard Perl module directories can wreck havoc other Perl programs on the same machine.
5. Installing large dependency chains on a machine takes a time as many modules have large tests suites that should run.
6. Installing large dependency chains increase the likelihood that at least one module will not install correctly. Solving these problems can be trivial, but require human intervention. We want the installation to be automatic.

To reach our goals and solve the problems we will do the following:

1. We create a script that based on list of modules will calculate the highest (we use the highest to ensure that a module is still available on CPAN) required version of all dependcies. This will be a tree of depencies. We can use CPAN::FindDependencies to simplify this script greatly.
2. The tree of dependecies will be then written to file in JSON format.
3. We create another script that will download all the .tar.gz files for the calculated depencies of the correct version. These .tar.gz files are stored in a centralised source code repository and shared for the entire met.no. Only .tar.gz files for version we don't have will be downloaded.
4. Based on the dependency tree create a Perl script that will install all the CPAN modules from the .tar.gz archives (cpanm can be used for this). The modules will then be installed to a specified directory to prevent conflicts with other installed versions. This script is largly meant for development machines and environments that are not Ubuntu/Debian. Or on machines that are not used exclusively for METAMOD.
5. Setup a build machine for each environment that is supported.
6. Create .deb packages for each supported environment from the downloaded .tar.gz files. This can be done using cpan2dist.pl included as part of CPANPLUS.

An alternative solution is to only use the Ubuntu package repositories and then persuade the Server section to make available Ubuntu server images for each Ubuntu release. We then restrict our selves to only the CPAN packages available in Ubuntu. If we need other package we make .deb packages for those packages and put them on the internal Ubuntu package repository. We also push them to Debian so that they will automatically included in the later distribution. This simplifies the dependency management somewhat, but does not make it easy to install METAMOD outside an Ubuntu/Debian environment and can give conflict when it comes to requirement on packages for different projects.