In the previous post I discussed about our requirements and a few observations on existing options for the long term approach. In this I am going to describe what I am doing to get us a working solution for the next 6 months based on SVR4 packaging.
In the previous post I already mentioned about the ‘spkg’ tool that I am writing using Python. The tool is essentially a package manager client that can use packages from a networked repository or a local repository on disk. In conjuction to ‘spkg’ I am also writing another utility to manage the repository(publishing or retiring packages, tagging releases etc.).
The repository is structured after the Blastwave approach to have a directory called “trunk” where all current packages are pushed. Each package filename also contains the version and architecture embedded so that we can have multiple versions of the same package in trunk. There is also a master catalog file. This is a text file containing one entry for each package version in the repository. The entry contains the following fields separated by spaces:
<common name> <encoded version> <package name> <package filename> <SHA1 checksum> <unencoded version string>
Apart for the catalog and package files there is another directory under trunk called “metainfo”. This directory contains subdirectories per package and per package version holding additional SVR4 package metadata: pkginfo, pkgmap, depend, copyright. These make various “spkg” operations easy. All these metadata is downloaded by the client when it updates it’s catalog.
Now when a release is made a snapshot of the state of trunk at the time of release is taken. No this is not a ZFS snapshot but rather a set of symbolic links pointing into files in trunk. This means a release specific directory is created with the same structure as trunk but with it’s own separate catalog. All other metadata under metainfo and package files are just symlinks into the same files in trunk. Since all package publishing happens in trunk, it becomes easy to promote a package into a release by creating the necessary symlinks and undating the release’s own catalog. It also makes possible upgrades from one specific release to another rather than just upgrading to latest. In addition it is also possible to make special fix-backport package revisions for a release by publishing a package into a release rather than into the trunk.
This structure makes it easy for spkg to compute dependencies during package operations, identify latest versions of packages, perform catalog and content search, list available packages and compare versions etc. It is also possible to tag an installation to a particular release so that spkg only looks at the release specific repository rather than the current bleeding-edge trunk.
This framework and spkg implementation allows multiple package repository sites to be listed in order of preference. Each site can supply a mirror list for distributed downloading. This makes for a very flexible organization.
The Version numbers in SVR4 packages are typically of the form VERSION=<version num>,REV=<datetime stamp>. This value is encoded into a large multi-precision integer and stored as a string with dots after every 4 digits. This way of encoding the version string makes version comparision trivial using Python’s Decimal multi-precision arithmetic class. Of course there are variants of the SVR4 version string (including creative pathologies!) that are handled during encoding.
The spkg package client today is functional providing a working package repository operation. Of course there are still a bunch of things to implement, like the following:
- Generic package grouping support
- More advanced searching. Re-implement search using Nucular
- Catalog signing
- Handle upgrade of spkg itself as a special case
- Split up config file so that user configuration is not clobbered
- Introduce spkg config interface
- Use Axel instead of wget to do chunked load-balanced downloads across mirrors
- Use spkg for BeleniX Constructor
- Implement spkg uninstall including recursive uninstall
- Ability to specify a package version in spkg install
Some of these are trivial and some moderately complex. The generic package grouping I believe will make for a very user-friendly repository view. We can have hundreds of highly granular packages for flexibility. We can in addition have generic package groups around functional and use-case based scenarios to give user-friendly package views without the user having to individually look at sometimes cryptic package names.