Every team managing more than a few Debian packages at some point faces issues with serving, or, if not that, how effective it’s being served. In our case, we used to use a different tool, Reprepro, which does a pretty good job on the package management, however it does not support having more than one version per package. This kind of constraint directly affects when we need to meet dependencies or even rollback to previous versions and was always problematic in achieving the required results when facing this kind of scenario.

Having this requirement in mind, we started researching the possible solutions that would bring us more flexibility with package versions while still maintaining a fairly simple and reliable interface. When we first started testing Aptly, it was clear that this tool would meet our requirements, which in addition to multi version packages, were:

  • Highly available repositories*
  • Easy to add new packages
  • Easy-to-update, mirrored (external) repositories

* It can be achieved either by publishing in AWS S3 (which is configurable directly in aptly.conf ) or using a distributed file system (e.g, Glusterfs).

The next step was of course testing as much as possible to cover the use cases we had and how to deal with unexpected situations. This step taught us a handful of valuable lessons that will be described in a moment, but just to have an idea, one important thing is that when you add a high amount of packages at the same time to a repository, make sure you have a high enough maximum of open files configured so that it will not crash, complaining that you have reached the maximum number of open files.

aptlz

Scenario goals to obtain:

  • Two repository servers serving in round-robin
  • In case one of them fails, the second will take over the entire traffic
  • Distributed and scalable file system to hold the packages data
  • Add new packages automatically to local repositories
  • Clean up packages considered too old
  • Keep mirrors up-to-date
  • Satellites in different locations

Setup overview

  • The servers setup is quite simple as it includes two main servers, which have Aptly installed; the data file system is mounted from a distributed file system (glusterfs), having the same data available to both servers. On the DNS side, we have configured both servers to respond as apt server in a round robin-way, which enabled us to not only distribute the load, but to also be sure that both are working fine, and in the event of a server going down, the remaining one will handle the traffic.
  • For adding new packages in addition to the repositories, we have a script which handles that by identifying new packages and then properly processing them until they are in the repository and are published. This step is transparent to whoever creates a new package version; it’s just a matter of a few seconds until the package is available in the repository.
  • As mentioned before, Aptly has a great feature that allows us to keep as many versions of packages as we need/want. However, as of now, it does not have a feature for retention policy, so we have a small script that makes sure that only a certain number of versions are kept.
  • Last but not least, we have to handle external mirrored repositories, which must be up-to-date, otherwise they are not of much use, and that we also have a script that will check for updates, update the mirror, take a snapshot, and finally publish the new snapshot.

For those not yet familiar with the terms snapshot and publish, they will shortly be explained in the next sections.

Installation

The installation is quite simple and is very well-explained in the Aptly documentation. It consists of three steps:

  1. Add repository to your lists file (  e.g. /etc./apt/sources.list )
    deb http://repo.aptly.info/ squeeze main
  2. Add key
    sudo apt-key adv –keyserver gnupg.net –recv-keys 9E3E53F19C7DE460
  3. Install aptly
    sudo apt-get update
    sudo get install aptly

For more information:

https://www.aptly.info/download/

Configuration

Configuration is quite simple and is restricted to a single file. This file is used, for example, when creating new mirrors or snapshots. It contains root directory, where Aptly will create its structure, default architectures, the number of concurrent downloads and so on.

By default, Aptly will look for the configuration in ~/.aptly.conf to then look for /etc/aptly.conf, in case none is found a new configuration file that is created in the current home directory.

Although we have a configuration file, most of the configuration is done by using Aptly options when running the commands. This proves to be helpful when we do not want to have a mirror for the default architectures, or even more importantly, when we need to define the default distribution (this is used when publishing).

Below, we can see an example of configuration.

Mirroring

Mirrors are an important part of our repository management. Having external repositories mirrored is very useful as we decrease the bandwidth usage and download packages much faster.

Repositories such as Puppet and Docker are mirrored and are periodically checked for new updates.

In order to have a repository mirrored, we simply create a new mirror by giving it a name and the repository address. Once it’s created, we run the update to download its content. This step may take several minutes or even hours depending on the repository size.

Once the mirror is updated, we can create a snapshot, and this snapshot is then published where packages are then made available to be used by apt tools.

Snapshots

The ability to create snapshots is a really great feature, and by using that, we can keep track of changes and easily rollback to a previous state of the repository. For example, repository A has 3 packages, and you just added three new packages, which are the newest version for the current ones, which then created a snapshot and published it. However, those new packages are broken for some reason, and you want to immediately remove them from your repository. In this case, you can just switch back to the previous snapshot that only contains the three original packages without their new versions.

Repositories

Repository features enable us to have our customized repositories, so this way we can only have the packages that really matter to a given scope and nothing else. This also allows us to better organize packages by distribution, among other things.

Publishing

Publishing is the last step in creating a new repository or mirroring one. Once you have all the data (packages) and a snapshot of the repository, then you can expose it, meaning that it will be available once you start serving it.

Here is where the mirrors and repositories “go live” and are ready to be consumed by apt. We publish the snapshot of either a mirror or repository. Repositories could be published directly, but it’s advisable to always create a snapshot after changes, as mentioned in the Snapshots section, it  makes a rollback easier.

The initial operation is to either publish a snapshot or repository, and later on, we can switch published snapshots and/or update local repositories. While publishing, we have the option to choose a new prefix (other than the default “.”), which leads Aptly to create a new directory tree, where the packages for the given repository will be placed.

Aptly has its own http server, which is useful for testing; for production, we should serve it with an HTTP server such as nginx or Apache HTTP by using https and authentication. In the example that follows, we use yet another option provided by Aptly publish, where the “skip-contents” is “true”. This option prevents Aptly from creating contents index (a list containing each packages contents), which takes a long time.

Having the contents index being created could lead to several lock issues as new packages arrive to the server and cannot be added to the repository due to the Aptly DB being locked.

Step-by-step to create mirrors, repositories and later publish them

Mirror creation and update:

Mirror [aptly_mirror]: http://repo.aptly.info/ squeeze successfully added.You can run ‘aptly mirror update aptly_mirror’ to download repository contents.

As shown, this is the mirror list and mirror show. We can even list all packages from the mirror by giving the -with-packages option to the Aptly mirror show command.

Creating a mirror snapshot:

Now that we have our mirror, we can create a repository and only add the packages we need:

To create a repository, you need to follow a similar syntax of the mirror creation.

Adding packages to the newly created repository:

Listing repository packages:

Creating a repository snapshot: it’s worth noticing that for repositories, we are able to publish it directly without snapshot, however, it’s much more convenient to have snapshots, so that this way you can move from one snapshot to another in case of packages issues. The Aptly documentation also advises to avoid publishing repositories directly unless it’s for testing.*

Just to illustrate, let’s remove one of the packages- it’s very simple:

Now, let’s check the already-created snapshot and the repository:

Here, we can see that what is inside the snapshot is untouched, and on the other hand, the repository no longer has the removed package.

Snapshots keep a state, and even if it’s published and a package is removed from a repository, it does not matter as the published content will remain the same unless we switch to a newly taken snapshot.

The last operation is publishing the repository snapshot; here, we need the GPG key to signing, and depending on the repository size, the –skip-contents option will be really helpful to avoid taking a long time during publishing or re-publishing.

We can also switch published snapshots. Below, we can see almost all the steps involved, starting by creating a new snapshot for the repository, then switching the older with the new repository:

In summary, Aptly provides many resources that allow us to create a stable apt server. It allows us to create customized repositories as well as mirror all external mirrors that we need. Along with that, it is flexible enough to be placed behind a load balancer, that being a regular load balancer or DNS).

References:

https://www.aptly.info/doc/overview/

AUTHOR: RAFAEL COSTA