Every team managing more than a few Debian packages at some point faces issues with serving, or, if not that, how effective it’s being served. In our case, we used to use a different tool, Reprepro, which does a pretty good job on the package management, however it does not support having more than one version per package. This kind of constraint directly affects when we need to meet dependencies or even rollback to previous versions and was always problematic in achieving the required results when facing this kind of scenario.
Having this requirement in mind, we started researching the possible solutions that would bring us more flexibility with package versions while still maintaining a fairly simple and reliable interface. When we first started testing Aptly, it was clear that this tool would meet our requirements, which in addition to multi version packages, were:
- Highly available repositories*
- Easy to add new packages
- Easy-to-update, mirrored (external) repositories
* It can be achieved either by publishing in AWS S3 (which is configurable directly in aptly.conf ) or using a distributed file system (e.g, Glusterfs).
The next step was of course testing as much as possible to cover the use cases we had and how to deal with unexpected situations. This step taught us a handful of valuable lessons that will be described in a moment, but just to have an idea, one important thing is that when you add a high amount of packages at the same time to a repository, make sure you have a high enough maximum of open files configured so that it will not crash, complaining that you have reached the maximum number of open files.
Scenario goals to obtain:
- Two repository servers serving in round-robin
- In case one of them fails, the second will take over the entire traffic
- Distributed and scalable file system to hold the packages data
- Add new packages automatically to local repositories
- Clean up packages considered too old
- Keep mirrors up-to-date
- Satellites in different locations
Setup overview
- The servers setup is quite simple as it includes two main servers, which have Aptly installed; the data file system is mounted from a distributed file system (glusterfs), having the same data available to both servers. On the DNS side, we have configured both servers to respond as apt server in a round robin-way, which enabled us to not only distribute the load, but to also be sure that both are working fine, and in the event of a server going down, the remaining one will handle the traffic.
- For adding new packages in addition to the repositories, we have a script which handles that by identifying new packages and then properly processing them until they are in the repository and are published. This step is transparent to whoever creates a new package version; it’s just a matter of a few seconds until the package is available in the repository.
- As mentioned before, Aptly has a great feature that allows us to keep as many versions of packages as we need/want. However, as of now, it does not have a feature for retention policy, so we have a small script that makes sure that only a certain number of versions are kept.
- Last but not least, we have to handle external mirrored repositories, which must be up-to-date, otherwise they are not of much use, and that we also have a script that will check for updates, update the mirror, take a snapshot, and finally publish the new snapshot.
For those not yet familiar with the terms snapshot and publish, they will shortly be explained in the next sections.
Installation
The installation is quite simple and is very well-explained in the Aptly documentation. It consists of three steps:
- Add repository to your lists file ( e.g. /etc./apt/sources.list )
deb http://repo.aptly.info/ squeeze main - Add key
sudo apt-key adv –keyserver gnupg.net –recv-keys 9E3E53F19C7DE460 - Install aptly
sudo apt-get update
sudo get install aptly
For more information:
https://www.aptly.info/download/
Configuration
Configuration is quite simple and is restricted to a single file. This file is used, for example, when creating new mirrors or snapshots. It contains root directory, where Aptly will create its structure, default architectures, the number of concurrent downloads and so on.
By default, Aptly will look for the configuration in ~/.aptly.conf to then look for /etc/aptly.conf, in case none is found a new configuration file that is created in the current home directory.
Although we have a configuration file, most of the configuration is done by using Aptly options when running the commands. This proves to be helpful when we do not want to have a mirror for the default architectures, or even more importantly, when we need to define the default distribution (this is used when publishing).
Below, we can see an example of configuration.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
|
{ “rootDir”: “/directory/to/store/aptly/data”,
“downloadConcurrency”: 6,
“downloadSpeedLimit”: 0,
“architectures”: [ “i386”, “amd64”, “source” ],
“dependencyFollowSuggests”: false,
“dependencyFollowRecommends”: false,
“dependencyFollowAllVariants”: false,
“dependencyFollowSource”: false, “gpgDisableSign”: false,
“gpgDisableVerify”: false,
“downloadSourcePackages”: false,
“ppaDistributorID”: “ubuntu”,
“ppaCodename”: “”,
“S3PublishEndpoints”: {},
“SwiftPublishEndpoints”: {}}
|
Mirroring
Mirrors are an important part of our repository management. Having external repositories mirrored is very useful as we decrease the bandwidth usage and download packages much faster.
Repositories such as Puppet and Docker are mirrored and are periodically checked for new updates.
In order to have a repository mirrored, we simply create a new mirror by giving it a name and the repository address. Once it’s created, we run the update to download its content. This step may take several minutes or even hours depending on the repository size.
Once the mirror is updated, we can create a snapshot, and this snapshot is then published where packages are then made available to be used by apt tools.
Snapshots
The ability to create snapshots is a really great feature, and by using that, we can keep track of changes and easily rollback to a previous state of the repository. For example, repository A has 3 packages, and you just added three new packages, which are the newest version for the current ones, which then created a snapshot and published it. However, those new packages are broken for some reason, and you want to immediately remove them from your repository. In this case, you can just switch back to the previous snapshot that only contains the three original packages without their new versions.
Repositories
Repository features enable us to have our customized repositories, so this way we can only have the packages that really matter to a given scope and nothing else. This also allows us to better organize packages by distribution, among other things.
Publishing
Publishing is the last step in creating a new repository or mirroring one. Once you have all the data (packages) and a snapshot of the repository, then you can expose it, meaning that it will be available once you start serving it.
Here is where the mirrors and repositories “go live” and are ready to be consumed by apt. We publish the snapshot of either a mirror or repository. Repositories could be published directly, but it’s advisable to always create a snapshot after changes, as mentioned in the Snapshots section, it makes a rollback easier.
The initial operation is to either publish a snapshot or repository, and later on, we can switch published snapshots and/or update local repositories. While publishing, we have the option to choose a new prefix (other than the default “.”), which leads Aptly to create a new directory tree, where the packages for the given repository will be placed.
Aptly has its own http server, which is useful for testing; for production, we should serve it with an HTTP server such as nginx or Apache HTTP by using https and authentication. In the example that follows, we use yet another option provided by Aptly publish, where the “skip-contents” is “true”. This option prevents Aptly from creating contents index (a list containing each packages contents), which takes a long time.
Having the contents index being created could lead to several lock issues as new packages arrive to the server and cannot be added to the repository due to the Aptly DB being locked.
Step-by-step to create mirrors, repositories and later publish them
Mirror creation and update:
1
2
3
4
5
|
aptly@server:~$ aptly mirror create aptly_mirror
http://repo.aptly.info/ squeeze main
Downloading http://repo.aptly.info/dists/squeeze/InRelease…
gpgv: Signature made Mon 27 Mar 2017 11:47:03 PM CEST using RSA key ID 9C7DE460
gpgv: Good signature from “Andrey Smirnov <me@smira.ru>”
|
Mirror [aptly_mirror]: http://repo.aptly.info/ squeeze successfully added.You can run ‘aptly mirror update aptly_mirror’ to download repository contents.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
|
aptly@server:~$ aptly mirror update aptly_mirror
Downloading http://repo.aptly.info/dists/squeeze/InRelease…
gpgv: Signature made Mon 27 Mar 2017 11:47:03 PM CEST using RSA key ID 9C7DE460
gpgv: Good signature from “Andrey Smirnov <me@smira.ru>”
Downloading & parsing package files...
Downloading http://repo.aptly.info/dists/squeeze/main/binary-amd64/Packages.bz2…
Downloading http://repo.aptly.info/dists/squeeze/main/binary-i386/Packages.bz2…
Building download queue...Download queue: 26 items (98.09 MiB)
Downloading
http://repo.aptly.info/pool/main/a/aptly/aptly_0.7_i386.deb…
Downloading
http://repo.aptly.info/pool/main/a/aptly/aptly_0.9.6_amd64.deb…
...
...
Mirror `aptly_mirror` has been successfully updated.
|
As shown, this is the mirror list and mirror show. We can even list all packages from the mirror by giving the -with-packages option to the Aptly mirror show command.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
|
aptly@server:~$ aptly mirror list
List of mirrors:
* [aptly_mirror]: http://repo.aptly.info/ squeeze
To get more information about mirror, run `aptly mirror show <name>`.
aptly@server:~$ aptly mirror show aptly_mirror
Name: aptly_mirror
Archive Root URL: http://repo.aptly.info/
Distribution: squeeze
Components: main
Architectures: amd64, i386
Download Sources: no
Download .udebs: no
Last update: 2017–04–06 16:39:26 CEST
Number of packages: 26
Information from release file:
Architectures: amd64 i386
Codename: squeeze
Components: main
Date: Mon, 27 Mar 2017 21:46:57 UTC
Description: Generated by aptly
Label: . squeeze
Origin: . squeeze
Suite: squeeze
|
Creating a mirror snapshot:
1
2
|
aptly@server:~$ aptly snapshot create aptly_snapshot from mirror aptly_mirror
Snapshot aptly_snapshot successfully created.You can run ‘aptly publish snapshot aptly_snapshot’ to publish snapshot as Debian repository.
|
Now that we have our mirror, we can create a repository and only add the packages we need:
To create a repository, you need to follow a similar syntax of the mirror creation.
1
2
|
aptly@server:~$ aptly repo create –architectures=“i386,amd64” –component=main –distribution=repo_dist new_repo
Local repo [new_repo] successfully added.You can run ‘aptly repo add new_repo …’ to add packages to repository.
|
Adding packages to the newly created repository:
1
2
3
4
5
6
|
aptly@server:~$ aptly repo add new_repo aptly_1.0.0_amd64.deb
Loading packages...
[+] aptly_1.0.0_amd64 added
aptly@server:~$ aptly repo add new_repo aptly_0.9.7_amd64.deb
Loading packages...
[+] aptly_0.9.7_amd64 added
|
Listing repository packages:
1
2
3
4
5
6
7
8
|
aptly@server:~$ aptly repo show –with–packages new_repo Name: new_repo
Comment:
Default Distribution: repo_dist
Default Component: main
Number of packages: 2
Packages:
aptly_0.9.7_amd64
aptly_1.0.0_amd64
|
Creating a repository snapshot: it’s worth noticing that for repositories, we are able to publish it directly without snapshot, however, it’s much more convenient to have snapshots, so that this way you can move from one snapshot to another in case of packages issues. The Aptly documentation also advises to avoid publishing repositories directly unless it’s for testing.*
1
2
|
aptly@server:~$ aptly snapshot create new_repo_snapshot from repo new_repo
Snapshot new_repo_snapshot successfully created.You can run ‘aptly publish snapshot new_repo_snapshot’ to publish snapshot as Debian repository.
|
Just to illustrate, let’s remove one of the packages- it’s very simple:
1
2
|
aptly@server:~$ aptly repo remove new_repo aptly_0.9.7_amd64Loading packages...
[–] aptly_0.9.7_amd64 removed
|
Now, let’s check the already-created snapshot and the repository:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
|
aptly@server:~$ aptly snapshot show –with–packages new_repo_snapshot
Name: new_repo_snapshot
Created At: 2017–04–06 16:48:35 CEST
Description: Snapshot from local repo [new_repo]
Number of packages: 2
Sources:
new_repo [local]
Packages:
aptly_0.9.7_amd64
aptly_1.0.0_amd64
aptly@server:~$ aptly repo show –with–packages new_repo Name: new_repo
Comment:
Default Distribution: repo_dist
Default Component: main
Number of packages: 1
Packages:
aptly_1.0.0_amd64
|
Here, we can see that what is inside the snapshot is untouched, and on the other hand, the repository no longer has the removed package.
Snapshots keep a state, and even if it’s published and a package is removed from a repository, it does not matter as the published content will remain the same unless we switch to a newly taken snapshot.
The last operation is publishing the repository snapshot; here, we need the GPG key to signing, and depending on the repository size, the –skip-contents option will be really helpful to avoid taking a long time during publishing or re-publishing.
1
2
3
4
5
|
aptly@server:~$ aptly publish snapshot –skip–contents=true –architectures=amd64,i386 new_repo_snapshot new_prefix
Loading packages...
Generating metadata files and linking package files...
Finalizing metadata files.........
You can also use `aptly serve` to publish your repositories over HTTP quickly.
|
We can also switch published snapshots. Below, we can see almost all the steps involved, starting by creating a new snapshot for the repository, then switching the older with the new repository:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
|
aptly@server:~$ aptly snapshot create new_repo_snapshot_2 from repo new_repo
Snapshot new_repo_snapshot_2 successfully created.
You can run ‘aptly publish snapshot new_repo_snapshot_2’ to publish snapshot as Debian repository.
aptly@server:~$ aptly snapshot show –with–packages new_repo_snapshot_2
Name: new_repo_snapshot_2
Created At: 2017–04–11 16:50:59 CEST
Description: Snapshot from local repo [new_repo]
Number of packages: 1Sources: new_repo [local]
Packages:
aptly_1.0.0_amd64
aptly@server:~$ aptly publish switch –skip–contents=true repo_dist new_prefix new_repo_snapshot_2
Loading packages...
Generating metadata files and linking package files...
Finalizing metadata files...
...
...
Publish for snapshot new_prefix/repo_dist [amd64, i386] publishes {main: [new_repo_snapshot_2]: Snapshot from local repo [new_repo]} has been successfully switched to new snapshot.
|
In summary, Aptly provides many resources that allow us to create a stable apt server. It allows us to create customized repositories as well as mirror all external mirrors that we need. Along with that, it is flexible enough to be placed behind a load balancer, that being a regular load balancer or DNS).
References:
https://www.aptly.info/doc/overview/
AUTHOR: RAFAEL COSTA