user

ProGet Debian & Linux Feeds: A Feature Refresh

Introduction

Alex Papadimoulis

Alex Papadimoulis


LATEST POSTS

CRAN (R) Feeds Come to ProGet! 27th November, 2023

Self-hosted and User-friendly CI/CD with BuildMaster 08th March, 2023

Inedo

ProGet Debian & Linux Feeds: A Feature Refresh

Posted on .

After upgrading to ProGet 2023.22, you may have noticed that your Debian feeds are now called “Debian (Classic)” feeds. And if you create a new Debian feed, it’ll be listed as a “Debian (New)” feed.

This maintenance release also includes a new feed type for Alpine Linux (APK) Packages, and support for connectors in RPM (Yum) Feeds.

This article will primarily talk about Debian feeds and the changes to them.

Background: ProGet 2023’s New Indexing System

ProGet has grown quite a bit from its humble beginnings as a private NuGet server. Although we always planned to support multiple types of packages, our design was influenced by NuGet-type feeds and npm-type feeds.

When we added other package types, it always felt that we were “bolting-on” something new. This was especially true with the Linux-based feeds. This is because, behind the scenes, each feed type was almost a unique product in itself with its own protocols, data model, and more, all loosely connected by ProGet’s web interface.

As ProGet has evolved beyond a repository into Software Composition Analysis (SCA) and Security, we needed a more unified approach. In ProGet 2023, we did much of the back-end work required to provide better first-class support to all package types.

Now, we are able to build on that foundation to improve some of our earlier feed types. This maintenance release captures some big improvements that would have otherwise been impractical.

Why is there a “Debian (New)” feed type?

Prior to ProGet 2023.22, Debian feeds lacked a number of features that have been requested. However, they were structurally difficult or even impossible to add to the existing feed type. To mitigate the risk of bugs/regressions, we simply created a new feed type called “Debian (New)” and renamed the older feed type to “Debian (Classic)”.

The other changes in this release (Alpine (APK) Feed and RPM Connectors) added new functionality, so there’s nothing to really break.

What’s changed in the new Debian feeds?

Distribution is no longer tied to feed

Debian repositories are organized by distribution, then component, and finally architecture. In “Debian (Classic)” feeds, the feed name was always used as the distribution. This would make managing packages awkward, particularly if you were publishing the same package for multiple distros.

With new Debian feeds, you assign a distribution to a package when it is uploaded to ProGet, and a feed can contain as many distributions as you like.

Connectors to remote repositories

The new Debian feeds support connectors, and you can add one to any external repository. They work like other connectors in ProGet, and will aggregate both local and remote packages. However, there’s definitely some room for improvement here, particularly when working with official public repositories.

For example, we don’t currently support “snapshots” for Debian repositories; this seems to be a popular feature in products like Aptly. We can definitely add snapshots, but we really need more user feedback; snapshots seem a bit archaic in a containerized world, so it may not make sense for us to implement.

Improved feed signing

Previously, you had to manually generate a signing key for each Debian feed. This process is now automatic; a new public/private key pair is created for you when a new Debian feed is created, and you can regenerate manually if needed. Additionally, we have fixed a number of issues with the signed Release/InRelease files used by apt.

Improved performance

Unlike many other repositories, ProGet generates much of its data on-demand. This means that you don’t have to wait for an indexing or publishing job to run to have access to a new package, but it has some drawbacks for types of repositories that use a different model.

  • “Debian (Classic)” always generate full indexes on demand, which worked well for small feeds, but did not scale well.
  • “Debian (New)” use differential indexing; release and package indexes are still generated as needed, but unchanged parts are cached and full updates are much much more efficient.

Migrating to the new Debian feeds

We’ve made migrating to Debian (New) feeds as easy as possible; on the manage feed page for a Debian (Classic) feed, you can just use the “Feed Migration” tool to copy packages from multiple “Debian (Classic)” feeds into a single “Debian (New)” feed.

In ProGet 2024, we plan to rename the feed names again to “Debian (Legacy)” and “Debian”, and eventually we’ll remove the “Debian (Legacy)” feed type. But for the time being, you can use both types of Debian feeds.

As such, we recommend moving to the need feed type when possible and keep in mind that it’s a new feed type — and like all new software, bugs are very possible.

Alex Papadimoulis

Alex Papadimoulis

Navigation