Day 20: RPM Package Management and manual GPG repository

5 minute read

Verifying and Installing Jenkins on Red Hat/RPM Systems: A Complete Deep Dive

Managing third-party repositories on Enterprise Linux systems often looks straightforward until you encounter security warnings, double cryptographic prompts, or GPG keyring mismatches.

This post consolidates how modern Jenkins installation, RPM package management, and manual GPG repository verification work under the hood.

1. Jenkins System & Java Requirements

Modern versions of Jenkins (LTS 2.555.1+ / Weekly 2.545+) enforce strict runtime environments.

Mandatory Runtimes: Jenkins requires Java 21 or Java 25.
Deprecated Runtimes: Support for Java 11 and Java 17 has been completely phased out.
Scope: This version restriction applies strictly to the environment running the Jenkins controller (jenkins.war) and its execution agents (remoting.jar). Your pipelines can still compile legacy codebases (like Java 8) using isolated tool configurations.

2. Managing the Local RPM Database

The rpm command interacts directly with the local system database to track installed binaries and trusted vendor keys.

Querying and Filtering Packages

Because standard listings yield massive outputs, use targeted filters to audit your system:

# List only installed packages
rpm -qa

# Count total installed packages
rpm -qa | wc -l

# Group and list regular packages (excluding GPG keys)
rpm -qa | grep -v '^gpg-pubkey'

# Group and list only trusted public keys
rpm -qa | grep '^gpg-pubkey'

Deleting a Trusted Public Key

If a repository key expires or needs rotation, it can be purged directly from the system database using its unique release hash:

# 1. Locate the exact gpg-pubkey name for Jenkins
rpm -qa gpg-pubkey* | xargs rpm -qi | grep -B 2 -A 5 -i jenkins

# 2. Delete the key from the RPM database
sudo rpm -e gpg-pubkey-14abfc68-654b9f2c

3. Cryptographic Verification: Key vs. Signature

A common point of confusion when looking at remote repository structures is distinguishing between the public key asset and the detached signature asset.

File Asset	URL Pattern	What It Actually Is
Public Key	`.../repodata/repomd.xml.key`	The public key block used to decrypt vendor signatures (acts as a generic alias link for dated files like `jenkins.io-2026.key`).
Detached Signature	`.../repodata/repomd.xml.asc`	The actual encrypted cryptographic seal generated by hashing the repository index metadata with the private authoring key.

4. Architecture Deep Dive: Why DNF Prompts for the Key Twice

When running sudo dnf install jenkins, you are prompted to accept the same GPG key fingerprint (0x14ABFC68) at two separate stages. DNF does not actually download the file twice over the network; instead, it uses the single key file to pass two isolated security gates.

Stage 1: Verifying the Repository Directory Metadata

DNF downloads the package repository directory index (repomd.xml) and its digital seal (repomd.xml.asc).
To ensure this list of available software hasn’t been modified by a malicious middleman, DNF reads the public key (repomd.xml.key).
Because this key is not yet present inside DNF’s temporary session keyring, it pauses the terminal and asks:
- “Is it ok to import this key to verify the repository metadata? [y/N]”
Accepting this allows DNF to securely parse the package list and resolve package dependencies.

Stage 2: Verifying the Heavy Package Payload File

Once you confirm the overall transaction, DNF downloads the actual application package binary file (jenkins-2.555.3-1.noarch.rpm).
Before writing these files to your permanent system directories, DNF hands the binary over to the core low-level system RPM database.
Because the core system RPM database is completely isolated from DNF’s temporary repository caching engine for security, the system database realizes it does not officially possess the key yet.
DNF pauses the transaction a second time to ask:
- “Is it ok to import this key into the permanent system RPM database? [y/N]”

Summary of the Checkpoints

Prompt #1 is the DNF Package Manager asking: “Can I trust this web repository’s menu list?”
Prompt #2 is the Core Operating System asking: “Can I permanently trust this specific binary application on my storage drive?”

5. Where the Public Key is Stored Individually

When you accept both prompts, the cryptographic key block is written to two entirely distinct locations on your filesystem, managed by completely separate subsystems:

1. DNF Engine Cache (Repository Level)

Storage Location: /var/cache/dnf/jenkins-stable-XXXXXXXXXXXX/pubring/ (or under /var/cache/dnf/jenkins-stable/gpgdir/)
Under the Hood: Stored as part of a temporary GPG keyring structure (pubring.kbx) that DNF loads exclusively to validate repository syncs and metadata updates.

2. System RPM Database (Operating System Level)

Storage Location: /var/lib/rpm/
Under the Hood: The key data is not saved as a loose plain text file. Instead, the cryptographic string is injected directly into the RPM database binary architecture (such as rpmdb.sqlite or Packages).
Because it lives in the core system index, the OS treats the public key as an installed software entity, which is why it shows up when running rpm -qa gpg-pubkey*.

6. How to Manually Verify Repository Integrity

To bypass automated package managers and prove a repository index is completely authentic and unaltered, you can perform a manual check using raw gpg utilities.

Step 1: Download the Repository Assets

# Download the data index
curl -o repomd.xml https://pkg.jenkins.io/rpm-stable/repodata/repomd.xml

# Download the detached signature
curl -o repomd.xml.asc https://pkg.jenkins.io/rpm-stable/repodata/repomd.xml.asc

# Download the matching public key
curl -o jenkins.key https://pkg.jenkins.io/rpm-stable/repodata/repomd.xml.key

Step 2: Populate Your User GPG Keyring

The system-wide rpm database is separate from your user-account GPG ring (~/.gnupg/pubring.kbx). Import the key to your personal environment profile:

gpg --import repomd.xml.key

Note: Use gpg --list-keys (or gpg -k) to verify the key block is active in your keyring.

Step 3: Execute the Cryptographic Check

Run the verification check by pointing the engine to the signature and data files:

gpg --verify repomd.xml.asc repomd.xml

Analyzing the Verification Output

A pristine, authentic repository index will return the following explicit confirmation:

gpg: Signature made Tue Jun  9 07:14:35 2026 UTC
gpg:                using RSA key 5E386EADB55F01504CAE8BCF7198F4B714ABFC68
gpg: Good signature from "Jenkins Project <[email protected]>" [unknown]
gpg: WARNING: This key is not certified with a trusted signature!

💡 Security Note on the Warning: The WARNING: This key is not certified message is completely normal. It is an inherent feature of GPG’s Web-of-Trust validation model indicating that you have mathematically verified the file, but have not manually assigned an explicit trust rating level to the vendor’s identity block.

Bidhan Sthapit