Bidhan Sthapit

Logging my devops journey

6 minute read

Building an Automated Internal PKI: From Local Testing to Production DevOps

Securing internal infrastructure is a critical responsibility for DevOps engineers. When managing internal tools like Sonatype Nexus, GitLab, or private APIs, using public Certificate Authorities (CAs) isn’t always feasible or cost-effective.

This post synthesizes a complete journey through SSL/TLS, breaking down the mechanics of the Global Trust Network, why modern browsers demand strict certificate structures, and how to build a production-grade Private Public Key Infrastructure (PKI) with automated renewals.

1. The Core Dilemma: Self-Signed vs. Trusted CA

When implementing HTTPS, developers generally choose between two routes:

  • Self-Signed Certificates: Free and easy to generate locally using tools like OpenSSL. However, because they lack a neutral third-party verification layer, public browsers will block them with a severe “Your connection is not private” warning.
  • Trusted CA Certificates: Issued by audited, universally trusted authorities (like Let’s Encrypt, DigiCert, or Sectigo) whose Root Certificates are hardcoded into operating systems like macOS and Windows.

Why You Can’t Just Use Let’s Encrypt for Everything

Public CAs like Let’s Encrypt require you to prove domain ownership via the internet using public DNS or HTTP challenges. If your target server (e.g., nexus.internal.local) sits deep within a private corporate VPC behind a VPN, Let’s Encrypt cannot reach it. To secure internal production networks safely without browser warnings, you must build your own Private Certificate Authority.

2. Demystifying the Global Chain of Trust

To understand how a private CA works, we must look at how browsers like Google Chrome evaluate public certificates on systems like macOS:

Use code with caution.[ Root CA ] ——–> Hardcoded into macOS Keychain Access (e.g., ISRG Root X1)│[ Intermediate CA ] -> Managed by the CA (e.g., Let’s Encrypt R10)│[ Leaf Certificate ] -> Installed on your web server (e.g., bidhansthapit.com.np)

  1. The Vault (Root CA): Top-tier authorities never use their Root Private Keys to sign daily website certificates. If a Root key leaks, the company falls completely out of trust globally. Instead, Root Private Keys are kept completely offline in physical, bomb-proof Hardware Security Modules (HSMs). They are used just once to sign Intermediate CAs.
  2. The Middlemen (Intermediate CAs): These online servers do the heavy lifting of signing individual requests.
  3. The Browser Check: When you visit a website, Chrome traces the signature on your certificate back to the Intermediate, and the Intermediate back to the Root. It then checks the macOS Keychain Access database. If a match is found (like Let’s Encrypt’s ISRG Root X1), the secure padlock lights up.

3. Designing a Zero-Prompt Production PKI Configuration

Modern browsers require explicit data properties to accept a certificate. Simple command-line prompts no longer cut it because browsers will reject any certificate missing a Subject Alternative Name (SAN) or an explicit Extended Key Usage (EKU) string.

The clean, DevOps-approved solution is to put all variables directly into passive configuration files. This strips complex parameters from your execution scripts, making them pristine and repeatable.

The Master CA Configuration (ca.cnf)

This file turns OpenSSL into a stateful, production-grade tracking engine by establishing a local indexing ledger (ca_index.txt) and a serial counter (rootCA.srl).

[ req ]
default_bits        = 4096      # Exponentially tougher for 10-year lifespan
prompt              = no
distinguished_name  = ca_dn
x509_extensions     = v3_ca
default_days        = 3650      # 10 Years
default_md          = sha256

[ ca_dn ]
C  = NP
O  = Internal Production Network
OU = Infrastructure DevOps
CN = Internal Production Root CA

[ v3_ca ]
subjectKeyIdentifier   = hash
authorityKeyIdentifier = keyid:always,issuer
basicConstraints       = critical, CA:true
keyUsage               = critical, digitalSignature, cRLSign, keyCertSign

[ ca ]
default_ca = production_ca

[ production_ca ]
database        = ./ca_index.txt
serial          = ./rootCA.srl
certs           = .
new_certs_dir   = .
private_key     = ./rootCA.key
certificate     = ./rootCA.crt
default_days    = 90            # Enforces short-lived 90-day security lifespan
default_md    = sha256
policy          = policy_loose
copy_extensions = copy          # Automatically copies SANs from the incoming CSR

The Client App Configuration (nexus.cnf)

This configuration establishes the requirements for your target application server, mapping out browser compatibility properties natively.

[ req ]
default_bits       = 2048       # 2048-bit keys are 5-8x faster than 4046 for high-traffic servers
prompt             = no
distinguished_name = nexus_dn
req_extensions     = v3_req     # Bakes SAN fields straight into the CSR

[ nexus_dn ]
C  = NP
O  = Internal Production Network
CN = nexus.internal.local

[ v3_req ]
keyUsage         = critical, digitalSignature, keyEncipherment
extendedKeyUsage = serverAuth, clientAuth
subjectAltName   = @alt_names

[ alt_names ]
DNS.1 = nexus.internal.local
DNS.2 = localhost
IP.1  = 127.0.0.1

4. The 3-Step Clean Execution Guide

With the configuration templates defined, prepare the tracking files once on your secure CA manager:

touch ca_index.txt
echo "1000" > rootCA.srl

Now, your execution commands map cleanly using simple file path arguments, entirely bypassing inline parameter flags:

Step 1: Initialize the Master Root CA

openssl req -x509 -new -nodes -config ca.cnf -keyout rootCA.key -out rootCA.crt
  • -x509 -new: Operational commands instructing OpenSSL to output a finalized Master Certificate instead of an intermediate request.
  • -nodes: “No DES”. Leaves the key unencrypted on the filesystem so background daemons can access it without a human typing a password at boot.

Step 2: Compile the Client Private Key & CSR

openssl req -new -nodes -config nexus.cnf -keyout nexus.key -out nexus.csr
  • Generates your 2048-bit server private key (nexus.key) and packages the domain metadata into an unsigned Certificate Signing Request (nexus.csr).

Step 3: Authoritatively Sign the Certificate

openssl ca -config ca.cnf -in nexus.csr -out nexus.crt -batch
  • Bypasses the stateless openssl x509 utility to leverage the stateful openssl ca database engine.
  • -batch: Suppresses all manual prompt blocks, making the execution instantly compatible with background scripts. This increments rootCA.srl and logs the transaction in ca_index.txt.

5. Industrial-Grade Automation: Systemd Timers over Cron

To maintain security compliance, short-lived 90-day internal certificates must rotate automatically. For production internal infrastructure, running bash scripts on standard crontab lines introduces operational blindspots.

Instead, a production DevOps architecture uses Systemd Timers. They integrate natively into system telemetry, log directly to journalctl, and can execute deployment hooks to notify monitoring stacks (like Slack, Datadog, or PagerDuty) if an execution step fails.

The Automated Renewal Script (/usr/local/bin/renew-nexus-cert.sh)

#!/usr/bin/env bash
set -euo pipefail

# 1. Regenerate ephemeral keys and request footprints
openssl req -new -nodes -config /etc/ssl/nexus.cnf -keyout /etc/ssl/nexus.key -out /tmp/nexus.csr

# 2. Re-sign authoritatively via the private PKI engine
openssl ca -config /opt/internal-ca/ca.cnf -in /tmp/nexus.csr -out /etc/nginx/ssl/nexus.crt -batch

# 3. Apply defensive permissions and safely reload proxies without breaking current user connections
chmod 600 /etc/ssl/nexus.key
systemctl reload nginx
echo "SSL rotation completed successfully."

The Systemd Service Unit (/etc/systemd/system/cert-renewal.service)

[Unit]
Description=Automated Internal Production SSL Certificate Renewal
After=network.target

[Service]
Type=oneshot
ExecStart=/usr/local/bin/renew-nexus-cert.sh
StandardOutput=journal
StandardError=journal

The Systemd Periodic Scheduler (/etc/systemd/system/cert-renewal.timer)

By running this check weekly, you create a robust safety cushion. If an upstream network or validation issue blocks the script on week one, your monitoring system will trigger an alert while your live instance still has weeks of buffer space left before expiration.

[Unit]
Description=Triggers internal SSL certificate renewal weekly

[Timer]
OnCalendar=Sun *-*-* 02:00:00
Persistent=true

[Install]
WantedBy=timers.target

Enable the automatic scheduling block instantly:

sudo systemctl daemon-reload
sudo systemctl enable --now cert-renewal.timer

6. Distributing the Trust Across the Cluster

Your automated infrastructure will function flawlessly behind the scenes, but clients will still display security exceptions until you explicitly inject your new root identity file into your cluster nodes and engineering environments:

  • On Your macOS Development Machine: Drag and drop your generated rootCA.crt file into the Keychain Access application under the System category. Double-click your entry, expand the Trust properties, and switch the option to Always Trust.
  • On Ubuntu/Debian Cluster Nodes: 
    sudo cp rootCA.crt /usr/local/share/ca-certificates/
sudo update-ca-certificates
  • On RHEL/CentOS Nodes: 
    sudo cp rootCA.crt /etc/pki/ca-trust/source/anchors/
sudo update-ca-trust

By executing this flow, you bridge the gap between architectural cryptography and enterprise automation, achieving a seamless, zero-trust infrastructure.

You May Also Enjoy