Day 05: Jenkins & Nginx: Production Infrastructure Guide

Jenkins & Nginx: Production Infrastructure Guide

1. Thread Summary

This thread covered the end-to-end setup of a production-ready Jenkins environment on AWS EC2. Key milestones included:

• Storage: Preparing and formatting new EBS volumes safely.
• Installation: Setting up Jenkins with Java 17 and official repositories.
• Security: Using Nginx as a Reverse Proxy to handle SSL (HTTPS) via Let’s Encrypt and DuckDNS.
• Deep Dive: Exploring Nginx request routing, “stealth” status codes (444), and protocol behavior.

---

2. Nginx Request Flow Logic

Nginx processes requests in a specific hierarchy. Understanding this is key to debugging.

The Decision Tree

1. Validation: Checks if the HTTP request is valid (e.g., must have a Host header).
2. Port Binding: Routes to the server block listening on the specific port (80 or 443).
3. Host Matching: Matches the Host header against server_name directives.
4. Fallback: If no match is found, Nginx uses the block marked default_server. If none is marked, it picks the first loaded block.

---

3. Analysis of Test Cases

We explored four distinct scenarios using curl to see how Nginx reacts:

Case	Command	Result	Reason
Standard HTTP	curl -I http://domain.com	301 Moved	Matches Port 80 block; triggers redirect.
Direct IP Access	curl http://1.2.3.4	444 / 404	No Host match; triggers the Catch-All/Default block.
Blank Host	curl -H "Host:" http://url	400 Bad Request	Protocol violation; HTTP/1.1 requires a Host value.
Spoofed Host	curl -H "Host: evil.com"	444 No Response	Valid protocol, but Host doesn’t match; triggers Catch-All.

---

4. Essential Command Reference

System & Storage

# Identify disk name
lsblk
# Check for existing filesystem (Safety check)
sudo file -s /dev/xvdf
# Format new volume
sudo mkfs -t ext4 /dev/xvdf
# Check listening ports
sudo ss -tunlp | grep -E ':(80|8080)\b'

Nginx Management

# Verify configuration syntax (Run after every edit!)
sudo nginx -t
# Reload configuration without dropping connections
sudo systemctl reload nginx
# View real-time access logs
tail -f /var/log/nginx/access.log

SSL & Certbot

# Get certificate and auto-configure Nginx
sudo certbot --nginx -d yourdomain.duckdns.org
# Test certificate renewal
sudo certbot renew --dry-run

---

5. Optimized Configuration Snippets

The “Stealth” Catch-All (Default Block)

Add this to the top of your config to drop bot/IP traffic silently.

server {
    listen 80 default_server;
    server_name _;
    return 444; # Closes connection immediately
}

---

6. The Optimized jenkins.conf Structure

Located at: /etc/nginx/conf.d/jenkins.conf

# 1. Upstream Definition: Defines where the Jenkins Java process is running.
upstream jenkins {
    keepalive 32;               # Keeps connections open to reduce latency
    server 127.0.0.1:8080;      # Local Jenkins port
}

# 2. HTTP Server: The "Receptionist" that forces everyone to use encryption.
server {
    listen 80;
    server_name merojenkins.duckdns.org;
    return 301 https://$host$request_uri; # Permanent redirect to HTTPS
}

# 3. HTTPS Server: The "Secure Proxy" that handles encryption and traffic logic.
server {
    listen 443 ssl;
    server_name merojenkins.duckdns.org;

    # SSL Paths (Automatically added/managed by Certbot)
    ssl_certificate /etc/letsencrypt/live/merojenkins.duckdns.org/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/merojenkins.duckdns.org/privkey.pem;

    location / {
        # Core Proxying
        proxy_pass http://jenkins;
        
        # Headers: Essential for Jenkins to know the client's real identity
        proxy_set_header Host $host;                          # Passes the domain name used
        proxy_set_header X-Real-IP $remote_addr;              # Passes the user's IP address
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; # Tracks the proxy chain
        proxy_set_header X-Forwarded-Proto $scheme;           # Tells Jenkins the user is using HTTPS

        # Performance & Compatibility
        proxy_http_version 1.1;      # Required for WebSockets (live console logs)
        proxy_set_header Connection ""; # Clears hop-by-hop headers for persistent connections
        proxy_buffering off;         # Prevents Nginx from waiting to send build logs
        proxy_request_buffering off; # Required for large file uploads/plugins
        
        # Security
        client_max_body_size 100m;   # Allows uploading larger Jenkins plugins/artifacts
    }
}

---

7. Top Takeaways

1. Never format without checking: Use file -s to ensure you aren’t wiping an existing filesystem on a detached EBS volume.
2. Master-Worker Model: Nginx uses a root master to grab port 80, but unprivileged workers to handle web traffic for security.
3. 444 is for Bots: Use return 444 to save bandwidth and hide your server identity from scanners.
4. Proxy Headers: Always pass X-Forwarded-Proto so the backend knows the connection is secure.