Day 04: Full Technical Runbook: Jenkins & Nginx on EC2

Full Technical Runbook: Jenkins & Nginx on EC2

1. Disk Initialization & Persistent Mounting

Scenario: Adding a new NVMe drive for Jenkins home directory.

• Format (Destructive): sudo mkfs -t xfs /dev/nvme1n1
• Mount & Ownership: sudo mount /dev/nvme1n1 /var/lib/jenkins/ sudo chown -R jenkins:jenkins /var/lib/jenkins/
• Permanent Mount (fstab):
  1. Get UUID: sudo blkid /dev/nvme1n1
  2. Edit: sudo vi /etc/fstab
  3. Add: UUID=your-uuid-here /var/lib/jenkins xfs defaults,nofail 0 2
  4. Test: sudo umount /var/lib/jenkins && sudo mount -a

2. Debugging Jenkins Service Failures

Scenario: Service is “flapping” or shows AccessDeniedException.

• Reset Systemd Throttling: sudo systemctl reset-failed jenkins
• Bypass Systemd to see Real-Time Errors: sudo -u jenkins /usr/bin/jenkins
• Identify Process Path & Overrides: systemctl cat jenkins
• Retrieve Initial Admin Password: sudo cat /var/lib/jenkins/secrets/initialAdminPassword

3. Nginx Reverse Proxy Configuration

File: /etc/nginx/conf.d/jenkins.conf

upstream jenkins {
    keepalive 32;
    server 127.0.0.1:8080;
}
server {
    listen 80;
    server_name _; # Catch-all for any IP/Domain
    location / {
        proxy_pass http://jenkins;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_buffering off; # Essential for live console logs
    }
}

• Apply Changes: sudo nginx -t && sudo systemctl restart nginx

4. Security & Access Control

• Fix Nginx “403 Forbidden” (SELinux): sudo setsebool -P httpd_can_network_connect 1
• Verify Port Binding: sudo netstat -tulpn | grep -E '80|8080'
• Analyze Configuration Defaults: sudo nginx -T | grep -e "configuration file" -e "listen 80"

5. Systemd Override Simulation

Scenario: Running a binary with strace to find “hidden” file errors.

• Edit Override: sudo systemctl edit nginx
• Correct syntax for replacement:

  [Service]
  ExecStart=
  ExecStart=/usr/bin/strace -o /tmp/nginx_trace.log -f /usr/sbin/nginx
  

• Trace Output: tail -f /tmp/nginx_trace.log