Day 02: Debugging Service Level Permission Issues

Forensic Mental Map: Debugging Linux Service Permissions

When a service like Jenkins fails silently or logs are empty, follow this structured exclusion process to find the “blocker.”

---

Phase 1: The Identity Check (Linux Filesystem)

Goal: Prove the service user has basic OS-level access.

• Check Ownership: Does the user own the target directory?
  • ls -ld /var/log/jenkins
• Check Permissions: Is the directory writable (755 or 775)?
  • ls -l /var/log/jenkins
• The “Impersonation” Test: Can the user manually create a file?
  • sudo -u jenkins touch /var/log/jenkins/test.txt
  • If this fails, the fix is: chown -R jenkins:jenkins /var/log/jenkins

---

Phase 2: The “Silent” Barrier (Systemd Sandboxing)

Goal: Check if Systemd is restricting the process beyond standard OS permissions.

• Inspect Restrictions: Look for security “sandboxing” flags.
  • systemctl cat jenkins | grep -E "Protect|ReadWrite|Private"
• Evaluate Flags:
  • ProtectSystem=full/strict: Prevents writing to system directories.
  • ReadWritePaths=: If this list exists, the service cannot write anywhere else.
• The “Tackle”: Add your path to the allowed list in your override:
  • ReadWritePaths=/var/log/jenkins/

---

Phase 3: The “Invisible Wall” (Security Policies)

Goal: Identify blocks from Mandatory Access Control (SELinux/AppArmor).

• Check Enforcement: Is a security module active?
  • getenforce (Should return Enforcing or Permissive)
• Audit Logs: Check for “denied” messages the app didn’t see.
  • sudo ausearch -m avc -ts recent
• The “Isolation” Test: Temporarily disable the wall.
  • sudo setenforce 0
  • If the log starts working, the fix is: sudo restorecon -Rv /var/log/jenkins

---

Phase 4: The “Forensic” Start (Manual Execution)

Goal: Bypass the service manager to see “raw” error output.

• Extract Command: Find exactly what command Systemd runs.
  • systemctl show jenkins -p ExecStart
• Manual Run: Run that exact command as the jenkins user.
  • sudo -u jenkins /usr/bin/java -jar ...
• Analyze Trace: Watch the terminal for “Java Stack Traces” or “Access Denied” errors that journalctl might have missed.

---

Phase 5: The Application Logic (Config vs. Reality)

Goal: Ensure the app is actually trying to write where you think it is.

• Path Verification: Does the path in override.conf match the physical folder?
• Buffering: Is the app holding data in memory?
  • Tackle: Trigger 20+ web requests to force a buffer flush.

Jenkins Administration & Linux Package Management Summary

1. Package Management (DNF vs. YUM)

• The Transition: DNF is the modern successor to YUM, offering superior dependency resolution (via libsolv) and performance. On modern systems, the yum command is typically a symlink to dnf.
• Repository Logic: Repositories host the software and metadata, while GPG keys serve as a “tamper-proof seal” to verify package authenticity.
• Source Lists: Unlike the single sources.list in Debian/Ubuntu, RPM-based systems use individual .repo files located in /etc/yum.repos.d/.

2. Jenkins Installation & Configuration

• The Workflow: Installation requires adding the official repo, importing the GPG key, installing Java (the runtime engine), and finally installing the Jenkins package.
• Service Defaults: Jenkins runs under a dedicated jenkins user and defaults to port 8080.
• Critical Paths:
  • Data: /var/lib/jenkins
  • Logs: /var/log/jenkins

3. The “Golden Rule” of Customization: Overrides

• Avoid Direct Edits: Never modify /usr/lib/systemd/system/jenkins.service directly; these changes are lost during updates.
• Use Overrides: Utilize sudo systemctl edit jenkins to create drop-in configurations.
• Syntax Matters: Custom variables (e.g., JENKINS_PORT, JENKINS_ENABLE_ACCESS_LOG) must be placed under a [Service] header to be recognized.
• Activation: Always run sudo systemctl daemon-reload and restart the service to apply changes.

4. Troubleshooting & The Forensic Mindset

• Validation vs. Raw Text:
  • systemctl cat: Displays the raw text on disk (useful for checking syntax).
  • systemctl show: Displays what the system has actually “parsed” and accepted.
• The Forensic Mental Map:
  1. Ownership: Verify directory owners via ls -ld.
  2. Impersonation: Test write permissions with sudo -u jenkins touch <file>.
  3. Visibility: Check if SELinux is blocking access using getenforce (test with setenforce 0).
  4. Raw Output: Run the Java startup command manually as the jenkins user to see “silent” errors missed by journalctl.

5. Access Logging

• Activation: Setting JENKINS_ENABLE_ACCESS_LOG=true generates an access_log for HTTP traffic.
• Empty Logs: If the log is empty, it usually indicates a lack of traffic to the port or that the output is currently buffered in memory.