Backups

1. Overview

• 3-2-1 strategy

  • 1st copy on the main disks
  • 2nd copy on local backup disk
  • 3rd copy of Tier 1 only in S3

• Tier 1 vs Tier 2 - Data is categorized into two tiers:

  • tier 1 - important data (encrypted, offsite + local)
  • tier 2 - replaceable data (local only)

2. Design

• Disk roles:
  • /srv - service definitions, docker configs, infra-as-code, scripts
  • /data - persistent app data (with subdirs per service)
  • /backup - dedicated backup disk
• Tagging
  • Scope: /srv + /data (Tier 1 + Tier 2)
  • Tagging: --tag local, --tag tier-mixed, --tag-srv-data

3. Implementation

• Environment Variables:
  • RESTIC_REPOSITORY=<local-repo>
  • RESTIC_PASSWORD_FILE=<password-location>
• Restic commands:

// manaul backup
sudo RESTIC_REPOSITORY=<local-repo> \
     RESTIC_PASSWORD_FILE=<password-location> \
     restic backup /srv /data \
       --tag local \
       --tag srv-data \
       --tag tier-mixed
      
// check snapshots
sudo RESTIC_REPOSITORY=<local-repo> \
     RESTIC_PASSWORD_FILE=<password-location> \
     restic snapshots

• A nightly cron job runs restic backup and a weekly job runs restic forget --prune.

Restore testing

• Monthly: restore a random snapshot to a temp directory and verify integrity