Tony's Proxmox Reinstall and Migration Guide

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
====== Tony's Proxmox Reinstall and Migration Guide ======

===== Use case =====

There are many guides for doing "migrations" or "reinstalls" of proxmox online, sometimes with backups, sometimes with live copies etc.

==== My needs ====

I have an existing Proxmox 7 server running with a single boot SSD using LVM for PVE. I have additional SSDs for hosting my LXC and VM images, and hard drives that are bind-mounted into the LXCs where bulk storage is needed (eg: Immich and Nextcloud data directories).

While this has backups of the containers, (and the bulk storage is RAID Z1), I am a single SSD failure away from the system not booting, which I consider catastrophic when it is a 6 hour $1000 flight away (thank Air Canada for that one).

I would like to reinstall PVE on this system, using two SSDs with ZFS RAID 1 for PVE, as 1) ZFS fits in far more with my current setup handling snapshots seemlessly, and 2) it gives me redundant boot and EFI partitions should one of the SSD's outright fail.

==== What is different ====

  * I have NO quorum, I use a single instance of PVE.
  * I can NOT do a live migration, as I have a single host, and will be reusing the hardware.
  * I do NOT want to install all my LXCs and VMs from a backup - they live on a separate SSD I will not be wiping, and I would like to reuse those pools as is.
  * I do NOT want to backup and restore PVE, I am changing SSD configurations (1 drive LVM to 2 drive ZFS)
  * My old setup had many levels of jank during a PVE 7 to 8 upgrade, avoid re-using unnecessary old configs


Again, what I want, is to reinstall PVE onto a new SSD with a new filesystem, reuse my existing VM files and pools already on the drives.
No backup/restore of LXCs. Just a simple, take everything, plunk it into new PVE.


===== The solution =====

Turns out the solution is actually relatively simple.

Since I tried a few things first and re-did this process twice, let me put some thoughts:

  * Proxmox reads the configs for your VMs and LXCs from /etc/pve/nodes.
  * I setup a full Proxmox Backup Server as an LXC another PVE instance (different location - not clustered)


==== The easy way ====

  - Make backups of everything you care about.
    - IF YOU WILL BE RE-USING SSD'S, MAKE SURE YOU DD A BACKUP OF THEM!
      - This one saved me and allowed me to repeat this procedure twice.
  - Grab the important config files you will need from the old host
    - /etc/pve/hosts
      - Ignore the keys and stuff, it'll cause pain and misery
    - /etc/wireguard/wg0.conf
      - If you rely on wireguard and want to be lazy and re-use old keys (not best security practice)
    - /etc/fstab
      - If you have non-standard mounts (I had an older btrfs HDD not in an zpool or lvm)
    - /etc/exports
      - If you have custom NFS exports to go from PVE over to VMs (LXCs just bind mount)
    - /etc/network/interfaces
      - If you have custom networking or VLAN setups
    - ~/.ssh/authorized_keys
    - Any special configs you have.
  - Swap out the SSDs for the new ones, prepare proxmox installation media
  - Install Proxmox fresh on your two new SSDs!
    - You should be able to boot into your new system
  - Bring your system roughly in line with your old setup. For me, this included
    - Re-setup network interfaces and bridges the way they were
    - Restore my fstab, wireguard, exports
      - Wireguard needs apt install wireguard wireguard-tools
      - systemctl enable wg-quick@wg0.service --now (start your tunnel at boot)
  - Re-import your storage devices (eg: my VM/LXC SSDs, data HDDs)
    - Make sure they are all detected under disks!
    - For my data HDD zpool, this was two steps
      - zpool import tonydata
        - causes it to show up in node -> disks
      - add it in the proxmox UI
        - datacenter -> storage -> add new -> zfs
    - For my LVM SSDs, this was 1 step as they were already listed under disks
      - add it in proxmox UI
        - datacenter -> storage -> add new -> lvm thin
    - For my old BTRFS HDD, this was 3 steps
      - Restore my old fstab (verifying UUID stayed the same)
      - mount -a
      -  datacenter -> storage -> add new -> btrfs
        - Remember to choose the types of storage this is (backup and data, not disk images in my case)
    - Export my NFS pools (this one I forgot to do until my VM was mad)
      - apt install nfs-kernel-server
      - restore /etc/exports
      - exportfs -av
  - Note, if you miss the above, it's probably not that bad, you'll be warned if things fail to start, it's iterative.
  - Restore /etc/pve/nodes/pve/lxc and /etc/pve/nodes/pve/qemu-server
    - You should now see your left bar populate back up with all your past lxc and VMs
    - Try starting them, maybe they work, maybe they error, if so look back to above.


That was it. Didn't end up using any of my backups from the backup server (surprisingly). Only used losetup on my dd to grab config files for networks and such retroactively to remember names and addresses.


===== Traps =====

  * Do NOT, try to blindly restore your /etc/pve
    * Doing so will leave your /etc/pve in a bad state with keys that are lingering
      * You will NOT be able to log in and will need to systemctl stop, clear /etc/pve and such
  * Connection error 401: permission denied - invalid PVE ticket
    * See my above point
  * Do not try to RSYNC into /etc/pve
    * It will fail. It is a FUSE mount and will refuse the temp files rsync makes.
    * Use scp or an rsync flag to go direct to output file.
  * Do NOT try to restore your PVE host from Proxmox Backup Server
    * Restoring the backup will yield errors with file overwrites
    * Allowing overwrites will break your configs in two and fail on /etc/pve
    * NOTE: proxmox-backup-client doesn't even backup /etc/pve by default (it only does root) so it's no use if you forgot that anyway
  * PROXMOX BACKUP SERVER IS A RED HERRING!!
    * In fact, if you don't need to back up and restore VMs, and you've taken a DD of your old SSD, you don't need it.
    * it is excellent software for offsite backing up items, just not my current use case
  * If it's not obvious, if you are using drive letters, like /dev/sda, sdb, YOU NEED TO STOP.
    * Please just use UUIDs, or heck even labels. After you reinstall, things WILL be shifting!!


In fact, let me do a whole segment on this

==== Proxmox backup server? ====

There seems to be some confusion with how this works, as a backup server can be added in two ways.

  - Datacenter -> Storage
    - This can be used for backing up the LXC and VMs on your system.
      - Note: VM and LXC items backed up this way will keep some config data without /etc/pve, specifically ram and cpu amounts
    - This can not be used to backup PVE itself
  - proxmox-backup-client
    - This can be used to backup PVE. by default, backs up a partition of your choosing
    - If you backup /, make sure you also include /etc/pve which is a fuse mount
      - If you already DDed the drive, you can always losetup and mount the disk image instead of using pxar.