How to install PROXMOX VE 3.1 on SOFTWARE-RAID

Pubblicato da admin in 22 agosto 2013
Pubblicato in: Linux, Proxmox. 20 commenti

I’ve been asked to install Proxmox VE 3.0 on a new server having a Intel RSTe raid controller. Unfortunately this raid controller is not well supported under linux and even if properly configured, when I tried to install Proxmox VE 3.0 from the provided ISO image I could not see the RAID disk but I was seeing the two separate disks.

After some search I found this good HowTo from Dominic Pratt (http://dominicpratt.de/proxmox-ve-3-0-software-raid/). It is in German, I translated with google and after some issues I could reach my goal.

I am posting more or less the same steps in English for future use, hopefully it will be better than translate the Dominic’s post with google (even if I do not bet a cent about this!)

Step 1) Download the ISO

Download from http://www.proxmox.com

Step 2) Install Proxmox  from the ISO 

Install on the disk /dev/sda

Step 3) Upgrade proxmox to the latest packages

apt-get update
apt-get dist-upgrade
apt-get install vim (to avoid issue with vi)

Step 4) Install Raid packages

apt-get update (Not necessary if you executed the same command in step 3)
apt-get install mdadm

During the installation mdadm asks two questions. Click OK with no modification

Step 5) Now we need to copy the partition table from sda to sdb

sfdisk -d /dev/sda | sfdisk -f /dev/sdb

Step 6) Convert the partitions on /dev/sdb to RAID disks

sfdisk -c /dev/sdb 1 fd
sfdisk -c /dev/sdb 2 fd

Step 7) Initialize the RAID

mdadm --create /dev/md0 --level=1 --raid-disks=2 missing /dev/sdb1
mdadm --create /dev/md1 --level=1 --raid-disks=2 missing /dev/sdb2

Step 8) Copy the /boot on /dev/md0

mkfs.ext3 /dev/md0
mkdir /mnt/md0
mount /dev/md0 /mnt/md0
cp -ax /boot/* /mnt/md0

Now we need to edit the /etc/fstab file. In the line containing UUID

Es.   UUID=c8ac59b8-ce54-4fca-8107-1b04aaa0194d /boot ext3 defaults 0 1

we replace the UUID part with /dev/md0

/dev/md0 /boot ext3 defaults 0 1

Step 9) Reboot the system

shutdown -r now

after the reboot we can check the result of the command:

mount | grep boot

it must show a line similar to this one:

/dev/md0 on /boot type ext3 (rw,relatime,errors=continue,user_xattr,acl,barrier=0,data=ordered)

Now we can tell to GRUB that we want to boot from /dev/md0

echo 'GRUB_DISABLE_LINUX_UUID=true' >> /etc/default/grub
echo 'GRUB_PRELOAD_MODULES="raid dmraid"' >> /etc/default/grub
echo raid1 >> /etc/modules
echo raid1 >> /etc/initramfs-tools/modules
grub-install /dev/sda
grub-install /dev/sdb
update-grub
update-initramfs -u

Step 10) Add /dev/sda1 to raid

sfdisk -c /dev/sda 1 fd
mdadm --add /dev/md0 /dev/sda1

Step 11) Move LVM on /dev/md1

Proxmox installation created a LVM on /dev/sda2 so we need to move it on /dev/md1 and then remove it from /dev/sda2

pvcreate /dev/md1
vgextend pve /dev/md1
pvmove /dev/sda2 /dev/md1

The pvmove steps will take some time depending on the CPU and the disk size (in my case, with a Intel(R) Xeon(R) CPU E5-2620 0 @ 2.00GHz and 1 TB, it took from 2 to 3 hours)

vgreduce pve / dev/sda2
pvremove / dev/sda2

Step 12) Add /dev/sda2 to raid

sfdisk-change-id /dev/sda2 fd
mdadm-add /dev/md1 /dev/sda2

Now, executing the command

/proc/mdstat

we can see the RAID is executing the sync of the disks.

Now I’ve a Proxmox VE 3.1-3 running on a Software RAID1

20 commenti su “How to install PROXMOX VE 3.1 on SOFTWARE-RAID”

  1. Pingback: Proxmox 3.1 auf Softraid mit GPT « ClearOS und Zarafa

  2. Dear Boff Blog,

    Can you show me how if i want to use RAID 10 ?
    I’ve installed proxmox in server with 4 unit 2TB HDD.
    Right now is only 1 HDD is used.
    please give me some clue…

    Thanks…
    Sorry for bad english.

    Rispondi

    • Got Proxmox 3.2 working on a raid10 like this:
      mdadm –create /dev/md0 –level=10 –raid-disks=4 missing /dev/sdb2 /dev/sdc2 /dev/sdd2
      mdadm –create /dev/md1 –level=10 –raid-disks=4 missing /dev/sdb3 /dev/sdc3 /dev/sdd3

      One problem i encountered was that the original proxmox 3.2 partitions were GPT, therefore I had to change the sfdisk commands to:
      sgdisk -R=/dev/sdb /dev/sda (copy partitions from sda to sdb; same command with sdc/sdd)
      sgdisk -t 2:fd00 /dev/sda (change partition type of /dev/sda2 to raid)

      Rispondi

  3. Did you not encounter this problem?

    root@proxmox:~# mdadm –create /dev/md0 –level=1 –raid-disks=2 missing /dev/sdb1
    mdadm: cannot open /dev/sdb1: No such file or directory
    root@proxmox:~# partprobe
    root@proxmox:~# mdadm –create /dev/md0 –level=1 –raid-disks=2 missing /dev/sdb1
    mdadm: cannot open /dev/sdb1: Device or resource busy
    root@proxmox:~# dmsetup table
    pve-swap: 0 8388608 linear 8:2 2048
    pve-root: 0 201326592 linear 8:2 8390656
    pve-data: 0 732463104 linear 8:2 209717248
    root@proxmox:~# dmsetup status
    pve-swap: 0 8388608 linear
    pve-root: 0 201326592 linear
    pve-data: 0 732463104 linear
    root@proxmox:~# cat /proc/mdstat
    Personalities : [raid1]
    md0 : active raid1 sdb[0]
    488255360 blocks super 1.2 [2/1] [U_]

    unused devices:
    root@proxmox:~# mdadm –create /dev/md1 –level=1 –raid-disks=2 missing /dev/sdb2
    mdadm: cannot open /dev/sdb2: Device or resource busy

    Etc…

    Rispondi

  4. Answering my own post, for a reason I can’t remember if I figured it out, but I had to simply do “mdadm –stop /dev/md0” and the same for md1. Then the raid creation worked. Also the partprobe is essential to do after you modify the partitions on any drives.

    So I did go through the rest of the instructions and, as I apprehended, I ended up with an unbootable system. Here I only had the ProxMox CD, nothing else. I could have reinstalled from scratch, but I had a couple of VMs (which I did not bothered to lose… but) I tried to recover the system anyway, in case somebody else follow this (and for my own short-term future reference).

    –First off, thanks for these instructions.

    — Second, I think it would be possible (and much safer) to configure all raid1 partitions on sdb ONLY at first and then boot from it using GRUB2 command line. If it fails, you can still boot normally from sda, if it succeed, THEN copy the raid1 partitions from sdb to sda.

    To boot from GRUB2 command line, when you see the grub menu, press c and at the grub prompt, type:

    grub> insmod raid
    grub> insmod mdraid1x
    grub> insmod lvm
    grub> set root=(md/0)
    grub> linux /vmlinuz-2.6.32-26-pve root=/dev/mapper/pve-root
    grub> initrd /initrd.img-2.6.32-26-pve
    grub> boot

    (once set root= is define, you can “ls /” to get the exact filenames for the kernel and the image)

    Then proceed and follow the instruction parts on adding sda to the raid running on sdb.

    — Third, I think the part that failed me in these instructions is that “dmraid” had to be replace by “mdraid1x” in GRUB_PRELOAD_MODULES.

    Since I knew ProxMox install on sda, I simply reinstalled ProxMox from the CD knowing sdb would be untouched (with my existing VMs). Once booted from (the newly reinstalled) sda, I could then reinstall the mdadm tools, which started the arrays on sdb and I could mount md0 to fix it… (chanding “dmraid” to “mdraid1x” and reissue “update-grub” and “update-initramfs -u”.

    To not screwup my virgin ProxMox on sda, I poweroff the system and disconnected the sda drive. So I simply booted from a one drive raid1 from sdb and it worked. I the poweroff, rehook the sda drive and manually boot from the grub command from md0. o ‘mbootdoff the rd, the root file system is from LVM on md1, and I have access to sda to add he partitions the raid1.

    Rispondi

  5. The last couple of steps have some syntax issues:

    vgreduce pve / dev/sda2 should be vgreduce pve /dev/sda2

    pvremove / dev/sda2 should be pvremove /dev/sda2

    sfdisk-change-id / dev / sda 2 fd should be sfdisk –change-id /dev/sda 2 fd

    mdadm-add / dev/md1 / dev/sda2 should be mdadm –add /dev/md1 /dev/sda2

    And /proc/mdstat should be either cat /proc/mdstat or watch cat /proc/mdstat (or something equivalent)

    Rispondi

  6. It works like a charm! I succesfully installed proxmox VE 3.1 with raid1 in an HP proliant ml350e gen8 server with dynamic smart array (soft raiid) disabled.

    Thank you very much

    Stefano

    Rispondi

  7. I’ve used proxmox 1.9 with mdadm (software raid1). After some the system started to hang from time to time. I’ve investigated the issue and it seems that when you use snapshot feature then this may cause the whole system hang. Is there are any similar issues with 3.x?

    Rispondi

  8. In step 7 i got
    mdadm: An option must be given to set the mode before a second device
    (/dev/md0) is listed
    and i can’t continue, can you help please?
    thanks

    Rispondi

  12. Pingback: Proxmox VE 3.1でソフトウェアRAIDを使ってみた（システム領域含む） | 俺的備忘録 〜なんかいろいろ〜

  13. Pingback: How to install PROXMOX VE 3.2 on SOFTWARE-RAID (MDRAID) | Articoll

  14. I find this very interesting tutorial and I’m starting to implement a raid1 config with proxmox 3.3.
    But instaling proxmox3 create a GPT partition table and nor fdisk nor sfdisk support GPT partitions.
    Any update needed for this tutorial with parted ?

    Rispondi

Lascia un commento