RAID6: Eine Platte entfernt - RAID degraded - dieselbe Platte wieder eingesteckt - wie kann man vollständigen rebuild vermeiden?

[framp]

Given I have no spare DS to test my assumption I setup a similar setup on one of my Raspberries:

3 devices with 3 partitions, partition 1 used for DSM (/dev/md0) , partition 2 used for SWAP (/dev/md1) and partition 3 (/dev/md2) used for a SP.

Then I umounted and stopped /dev/md2 and removed /dev/sda. /dev/md0 and /dev/md1 degraded as expected. /dev/md2 was not listed in /proc/mdstat and didn't get notified of the removal of a disk.

Then I inserted /dev/sda again and added /dev/sda1, /dev/sda2 and /dev/sda3. /dev/md0 and /dev/md1 are rebuilded because they were active when I removed /dev/sda but /dev/md2 is immediately available again. This proves my assumption is correct: If the RAID is inactive the removal of a disk will not be detected by mdadm if the RAID5 is deactivated and a reinsertion of the disk will not initiate a rebuild

I created a small script in order to document all the steps.

pi@raspberrypi-bookworm64-lite-regression-gpt:~ $ sudo ./setupADM.sh 1
/bin/bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
*** Removing mdadm config
rm: cannot remove '/etc/mdadm/mdadm.conf': No such file or directory

*** Umounting RAIDs
umount: /dev/md0: not mounted.
umount: /dev/md1: not mounted.
umount: /dev/md2: not mounted.

*** Wipe RAID partition
umount: /dev/md0: not mounted.
/dev/md0: 2 bytes were erased at offset 0x00000438 (ext4): 53 ef
umount: /dev/md1: not mounted.
/dev/md1: 2 bytes were erased at offset 0x00000438 (ext4): 53 ef
umount: /dev/md2: not mounted.
/dev/md2: 2 bytes were erased at offset 0x00000438 (ext4): 53 ef

*** Stopping RAIDs
mdadm: stopped /dev/md0
mdadm: stopped /dev/md1
mdadm: stopped /dev/md2

*** Wipe partitions
umount: /dev/sda1: not mounted.
umount: /dev/sda2: not mounted.
umount: /dev/sda3: not mounted.
/dev/sda1: 4 bytes were erased at offset 0x00001000 (linux_raid_member): fc 4e 2b a9
/dev/sda2: 4 bytes were erased at offset 0x00001000 (linux_raid_member): fc 4e 2b a9
/dev/sda3: 4 bytes were erased at offset 0x00001000 (linux_raid_member): fc 4e 2b a9
/dev/sda: 8 bytes were erased at offset 0x00000200 (gpt): 45 46 49 20 50 41 52 54
/dev/sda: 8 bytes were erased at offset 0x1d9bffe00 (gpt): 45 46 49 20 50 41 52 54
/dev/sda: 2 bytes were erased at offset 0x000001fe (PMBR): 55 aa
umount: /dev/sdb1: not mounted.
umount: /dev/sdb2: not mounted.
umount: /dev/sdb3: not mounted.
/dev/sdb1: 4 bytes were erased at offset 0x00001000 (linux_raid_member): fc 4e 2b a9
/dev/sdb2: 4 bytes were erased at offset 0x00001000 (linux_raid_member): fc 4e 2b a9
/dev/sdb3: 4 bytes were erased at offset 0x00001000 (linux_raid_member): fc 4e 2b a9
/dev/sdb: 8 bytes were erased at offset 0x00000200 (gpt): 45 46 49 20 50 41 52 54
/dev/sdb: 8 bytes were erased at offset 0x3a3dffe00 (gpt): 45 46 49 20 50 41 52 54
/dev/sdb: 2 bytes were erased at offset 0x000001fe (PMBR): 55 aa
umount: /dev/sdc1: not mounted.
umount: /dev/sdc2: not mounted.
umount: /dev/sdc3: not mounted.
/dev/sdc1: 4 bytes were erased at offset 0x00001000 (linux_raid_member): fc 4e 2b a9
/dev/sdc2: 4 bytes were erased at offset 0x00001000 (linux_raid_member): fc 4e 2b a9
/dev/sdc3: 4 bytes were erased at offset 0x00001000 (linux_raid_member): fc 4e 2b a9
/dev/sdc: 8 bytes were erased at offset 0x00000200 (gpt): 45 46 49 20 50 41 52 54
/dev/sdc: 8 bytes were erased at offset 0xede9ffe00 (gpt): 45 46 49 20 50 41 52 54
/dev/sdc: 2 bytes were erased at offset 0x000001fe (PMBR): 55 aa

*** Zero superblock
mdadm: Couldn't open /dev/sda1 for write - not zeroing
mdadm: Couldn't open /dev/sdb1 for write - not zeroing
mdadm: Unrecognised md component device - /dev/sdc1
mdadm: Couldn't open /dev/sda2 for write - not zeroing
mdadm: Couldn't open /dev/sdb2 for write - not zeroing
mdadm: Couldn't open /dev/sdc2 for write - not zeroing
mdadm: Couldn't open /dev/sda3 for write - not zeroing
mdadm: Couldn't open /dev/sdb3 for write - not zeroing
mdadm: Couldn't open /dev/sdc3 for write - not zeroing

*** Allocate partitions
Creating new GPT entries in memory.
The operation has completed successfully.
The operation has completed successfully.
The operation has completed successfully.
Creating new GPT entries in memory.
The operation has completed successfully.
The operation has completed successfully.
The operation has completed successfully.
Creating new GPT entries in memory.
The operation has completed successfully.
The operation has completed successfully.
The operation has completed successfully.

*** Creating RAIDs
mdadm: Note: this array has metadata at the start and
    may not be suitable as a boot device.  If you plan to
    store '/boot' on this device please ensure that
    your boot-loader understands md/v1.x metadata, or use
    --metadata=0.90
mdadm: size set to 1046528K
Continue creating array? mdadm: Defaulting to version 1.2 metadata
mdadm: array /dev/md0 started.
mdadm: Note: this array has metadata at the start and
    may not be suitable as a boot device.  If you plan to
    store '/boot' on this device please ensure that
    your boot-loader understands md/v1.x metadata, or use
    --metadata=0.90
mdadm: size set to 1046528K
Continue creating array? mdadm: Defaulting to version 1.2 metadata
mdadm: array /dev/md1 started.
mdadm: layout defaults to left-symmetric
mdadm: layout defaults to left-symmetric
mdadm: chunk size defaults to 512K
mdadm: size set to 1046528K
mdadm: Defaulting to version 1.2 metadata
mdadm: array /dev/md2 started.

*** Formating RAID
mke2fs 1.47.0 (5-Feb-2023)
Creating filesystem with 261632 4k blocks and 65408 inodes
Filesystem UUID: a1d3c91c-c74e-40c7-84c6-d244196e4631
Superblock backups stored on blocks:
    32768, 98304, 163840, 229376

Allocating group tables: done                          
Writing inode tables: done                          
Creating journal (4096 blocks): done
Writing superblocks and filesystem accounting information: done

mke2fs 1.47.0 (5-Feb-2023)
Creating filesystem with 261632 4k blocks and 65408 inodes
Filesystem UUID: b9cc8b81-9ecd-47f6-ab79-18bda14ad2cb
Superblock backups stored on blocks:
    32768, 98304, 163840, 229376

Allocating group tables: done                          
Writing inode tables: done                          
Creating journal (4096 blocks): done
Writing superblocks and filesystem accounting information: done

mke2fs 1.47.0 (5-Feb-2023)
Creating filesystem with 523264 4k blocks and 130816 inodes
Filesystem UUID: 62e9477d-a929-4d02-bccb-b54dcae91fd1
Superblock backups stored on blocks:
    32768, 98304, 163840, 229376, 294912

Allocating group tables: done                          
Writing inode tables: done                          
Creating journal (8192 blocks): done
Writing superblocks and filesystem accounting information: done

Personalities : [linear] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md2 : active raid5 sdc3[3] sdb3[1] sda3[0]
      2093056 blocks super 1.2 level 5, 512k chunk, algorithm 2 [3/2] [UU_]
          resync=DELAYED
     
md1 : active raid1 sdc2[2] sdb2[1] sda2[0]
      1046528 blocks super 1.2 [3/3] [UUU]
          resync=DELAYED
     
md0 : active raid1 sdc1[2] sdb1[1] sda1[0]
      1046528 blocks super 1.2 [3/3] [UUU]
      [=>...................]  resync =  7.5% (79296/1046528) finish=7.1min speed=2265K/sec
     
unused devices: <none>

... waiting for RAID1 and RAID5 builds to finish with watch /proc/mdstat ...

 

[framp]

pi@raspberrypi-bookworm64-lite-regression-gpt:~ $ cat /proc/mdstat

Personalities : [linear] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md2 : active raid5 sdc3[3] sdb3[1] sda3[0]
      2093056 blocks super 1.2 level 5, 512k chunk, algorithm 2 [3/3] [UUU] 

md1 : active raid1 sdc2[2] sdb2[1] sda2[0]
      1046528 blocks super 1.2 [3/3] [UUU]    

md0 : active raid1 sdc1[2] sdb1[1] sda1[0]
      1046528 blocks super 1.2 [3/3] [UUU]   
unused devices: <none>

... creating some dummy files in the three RAIDs ...

pi@raspberrypi-bookworm64-lite-regression-gpt:~ $ sudo ./setupADM.sh 2

/bin/bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)

*** Persisting mdadm config

ARRAY /dev/md0 level=raid1 num-devices=3 metadata=1.2 name=0 UUID=1f5be406:c2a8f8cf:9365202e:17b85e2c
   devices=/dev/sda1,/dev/sdb1,/dev/sdc1
ARRAY /dev/md1 level=raid1 num-devices=3 metadata=1.2 name=1 UUID=f71f2ad5:fc67d341:5864504a:b9c12870
   devices=/dev/sda2,/dev/sdb2,/dev/sdc2
ARRAY /dev/md2 level=raid5 num-devices=3 metadata=1.2 name=2 UUID=17265588:1599d613:2faea66a:f0a41d26
   devices=/dev/sda3,/dev/sdb3,/dev/sdc3

*** Creating mount points
mkdir: cannot create directory '/mnt/md0': File exists
mkdir: cannot create directory '/mnt/md1': File exists
mkdir: cannot create directory '/mnt/md2': File exists

*** Mounting RAIDs

*** Creating files

/mnt/md0:
DSM  lost+found

/mnt/md1:
SWAP  lost+found

/mnt/md2:
SP  lost+found

... deactivating the SP ...

 

[framp]

pi@raspberrypi-bookworm64-lite-regression-gpt:~ $ sudo ./setupADM.sh 3

/bin/bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)

*** Deactivate /dev/md2

mdadm: stopped /dev/md2
Personalities : [linear] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md1 : active raid1 sdc2[2] sdb2[1] sda2[0]
      1046528 blocks super 1.2 [3/3] [UUU]  

md0 : active raid1 sdc1[2] sdb1[1] sda1[0]
      1046528 blocks super 1.2 [3/3] [UUU]     
unused devices: <none>

... now /dev/md0 and /dev/md1 are degraded and /dev/md2 is inactive

pi@raspberrypi-bookworm64-lite-regression-gpt:~ $ cat /proc/mdstat

Personalities : [linear] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md1 : active raid1 sdc2[2] sdb2[1]
      1046528 blocks super 1.2 [3/2] [_UU]

md0 : active raid1 sdc1[2] sdb1[1]
      1046528 blocks super 1.2 [3/2] [_UU]
unused devices: <none>

... now the removed device was inserted again

pi@raspberrypi-bookworm64-lite-regression-gpt:~ $ cat /proc/mdstat
Personalities : [linear] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]

md2 : active raid5 sdc3[3] sdb3[1] sda3[0]
      2093056 blocks super 1.2 level 5, 512k chunk, algorithm 2 [3/3] [UUU]
  
md1 : active raid1 sda2[3] sdc2[2] sdb2[1]
      1046528 blocks super 1.2 [3/2] [_UU]
          resync=DELAYED

md0 : active raid1 sda1[3] sdc1[2] sdb1[1]
      1046528 blocks super 1.2 [3/2] [_UU]
      [>....................]  recovery =  3.4% (36608/1046528) finish=1.3min speed=12202K/sec
unused devices: <none>

and /dev/md0 and /dev/md1 are rebuilded. But /dev/md2 is active immediately

I have no clue whether this will also work a Synology running DSM - but a pure Linux without any modifications is able to handle my scenario.

 

[Benares]

Öhm, was treibst du da gerade? Dein Prompt (pi@raspberrypi-bookworm64-lite-regression-gpt) lässt auf einen PI schließen.

 

[framp]

Given I have no spare DS to test my assumption I setup a similar setup on one of my Raspberries:

Das steht doch ganz am Anfang .. oder willst Du mir eine Spare DS spenden um das ganze noch mal auf einer DS zu testen?

 

[framp]

Falls jemand nachvollziehen will was ich gemacht habe - anbei das Script mit dem ich die 4 Schritte ausgeführt habe.

 

[framp]

Sorry - keine Ahnung warum - aber wenn ich das Script anhänge erscheint es trotzdem nicht im Thread. Wer Interesse hat kann mich per PN kontaktieren.

Ich habe das Script auf github hochgeladen.

 

[Benares]

Sorry, da kann dir wirklich keiner folgen. Du rufst da irgendwelche Scripte auf (z.B. setupADM.sh) ohne den Quelltext zu zeigen/verlinken und postest dann die Ausgaben. Wer soll das blicken?

 

[framp]

Ich habe das Script auf github hochgeladen.

Ich habe dem Script auf github einen etwas sprechenderen Namen gegeben.

ohne den Quelltext zu zeigen/verlinken

Leider funktioniert das Attachement des Scripts nicht - keine Ahnung warum. Darum habe ich es auf mein github repo hochgeladen. Wo ist Dein Problem? Funktioniert der Link auf mein Repo in meinem Thread nicht? ... Ich denke hier hat sich was überschnitten ...

 

[framp]

@Benares Ich habe hiermit mich mal in mdadm eingelesen um zu verifizieren dass meine Vermutung stimmt und Du kannst alles mit meinem verlinkten Script nachvollziehen - sofern Du eine Raspberry oder ein anderes Linux besitzt. Meine Vermutung stimmt jedenfalls . Ob das auf einer Syno funktioniert weiss ich nicht. Die haben ja ziemlich viel an Linux rumgefummelt. Aber Du kannst mir gerne eine spare Syno spenden damit ich das nochmal auf einer Syno verifizieren kann

 

[DaveR]

Correct. With a Synology NAS that supports the DX517 you don't the script.

 

[DaveR]

I'll test your script on my DS925+ later today.

I didn't know you had a website of Linux tip and tricks (in German and English). I'm impressed. But I did notice a typo in the 10 Years raspiBackup section. It says "0 years ago today the first version of raspiBackup was stored in my local cvs". Shouldn't that be "10 years ago..."?

 

[framp]

I just fixed the typo

I'll test your script on my DS925+ later today.

Oh - that's great. Will be interesting whether it works on DSM also

I actually created the script because it took me a while to get all commands I need compiled and had to start over a lot of time. It's actually the collection of the commands you have to execute to setup the three RAIDs and later on to add the removed disk again to the RAIDs. Don't hesitate to ask any questions if you need help. I'll try to help you to get the test executed on your system.

 

[Benares]

Ich denke nicht, dass dein Script auf einer DS funktioniert, das wird schon am umount scheitern. Ich hatte mir mal notiert, dass man auf einer DS erstmal mit "synospace --stop-all-spaces" (DSM7) bzw. "syno_poweroff_task -d" (<DSM7) alles mögliche stoppen muss, bevor ein umount klappt. Probiert habe ich das allerdings noch nie.

 

[framp]

Ich denke nicht, dass dein Script auf einer DS funktioniert,

1 zu 1 definitiv nicht. Aber ich bin mir sicher Dave weiss wie man DSM äquivalente Funktionen nutzen kann. Jedenfalls kann man im Script die notwendigen Schritte erkennen und an DSM anpassen.