The problem was, there was nowhere room on my harddisks to create
a real disk partition for swapping, so I had to make room.
Luckily, due to its design, LVM allows stuff to be migrated across
harddisks.
I deleted unneeded logical volumes and shrank some others, and used
lvm2defrag until I
arrived in the situation that the
sdc2 physical volume was
completely empty, and could be safely removed from the volume group.
lvm> pvs
PV VG Fmt Attr PSize PFree
/dev/sda1 owner lvm2 a- 372.61G 29.11G
/dev/sdb1 owner lvm2 a- 372.61G 5.96G
/dev/sdc2 owner lvm2 a- 186.11G 186.11G
This is where I did a mistake.
lvm> pvremove /dev/sdc2
Can't pvremove physical volume "/dev/sdc2" of volume group "owner" without -ff
Huh?
lvm> pvremove --help
pvremove: Remove LVM label(s) from physical volume(s)
pvremove
[-d|--debug]
[-f[f]|--force [--force]
[-h|-?|--help]
[-t|--test]
[-v|--verbose]
[-y|--yes]
[--version]
PhysicalVolume [PhysicalVolume...]
lvm> pvremove -f /dev/sdc2
Can't pvremove physical volume "/dev/sdc2" of volume group "owner" without -ff
lvm> pvremove -ff /dev/sdc2
Really WIPE LABELS from physical volume "/dev/sdc2" of volume group "owner" [y/n]? n
/dev/sdc2: physical volume label not removed
Command failed with status code 5.
lvm> pvremove -ff /dev/sdc2
Really WIPE LABELS from physical volume "/dev/sdc2" of volume group "owner" [y/n]? y
WARNING: Wiping physical volume label from /dev/sdc2 of volume group "owner"
Labels on physical volume "/dev/sdc2" successfully wiped
I wondered why it required these double --force flags. Soon I wondered no longer.
lvm> lvs
Couldn't find device with uuid '92MWt7-CPjs-Zt5D-iac0-bsKg-wTIr-Vj4Kc7'.
Couldn't find all physical volumes for volume group owner.
Couldn't find device with uuid '92MWt7-CPjs-Zt5D-iac0-bsKg-wTIr-Vj4Kc7'.
Couldn't find all physical volumes for volume group owner.
Couldn't find device with uuid '92MWt7-CPjs-Zt5D-iac0-bsKg-wTIr-Vj4Kc7'.
Couldn't find all physical volumes for volume group owner.
Couldn't find device with uuid '92MWt7-CPjs-Zt5D-iac0-bsKg-wTIr-Vj4Kc7'.
Couldn't find all physical volumes for volume group owner.
Volume group "owner" not found
Huh???
lvm> pvs
Couldn't find device with uuid '92MWt7-CPjs-Zt5D-iac0-bsKg-wTIr-Vj4Kc7'.
Couldn't find device with uuid '92MWt7-CPjs-Zt5D-iac0-bsKg-wTIr-Vj4Kc7'.
Couldn't find device with uuid '92MWt7-CPjs-Zt5D-iac0-bsKg-wTIr-Vj4Kc7'.
Couldn't find device with uuid '92MWt7-CPjs-Zt5D-iac0-bsKg-wTIr-Vj4Kc7'.
Couldn't find device with uuid '92MWt7-CPjs-Zt5D-iac0-bsKg-wTIr-Vj4Kc7'.
Couldn't find all physical volumes for volume group owner.
Couldn't find device with uuid '92MWt7-CPjs-Zt5D-iac0-bsKg-wTIr-Vj4Kc7'.
Couldn't find all physical volumes for volume group owner.
Couldn't find device with uuid '92MWt7-CPjs-Zt5D-iac0-bsKg-wTIr-Vj4Kc7'.
Couldn't find all physical volumes for volume group owner.
Couldn't find device with uuid '92MWt7-CPjs-Zt5D-iac0-bsKg-wTIr-Vj4Kc7'.
Couldn't find all physical volumes for volume group owner.
Volume group "owner" not found
Skipping volume group owner
<repeated three times>
lvm> lvscan
Couldn't find device with uuid '92MWt7-CPjs-Zt5D-iac0-bsKg-wTIr-Vj4Kc7'.
Couldn't find all physical volumes for volume group owner.
Couldn't find device with uuid '92MWt7-CPjs-Zt5D-iac0-bsKg-wTIr-Vj4Kc7'.
Couldn't find all physical volumes for volume group owner.
Couldn't find device with uuid '92MWt7-CPjs-Zt5D-iac0-bsKg-wTIr-Vj4Kc7'.
Couldn't find all physical volumes for volume group owner.
Couldn't find device with uuid '92MWt7-CPjs-Zt5D-iac0-bsKg-wTIr-Vj4Kc7'.
Couldn't find all physical volumes for volume group owner.
Volume group "owner" not found
At this point, it became soon apparent to me that I screwed up, majorly.
See, I used this command:
pvremove Remove LVM label(s) from physical volume(s)
When I should have in fact used this command:
vgreduce Remove physical volume(s) from a volume group
Luckily, the server was still operating just fine. It's just that the LVM
tools did not seem to honor the existence of my volume group anymore.
I became desperately aware of the fact that if the server is now rebooted,
it may not be able to access any volumes at all.
Literally
all data on the server would be lost!
First, I tried to use LVM's tools to rescue the lost partition.
lvm> lvscan
Couldn't find device with uuid '92MWt7-CPjs-Zt5D-iac0-bsKg-wTIr-Vj4Kc7'.
Couldn't find all physical volumes for volume group owner.
Couldn't find device with uuid '92MWt7-CPjs-Zt5D-iac0-bsKg-wTIr-Vj4Kc7'.
Couldn't find all physical volumes for volume group owner.
Couldn't find device with uuid '92MWt7-CPjs-Zt5D-iac0-bsKg-wTIr-Vj4Kc7'.
Couldn't find all physical volumes for volume group owner.
Couldn't find device with uuid '92MWt7-CPjs-Zt5D-iac0-bsKg-wTIr-Vj4Kc7'.
Couldn't find all physical volumes for volume group owner.
Volume group "owner" not found
lvm> pvdata --help
pvdata: Display the on-disk metadata for physical volume(s)
pvdata
[-a|--all]
[-d|--debug]
[-E|--physicalextent]
[-h|-?|--help]
[-L|--logicalvolume]
[-P[P]|--physicalvolume [--physicalvolume]
[-U|--uuidlist]
[-v[v]|--verbose [--verbose]
[-V|--volumegroup]
[--version]
PhysicalVolume [PhysicalVolume...]
lvm> pvdata
There's no 'pvdata' command in LVM2.
Use lvs, pvs, vgs instead; or use vgcfgbackup and read the text file backup.
Metadata in LVM1 format can still be displayed using LVM1's pvdata command.
I also tried
vgcfgrestore, which is supposed to restore lost metadata.
It didn't help either.
Soon I realized, that I need to look further for help.
I looked into the partitions themselves. I wanted to see what the partition has eaten.
root@chii:/dev# hexdump -C /dev/sdc2 |more
00000000 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
00000800 00 07 c0 00 0c 00 00 00 80 94 f2 45 00 07 c8 00 |.........E...|
00000810 0c 00 00 00 80 94 f2 46 04 07 d0 00 0c 00 00 00 |......F.......|
00000820 80 94 f2 47 00 07 d8 00 0c 00 00 00 80 94 f2 48 |..G.........H|
00000830 00 07 e0 00 0c 00 00 00 80 94 f2 49 00 07 e8 00 |.........I...|
00000840 0c 00 00 00 80 94 f2 4a 04 07 f0 00 0c 00 00 00 |......J.......|
00000850 80 94 f2 4b 00 07 f8 00 0c 00 00 00 80 94 f2 4c |..K.........L|
00000860 00 08 00 00 0c 00 00 00 80 94 f2 4d 00 08 08 00 |..........M....|
<snip>
00000ff0 0c 00 00 00 80 94 f3 11 04 0d 10 00 0c 00 00 00 |...............|
00001000 ae fa 4b 8a 20 4c 56 4d 32 20 78 5b 35 41 25 72 |K. LVM2 x[5A%r|
00001010 30 4e 2a 3e 01 00 00 00 00 10 00 00 00 00 00 00 |0N*>............|
00001020 00 f0 02 00 00 00 00 00 00 ca 00 00 00 00 00 00 |..............|
00001030 a0 0c 00 00 00 00 00 00 e1 1c c6 87 00 00 00 00 |.............|
00001040 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
00001200 2d 62 73 4b 67 2d 77 54 49 72 2d 56 6a 34 4b 63 |-bsKg-wTIr-Vj4Kc|
00001210 37 22 0a 64 65 76 69 63 65 20 3d 20 22 2f 64 65 |7".device = "/de|
00001220 76 2f 73 64 63 32 22 0a 0a 73 74 61 74 75 73 20 |v/sdc2"..status |
00001230 3d 20 5b 22 41 4c 4c 4f 43 41 54 41 42 4c 45 22 |= ["ALLOCATABLE"|
root@chii:/dev# hexdump -C /dev/sdb1 |more
00000000 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
00000200 4c 41 42 45 4c 4f 4e 45 01 00 00 00 00 00 00 00 |LABELONE........|
00000210 87 0a f4 de 20 00 00 00 4c 56 4d 32 20 30 30 31 |.. ...LVM2 001|
00000220 64 66 4f 78 66 46 41 56 54 59 44 63 44 67 71 4e |dfOxfFAVTYDcDgqN|
00000230 36 44 70 47 65 70 46 34 32 57 49 42 44 63 56 62 |6DpGepF42WIBDcVb|
00000240 00 04 f9 26 5d 00 00 00 00 00 03 00 00 00 00 00 |..&]...........|
00000250 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00000260 00 00 00 00 00 00 00 00 00 10 00 00 00 00 00 00 |................|
00000270 00 f0 02 00 00 00 00 00 00 00 00 00 00 00 00 00 |...............|
00000280 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
00001000 c7 58 f6 36 20 4c 56 4d 32 20 78 5b 35 41 25 72 |X6 LVM2 x[5A%r|
00001010 30 4e 2a 3e 01 00 00 00 00 10 00 00 00 00 00 00 |0N*>............|
00001020 00 f0 02 00 00 00 00 00 00 f4 00 00 00 00 00 00 |..............|
00001030 a0 0c 00 00 00 00 00 00 e1 1c c6 87 00 00 00 00 |.............|
00001040 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
00001200 31 0a 0a 73 65 67 6d 65 6e 74 31 20 7b 0a 73 74 |1..segment1 {.st|
00001210 61 72 74 5f 65 78 74 65 6e 74 20 3d 20 30 0a 65 |art_extent = 0.e|
00001220 78 74 65 6e 74 5f 63 6f 75 6e 74 20 3d 20 32 35 |xtent_count = 25|
00001230 0a 0a 74 79 70 65 20 3d 20 22 73 74 72 69 70 65 |..type = "stripe|
<snip>
root@chii:/dev# hexdump -C /dev/sda1 |more
00000000 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
00000200 4c 41 42 45 4c 4f 4e 45 01 00 00 00 00 00 00 00 |LABELONE........|
00000210 54 73 f8 85 20 00 00 00 4c 56 4d 32 20 30 30 31 |Ts. ...LVM2 001|
00000220 78 64 4e 63 70 6c 32 6b 65 46 73 73 45 68 5a 71 |xdNcpl2keFssEhZq|
00000230 4e 69 4f 69 47 65 79 58 55 32 41 38 6a 66 56 77 |NiOiGeyXU2A8jfVw|
00000240 00 04 f9 26 5d 00 00 00 00 00 03 00 00 00 00 00 |..&]...........|
00000250 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00000260 00 00 00 00 00 00 00 00 00 10 00 00 00 00 00 00 |................|
00000270 00 f0 02 00 00 00 00 00 00 00 00 00 00 00 00 00 |...............|
00000280 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
00001000 c7 58 f6 36 20 4c 56 4d 32 20 78 5b 35 41 25 72 |X6 LVM2 x[5A%r|
00001010 30 4e 2a 3e 01 00 00 00 00 10 00 00 00 00 00 00 |0N*>............|
00001020 00 f0 02 00 00 00 00 00 00 f4 00 00 00 00 00 00 |..............|
00001030 a0 0c 00 00 00 00 00 00 e1 1c c6 87 00 00 00 00 |.............|
00001040 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
00001200 31 0a 0a 73 65 67 6d 65 6e 74 31 20 7b 0a 73 74 |1..segment1 {.st|
00001210 61 72 74 5f 65 78 74 65 6e 74 20 3d 20 30 0a 65 |art_extent = 0.e|
00001220 78 74 65 6e 74 5f 63 6f 75 6e 74 20 3d 20 32 35 |xtent_count = 25|
00001230 0a 0a 74 79 70 65 20 3d 20 22 73 74 72 69 70 65 |..type = "stripe
<snip>
As I looked at this information, it became apparent to me what pvremove had done
to the partition. It had overwritten a section from 00000200..000000FFF, which
describes the contents of the LVM physical volume. So to make it functional again,
I would need to reconstruct that section from the disk.
"That should be easy", I thought to myself, as I've already got two backups,
on
sda1 and
sdb1.
Just in case I accidentally do even more harm, I made some backups:
root@chii:/dev# dd if=/dev/sdc2 bs=16 count=288 of=/sdc2-backup
root@chii:/dev# dd if=/dev/sdb1 bs=16 count=288 of=/sdb1-backup
root@chii:/dev# dd if=/dev/sda1 bs=16 count=288 of=/sda1-backup
And then I copied the section from
sdb1 to
sdc2:
root@chii:/dev# dd if=/dev/sdb1 bs=16 count=288 of=sdc2
288+0 records in
288+0 records out
4608 bytes (4.6 kB) copied, 0.00936218 s, 492 kB/s
root@chii:/dev# hexdump -C /dev/sdc2 |more
00000000 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
00000200 4c 41 42 45 4c 4f 4e 45 01 00 00 00 00 00 00 00 |LABELONE........|
00000210 87 0a f4 de 20 00 00 00 4c 56 4d 32 20 30 30 31 |.. ...LVM2 001|
00000220 64 66 4f 78 66 46 41 56 54 59 44 63 44 67 71 4e |dfOxfFAVTYDcDgqN|
00000230 36 44 70 47 65 70 46 34 32 57 49 42 44 63 56 62 |6DpGepF42WIBDcVb|
00000240 00 04 f9 26 5d 00 00 00 00 00 03 00 00 00 00 00 |..&]...........|
00000250 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00000260 00 00 00 00 00 00 00 00 00 10 00 00 00 00 00 00 |................|
00000270 00 f0 02 00 00 00 00 00 00 00 00 00 00 00 00 00 |...............|
00000280 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
00001000 c7 58 f6 36 20 4c 56 4d 32 20 78 5b 35 41 25 72 |X6 LVM2 x[5A%r|
00001010 30 4e 2a 3e 01 00 00 00 00 10 00 00 00 00 00 00 |0N*>............|
00001020 00 f0 02 00 00 00 00 00 00 f4 00 00 00 00 00 00 |..............|
00001030 a0 0c 00 00 00 00 00 00 e1 1c c6 87 00 00 00 00 |.............|
00001040 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
00001200 2d 62 73 4b 67 2d 77 54 49 72 2d 56 6a 34 4b 63 |-bsKg-wTIr-Vj4Kc|
00001210 37 22 0a 64 65 76 69 63 65 20 3d 20 22 2f 64 65 |7".device = "/de|
00001220 76 2f 73 64 63 32 22 0a 0a 73 74 61 74 75 73 20 |v/sdc2"..status |
00001230 3d 20 5b 22 41 4c 4c 4f 43 41 54 41 42 4c 45 22 |= ["ALLOCATABLE"|
<snip>
This did not help: LVM still complained.
Couldn't find device with uuid '92MWt7-CPjs-Zt5D-iac0-bsKg-wTIr-Vj4Kc7'.
I checked out the volume group backup file at
/etc/lvm/backup/owner to see
what are the "uuid"s for
sda1 and
sdb1, and sure enough, they matched
the regions at 0220..023F in each of those partitions.
So I entered the uuid to
sdc2 manually:
echo -n '92MWt7CPjsZt5Diac0bsKgwTIrVj4Kc7' | dd of=/dev/sdc2 bs=1 seek=544 conv=notrunc
32+0 records in
32+0 records out
32 bytes (32 B) copied, 0.00629778 s, 5.1 kB/s
root@chii:/dev# hexdump -C /dev/sdc2 |more
00000000 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
00000200 4c 41 42 45 4c 4f 4e 45 01 00 00 00 00 00 00 00 |LABELONE........|
00000210 87 0a f4 de 20 00 00 00 4c 56 4d 32 20 30 30 31 |.. ...LVM2 001|
00000220 39 32 4d 57 74 37 43 50 6a 73 5a 74 35 44 69 61 |92MWt7CPjsZt5Dia|
00000230 63 30 62 73 4b 67 77 54 49 72 56 6a 34 4b 63 37 |c0bsKgwTIrVj4Kc7|
00000240 00 04 f9 26 5d 00 00 00 00 00 03 00 00 00 00 00 |..&]...........|
<snip>
To my surprise, this did not help. LVM was still complaining as before.
So I investigated in more detail:
lvm> pvs -d -v -v -d
Setting global/locking_type to 1
File-based locking selected.
Setting global/locking_dir to /var/lock/lvm
report/aligned not found in config: defaulting to 1
report/buffered not found in config: defaulting to 1
report/headings not found in config: defaulting to 1
report/separator not found in config: defaulting to
report/prefixes not found in config: defaulting to 0
report/quoted not found in config: defaulting to 1
report/columns_as_rows not found in config: defaulting to 0
report/pvs_sort not found in config: defaulting to pv_name
report/pvs_cols_verbose not found in config: defaulting to pv_name,vg_name,pv_fmt,pv_attr,pv_size,pv_free,dev_size,pv_uuid
Scanning for physical volume names
/dev/sda1: lvm2 label detected
Label checksum incorrect on /dev/sdc2 - ignoring
/dev/sdc2: No label detected
/dev/sdb1: lvm2 label detected
Label checksum incorrect on /dev/sdc2 - ignoring
<snip>
It's complaining about a
label checksum, and consequently ignoring
the whole partition. This would be troublesome. How can I fix a broken
checksum?
At this point, I realized where the solution is. It is in the
source code!
So I looked up Linux kernel source code.
root@chii:/usr/src/linux/drivers/md# grep -i checksum dm*
root@chii:/usr/src/linux/drivers/md#
Huh? No matches? It must be userspace then.
root@chii:/usr/local/src# apt-get source lvm2
Reading package lists... Done
Building dependency tree
Reading state information... Done
Need to get 610kB of source archives.
Get:1 http://ftp.de.debian.org testing/main lvm2 2.02.39-2 (dsc) [1132B]
Get:2 http://ftp.de.debian.org testing/main lvm2 2.02.39-2 (tar) [594kB]
Get:3 http://ftp.de.debian.org testing/main lvm2 2.02.39-2 (diff) [14.5kB]
Fetched 610kB in 1s (306kB/s)
dpkg-source: extracting lvm2 in lvm2-2.02.39
dpkg-source: info: unpacking lvm2_2.02.39.orig.tar.gz
dpkg-source: info: applying lvm2_2.02.39-2.diff.gz
root@chii:/usr/local/src# cd lvm2*/
root@chii:/usr/local/src/lvm2-2.02.39# find -type f|xargs grep -i checksum
./po/lvm2.po:msgid "%s: Checksum error"
./po/lvm2.po:msgid "Incorrect metadata area header checksum"
./po/lvm2.po:msgid "Label checksum incorrect on %s - ignoring"
./lib/label/label.c: log_info("Label checksum incorrect on %s - "
<snip>
Jackpot! So I would need to check out what's
lib/label/label.c doing
when it complains about a checksum.
if (calc_crc(INITIAL_CRC, &lh->offset_xl, LABEL_SIZE -
uintptr_t) &lh->offset_xl - (uintptr_t) lh !=
xlate32(lh->crc_xl {
log_info("Label checksum incorrect on %s - "
"ignoring", dev_name(dev;
continue;
}
Oh, CRC32? Neat. How about
label.h?
/* On disk - 32 bytes */
struct label_header {
int8_t id[8]; /* LABELONE */
uint64_t sector_xl; /* Sector number of this label */
uint32_t crc_xl; /* From next field to end of sector */
uint32_t offset_xl; /* Offset from start of struct to contents */
int8_t type[8]; /* LVM2 001 */
} __attribute__ packed;
Oh, neater! So now I know the structure of the LVM disklabel header.
00000200 4c 41 42 45 4c 4f 4e 45 01 00 00 00 00 00 00 00 |LABELONE........|
00000210 87 0a f4 de 20 00 00 00 4c 56 4d 32 20 30 30 31 |.. ...LVM2 001|
00000220 39 32 4d 57 74 37 43 50 6a 73 5a 74 35 44 69 61 |92MWt7CPjsZt5Dia|
00000230 63 30 62 73 4b 67 77 54 49 72 56 6a 34 4b 63 37 |c0bsKgwTIrVj4Kc7|
00000240 00 04 f9 26 5d 00 00 00 00 00 03 00 00 00 00 00 |..&]...........|
00000250 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00000260 00 00 00 00 00 00 00 00 00 10 00 00 00 00 00 00 |................|
00000270 00 f0 02 00 00 00 00 00 00 00 00 00 00 00 00 00 |...............|
00000280 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
So in this, I would need to replace
87 0a f4 de with the CRC32 checksum
of the data from 0214 to 03FF (end of the sector). That's easy. We have a CRC32 tool.
To verify that I'm getting it right, I checked using one of the intact partitions.
root@chii:/dev# hexdump -C /dev/sda1 |head -n 20
00000000 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
00000200 4c 41 42 45 4c 4f 4e 45 01 00 00 00 00 00 00 00 |LABELONE........|
00000210 54 73 f8 85 20 00 00 00 4c 56 4d 32 20 30 30 31 |Ts. ...LVM2 001|
00000220 78 64 4e 63 70 6c 32 6b 65 46 73 73 45 68 5a 71 |xdNcpl2keFssEhZq|
00000230 4e 69 4f 69 47 65 79 58 55 32 41 38 6a 66 56 77 |NiOiGeyXU2A8jfVw|
00000240 00 04 f9 26 5d 00 00 00 00 00 03 00 00 00 00 00 |..&]...........|
00000250 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00000260 00 00 00 00 00 00 00 00 00 10 00 00 00 00 00 00 |................|
00000270 00 f0 02 00 00 00 00 00 00 00 00 00 00 00 00 00 |...............|
00000280 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
root@chii:/dev# dd if=/dev/sda1 skip=$[512+16+4] bs=1 count=$[512-8-8-4] > ~/tmptmp; crc32 ~/tmptmp
492+0 records in
492+0 records out
492 bytes (492 B) copied, 0.0014547 s, 338 kB/s
b7b1c8e3
Hmm, this does not match. B7B1C8E3 is not the same as 85F87354.
Why does it not match?
Well, the secret turned out to be that the INITIAL_CRC value in the source
code from
label.c is not the same as the initial CRC value used in standard
CRC32 implementation.
I checked
lib/misc/crc.h and found this:
#define INITIAL_CRC 0xf597a6cf
uint32_t calc_crc(uint32_t initial, const void *buf, uint32_t size);
So I would need to write a custom CRC32 program that would use this particular
initialization value for the CRC checksum.
Luckily, the CRC32 program I used was a Perl script, so it would be easy to edit it.
Here's the original:
#!/usr/bin/perl -w
eval 'exec /usr/bin/perl -w -S $0 ${1+"$@"}'
if 0; # not running under some shell
# computes and prints to stdout the CRC-32 values of the given files
use lib qw( blib/lib lib );
use Archive::Zip;
use FileHandle;
my $totalFiles = scalar(@ARGV);
foreach my $file (@ARGV) {
if ( -d $file ) {
warn "$0: ${file}: Is a directory\n";
next;
}
my $fh = FileHandle->new();
if ( !$fh->open( $file, 'r' ) ) {
warn "$0: $!\n";
next;
}
binmode($fh);
my $buffer;
my $bytesRead;
my $crc = 0;
while ( $bytesRead = $fh->read( $buffer, 32768 ) ) {
$crc = Archive::Zip::computeCRC32( $buffer, $crc );
}
printf( "%08x", $crc );
print("\t$file") if ( $totalFiles > 1 );
print("\n");
}
Here's how I changed it:
<snip>
my $crc = ~0xf597a6cf;
while ( $bytesRead = $fh->read( $buffer, 32768 ) ) {
$crc = Archive::Zip::computeCRC32( $buffer, $crc );
}
$crc = ~$crc;
printf( "%08x", $crc );
<snip>
Now to test:
root@chii:/usr/local/src/lvm2-2.02.39/lib# dd if=/dev/sda1 skip=$[512+16+4] bs=1 count=$[512-8-8-4] > ~/tmptmp; /root/crc32tmp ~/tmptmp
492+0 records in
492+0 records out
492 bytes (492 B) copied, 0.00149226 s, 330 kB/s
85f87354
85f87354 -- that's a match. Now I would take a checksum of
sdc2 and poke it back in:
root@chii:/dev# dd if=/dev/sdc2 skip=$[512+16+4] bs=1 count=$[512-8-8-4] > ~/tmptmp; ~/crc32tmp ~/tmptmp
492+0 records in
492+0 records out
492 bytes (492 B) copied, 0.0147573 s, 33.3 kB/s
414798fb
root@chii:/dev# echo -ne '\xfb\x98\x47\x41'|dd of=/dev/sdc2 bs=1 seek=$[512+16]
4+0 records in
4+0 records out
4 bytes (4 B) copied, 0.00801839 s, 0.5 kB/s
root@chii:/dev# hexdump -C /dev/sdc2 |head -n 20
00000000 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
00000200 4c 41 42 45 4c 4f 4e 45 01 00 00 00 00 00 00 00 |LABELONE........|
00000210 fb 98 47 41 20 00 00 00 4c 56 4d 32 20 30 30 31 |.GA ...LVM2 001|
00000220 39 32 4d 57 74 37 43 50 6a 73 5a 74 35 44 69 61 |92MWt7CPjsZt5Dia|
00000230 63 30 62 73 4b 67 77 54 49 72 56 6a 34 4b 63 37 |c0bsKgwTIrVj4Kc7|
00000240 00 04 f9 26 5d 00 00 00 00 00 03 00 00 00 00 00 |..&]...........|
<snip>
Time to test whether it worked!
Next I deleted
sdc2 and replaced it with a swap partition,
and tried the hibernate option, but it didn't work. For some reason,
after writing 1 GB of stuff to the swap partition and shutting down
all non-essential harddisk, the kernel suddenly was unable to find
any swap device!
[1749825.798000] PM: Cannot find swap device, try swapon -a.
[1749826.271752] Restarting tasks ... done.
And it brought the system back running. I guess Linux has problems
hibernating to SATA devices. But that's another story.
The point of this article was to show how to salvage a partition
accidentally destroyed with
pvremove, and that it did :)
In retrospect, this command would probably have saved me the hassle
of doing hex editing and stuff:
pvcreate /dev/sdc2 --uiid '92MWt7-CPjs-Zt5D-iac0-bsKg-wTIr-Vj4Kc7'
It would create a disklabel on
sdc2 with exactly the right UUID,
without touching the volume group metatables, i.e. filling the precise
information that was missing, which I did manually. Oh well. The next
time I'll know that. At least I learned a thing or two: about the format
of LVM disklabels, and to never ignore two forcing flags
and capital letter warnings.
In 2010, a Michal Docekal wrote to me, sharing this detail:
I just got to your article named "How to undo pvremove"
on your web page. I got to a very similar situation a while ago:
debian:~# pvremove -ff /dev/hdb1
debian:~# vgdisplay
Couldn't find device with uuid '3wvOQk-M0uS-XJN6-nlrl-x8Uz-CJDF-in2fEs'. Couldn't find all physical volumes for volume group data.
...
Volume group "data" doesn't exist
In the end, I discovered a much faster way to undo this:
vgreduce data --removemissing
And that was it. I don't know if there was this option back in 2008,
but in any way, it would be good if you added this information to the
article, so that other desperate admins can find it.