Krdytkyu

TLDR

How can I recover files from a hard disk containing LVM2 partitions after extracting the disk from a dead server?

In Brief

My headless email server appliance stopped responding to network connection attempts (IMAP, SSH, Ping, ...) and recovery attempts using a install/rescue image on a USB flash drive failed to get anywhere.

The appliance is a proprietary plug&play device based on a PowerPC port of Debian with web-based administration.

I have backups but they are incomplete (new email etc since backup). I need to recover data from this disk.

I removed the hard disk and attached it via a SATA-to-USB adapter to a freshly installed Ubuntu 16.04 on a desktop PC.

I found the appliance had used LVM but I was unable to mount the LVM partition on my recovery-PC. Hardware errors were reported.

How to recover data?

Full Details

I have a headless Linux mailserver appliance which stopped working. It was running a customised Debian on PowerPC (MPC8313E). After trying to resurrect it, I removed the hard-disk and used a SATA-to-USB adapter to attach the hard-disk to a Ubuntu 16.04 x86_64 desktop PC.

Ubuntu complained about LVM2 so I did sudo apt-get install lvm2 and tried to find out a little about LVM2, after which I tried the following

$ sudo lvdisplay

  --- Logical volume ---

  LV Path                /dev/bubba/storage

  LV Name                storage

  VG Name                bubba

  LV UUID                TlYu8Y-JahI-lnNT-Y8Tk-w8TE-CEn7-WD8Ytc

  LV Write Access        read/write

  LV Creation host, time ,  

  LV Status              available

  # open                 1

  LV Size                921.08 GiB

  Current LE             235797

  Segments               1

  Allocation             inherit

  Read ahead sectors     auto    

  - currently set to     256     

  Block device           253:0



$ sudo pvs

  /dev/bubba/storage: read failed after 0 of 4096 at 0: Input/output error

  /dev/bubba/storage: read failed after 0 of 4096 at 989004234752: Input/output error

  /dev/bubba/storage: read failed after 0 of 4096 at 989004292096: Input/output error

  /dev/bubba/storage: read failed after 0 of 4096 at 4096: Input/output error

Here's the disk partitioning

I tried some further manipulation:

$ sudo vgscan

  Reading all physical volumes.  This may take a while...

  Found volume group "bubba" using metadata type lvm2

$ sudo lvchange -an bubba/storage

$ sudo vgchange -an bubba

  0 logical volume(s) in volume group "bubba" now active

$ sudo vgscan

  Reading all physical volumes.  This may take a while...

  Found volume group "bubba" using metadata type lvm2

$ sudo vgchange -ay bubba

  1 logical volume(s) in volume group "bubba" now active

$ sudo lvscan

  ACTIVE            '/dev/bubba/storage' [921.08 GiB] inherit

$ sudo lvchange -ay bubba/storage

$ sudo lvdisplay

  --- Logical volume ---

  LV Path                /dev/bubba/storage

  LV Name                storage

  VG Name                bubba

  LV UUID                TlYu8Y-JahI-lnNT-Y8Tk-w8TE-CEn7-WD8Ytc

  LV Write Access        read/write

  LV Creation host, time ,

  LV Status              available

  # open                 0

  LV Size                921.08 GiB

  Current LE             235797

  Segments               1

  Allocation             inherit

  Read ahead sectors     auto

  - currently set to     256

  Block device           253:0



$ sudo vgdisplay bubba

  --- Volume group ---

  VG Name               bubba

  System ID

  Format                lvm2

  Metadata Areas        1

  Metadata Sequence No  2

  VG Access             read/write

  VG Status             resizable

  MAX LV                0

  Cur LV                1

  Open LV               0

  Max PV                0

  Cur PV                1

  Act PV                1

  VG Size               921.08 GiB

  PE Size               4.00 MiB

  Total PE              235797

  Alloc PE / Size       235797 / 921.08 GiB

  Free  PE / Size       0 / 0

  VG UUID               1AHmxk-we3d-86Ji-UxJ1-jO35-ViDm-swxwRT



$ sudo lvmdiskscan

  /dev/ram0          [      64.00 MiB]

  /dev/bubba/storage [     921.08 GiB]

  /dev/ram1          [      64.00 MiB]

  /dev/sda1          [     294.09 GiB]

  /dev/ram2          [      64.00 MiB]

  /dev/ram3          [      64.00 MiB]

  /dev/ram4          [      64.00 MiB]

  /dev/ram5          [      64.00 MiB]

  /dev/sda5          [       4.00 GiB]

  /dev/ram6          [      64.00 MiB]

  /dev/ram7          [      64.00 MiB]

  /dev/ram8          [      64.00 MiB]

  /dev/ram9          [      64.00 MiB]

  /dev/ram10         [      64.00 MiB]

  /dev/ram11         [      64.00 MiB]

  /dev/ram12         [      64.00 MiB]

  /dev/ram13         [      64.00 MiB]

  /dev/ram14         [      64.00 MiB]

  /dev/ram15         [      64.00 MiB]

  /dev/sdf1          [       9.32 GiB]

  /dev/sdf2          [     921.09 GiB] LVM physical volume

  /dev/sdf3          [       1.10 GiB]

  1 disk

  20 partitions

  0 LVM physical volume whole disks

  1 LVM physical volume

Ending with

$ sudo mount /dev/bubba/storage /mnt/usb

mount: wrong fs type, bad option, bad superblock on /dev/mapper/bubba-storage,

       missing codepage or helper program, or other error



       In some cases useful info is found in syslog - try

       dmesg | tail or so.

$ dmesg | tail -n 15

[ 1108.474708] sd 5:0:0:0: [sdf] tag#0 FAILED Result: hostbyte=DID_ERROR driverbyte=DRIVER_SENSE

[ 1108.474717] sd 5:0:0:0: [sdf] tag#0 Sense Key : Hardware Error [current]

[ 1108.474722] sd 5:0:0:0: [sdf] tag#0 Add. Sense: No additional sense information

[ 1108.474729] sd 5:0:0:0: [sdf] tag#0 CDB: Read(10) 28 00 01 2a 55 41 00 00 38 00

[ 1108.474734] blk_update_request: I/O error, dev sdf, sector 19551553

[ 1250.060801] sd 5:0:0:0: [sdf] tag#0 FAILED Result: hostbyte=DID_ERROR driverbyte=DRIVER_SENSE

[ 1250.060810] sd 5:0:0:0: [sdf] tag#0 Sense Key : Hardware Error [current]

[ 1250.060815] sd 5:0:0:0: [sdf] tag#0 Add. Sense: No additional sense information

[ 1250.060822] sd 5:0:0:0: [sdf] tag#0 CDB: Read(10) 28 00 01 2a 55 41 00 00 38 00

[ 1250.060827] blk_update_request: I/O error, dev sdf, sector 19551553

[ 1365.599683] sd 5:0:0:0: [sdf] tag#0 FAILED Result: hostbyte=DID_ERROR driverbyte=DRIVER_SENSE

[ 1365.599692] sd 5:0:0:0: [sdf] tag#0 Sense Key : Hardware Error [current]

[ 1365.599696] sd 5:0:0:0: [sdf] tag#0 Add. Sense: No additional sense information

[ 1365.599703] sd 5:0:0:0: [sdf] tag#0 CDB: Read(10) 28 00 01 2a 55 41 00 00 38 00

[ 1365.599709] blk_update_request: I/O error, dev sdf, sector 19551553

Am I right to conclude that I am unlikely to be able to mount or recover anything from this disk - or are there other things I could try?

Recovery

I was able to extract files from the LVM partition without mounting it by installing the recovery tool testdisk

Something like

sudo apt install testdisk

then

sudo testdisk /dev/bubba/storage

I accepted the default values suggested by testdisk (partition types etc).

This presents a textual user interface (TUI) which let me browse to the directory containing my data and copy it's contents to a drive on my desktop
PC.

It reported errors on a tiny percentage (e.g. 50 out of 50000) of files. The recovered files are readable mbox files - although there are files with identical duplicate contents. Possibly due to the way dovecot manages diskspace or maybe artefacts of the way testdisk recovers data, they don't look like the sort of cross linking I've ever seen with filesystem corruption.

I count this a success, I can now investigate how to merge the files I recovered with what I was able to restore from a backup and eliminate duplicates. I should be able to produce a set of files I can use with an email backend or frontend application.

I have no connection with testdisk other than as a new user of the tool.

Summary

• remove disk from dead system


• attach disk to freshly installed Ubuntu PC


• install LVM and attempt to mount filesystem


• install testdisk and use it to copy folders to Ubuntu PC

Funny I should find this answer having issues with exactly the same proprietary appliance!

In my case, the system wouldn't run any more. The disk's S.M.A.R.T status reported OK, but also reported a gradually increasing number of bad block, around 1250 of them.

The appliances backup tool never really worked and my manual backup was a little bit old.

Testdisk was capable of recovering some of the data, but failed a other data. After a lot of attempts, I did the following.

1. I bought a new USB hard disk, twice the size of the bad disk.


2. I created a 1TB empty partition on the new disk.


3. I used ddrescue to copy the old partition in several runs. This
   succeeded (with copying speeds slowing down to 100 B/s near the end) up to 99.99%. Use a map file to make the process incremental and restartable, especially on a device with hardware issues.


4. Since the partition was an physical volume of a LVM group, it couldn't be mounted alongside the original volume. The low level ddrecue action made it a duplicate PV. The easiest way was to shut down the system, unplug the old disk and restart the system. The alternative of meddling with LVM PV's, VG's, activation and mounting is quite cumbersome.


5. Check the file system. I suggest using esfsck with -z undo_file.

All in all, I think I got most of my data back. I haven't encountered corrupted files yet.