Oracle Linux 6.4误删VG之恢复过程

一、项目背景描述
1、OSS现网测试数据库因大量小事物频繁提交运行非常缓慢。经分析为DS3950存储所在磁盘I/O存在瓶颈，大量等待事件，性能受限。另外，开发同事没有优化意识，没将小事物做成批量提交方式。
2、在DS3950上，9块600G硬盘(8块+1块热备)做了RAID5阵列，lun01，lun02，lun03，lun04，均为200G，映射给OSS数据库服务器使用。
3、在操作系统上，lun01，lun02构成了vg_ossdb卷组，vg_ossdb下只有一个LV－－lvoradata挂载在/oradata上。最近因数据增长过快，又将lun03，lun04以vgextend方式扩到了vg_ossdb卷组中去，但尚未扩大lvoradata。
4、在数据库上，Oracle软件安装在本地磁盘/oracle上，数据库安装在/oradata上。


二、项目改造计划及步骤
1、停掉数据库，将/oradata目录全备份到另一台备用PC机器上。
2、由于DS3950存储空闲空很大，可修改其阵列级别从RAID5变为RAID10。
3、由于lun03，lun04尚未使用，领导要求先从vg_ossdb中移除lun03，lun04，并在存储上unmap掉，便于阵列级别变更。
4、由DBA优化SQL，将小事物尽量做到批量提交方式。


三、系统环境及数据版本说明
[root@ol64 /]# cat /etc/issue
Oracle Linux Server release 6.4
Kernel \r on an \m


[root@ol64 /]# uname -a
Linux ol64.com 2.6.39-400.17.1.el6uek.x86_64 #1 SMP Fri Feb 22 18:16:18 PST 2013 x86_64 x86_64 x86_64 GNU/Linux


SQL> select * from v$version;
BANNER
--------------------------------------------------------------------------------
Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
PL/SQL Release 11.2.0.4.0 - Production
CORE 11.2.0.4.0 Production
TNS for Linux: Version 11.2.0.4.0 - Production
NLSRTL Version 11.2.0.4.0 - Production


四、改造过程中因误用vgremove将vg_ossdb卷组删除，而其本意是想用vgreduce移除/dev/sdd，/dev/sde的。
[root@ol64 /]# umount /oradata/ #卸载文件系统
[root@ol64 /]# vgchange -an /dev/vg_ossdb #将卷组置于非激活状态
0 logical volume(s) in volume group "vg_ossdb" now active
[root@ol64 /]# vgremove vg_ossdb /dev/sdb /dev/sdc /dev/sdd /dev/sde #误用vgremove命令删除了vg_ossdb
Do you really want to remove volume group "vg_ossdb" containing 1 logical volumes? [y/n]: y
Do you really want to remove active logical volume lvoradata? [y/n]: y
Logical volume "lvoradata" successfully removed
Volume group "vg_ossdb" successfully removed
Volume group "sdb" not found
Volume group "sdc" not found
Volume group "sdd" not found
Volume group "sde" not found
##################################################################
正确的操作应该是用vgreduce命令从vg_ossdb中移除/dev/sdd和/dev/sde
[root@ol64 /]# vgreduce vg_ossdb /dev/sdd
Removed "/dev/sdd" from volume group "vg_ossdb"
[root@ol64 /]# vgreduce vg_ossdb /dev/sde
Removed "/dev/sde" from volume group "vg_ossdb"
#################################################################
再用pvremove命令移除/dev/sdd和/dev/sde
[root@ol64 ~]# pvremove /dev/sdd
Labels on physical volume "/dev/sdd" successfully wiped
[root@ol64 ~]# pvremove /dev/sde
Labels on physical volume "/dev/sde" successfully wiped


[root@ol64 ~]# pvdisplay #发现/dev/sdb和/dev/sdc所在VG Name为空，冒汗ing.
--- Physical volume ---
PV Name /dev/sda2
VG Name vg_ol64
PV Size 199.51 GiB / not usable 3.00 MiB
Allocatable yes
PE Size 4.00 MiB
Total PE 51074
Free PE 33660
Allocated PE 17414
PV UUID 0dyB8L-p7ZM-Mkcw-76ae-DXPh-U6zg-9kIQ8z


"/dev/sdb" is a new physical volume of "200.00 GiB"
--- NEW Physical volume ---
PV Name /dev/sdb
VG Name
PV Size 200.00 GiB
Allocatable NO
PE Size 0
Total PE 0
Free PE 0
Allocated PE 0
PV UUID Ui9wea-II1q-KOx0-96pA-4epf-9hlc-4NFDJF


"/dev/sdc" is a new physical volume of "200.00 GiB"
--- NEW Physical volume ---
PV Name /dev/sdc
VG Name
PV Size 200.00 GiB
Allocatable NO
PE Size 0
Total PE 0
Free PE 0
Allocated PE 0
PV UUID 4VKCJ9-G6kL-QJgg-Titf-UNA8-d3QZ-ZTES3P


[root@ol64 ~]# vgscan #vgscan时找不到vg_ossdb信息，心跳加速，冒汗ing.
Reading all physical volumes. This may take a while...
Found volume group "vg_ol64" using metadata type lvm2


[root@ol64 ~]# lvscan #lvscan时找不到lvoradata信息，大脑一片空白，快要晕倒.
ACTIVE '/dev/vg_ol64/lvopt' [10.01 GiB] inherit
ACTIVE '/dev/vg_ol64/lvroot' [40.01 GiB] inherit
ACTIVE '/dev/vg_ol64/lvswap' [8.00 GiB] inherit
ACTIVE '/dev/vg_ol64/lvhome' [10.01 GiB] inherit


我怎么会用vgremove命令？？完了！！！vg_ossdb被我误删除了。赶紧google和baidu上搜索vg被删除后如何恢复，可压根找不到相关资料。冷静下来后开始思考/etc/lvm/目录下各文件用途。

五、VG恢复思路
1、在/etc/lvm/存放着LVM的配置、归档、备份等信息。
[root@ol64 ~]# ls -l /etc/lvm
total 52
drwx------. 2 root root 4096 Nov 18 08:30 archive
drwx------. 2 root root 4096 Nov 18 08:30 backup
drwx------. 2 root root 4096 Feb 24 2013 cache
-rw-r--r--. 1 root root 37554 Feb 24 2013 lvm.conf


2、在/etc/lvm/backup/下存放着vg的备份信息，但我在操作vg_ossdb之前没有备份其信息到其它目录。
[root@ol64 lvm]# ls /etc/lvm/backup/
total 4
-rw-------. 1 root root 2575 Nov 12 09:09 vg_ol64


3、在 /etc/lvm/archive/下存放着VG及LV调整前的归档信息，即VG变更或LV变更前均会备份当前信息。
[root@ol64 ~]# ls -l /etc/lvm/archive/
total 32
-rw-------. 1 root root 2576 Nov 12 09:09 vg_ol64_00000-1722993391.vg
-rw-------. 1 root root 883 Nov 18 08:03 vg_ossdb_00000-2033719300.vg
-rw-------. 1 root root 883 Nov 18 08:04 vg_ossdb_00001-1635801039.vg
-rw-------. 1 root root 1122 Nov 18 08:05 vg_ossdb_00002-1283186973.vg
-rw-------. 1 root root 883 Nov 18 08:05 vg_ossdb_00003-1708919759.vg
-rw-------. 1 root root 1139 Nov 18 08:05 vg_ossdb_00004-18964421.vg
-rw-------. 1 root root 1728 Nov 18 08:30 vg_ossdb_00005-533258090.vg
-rw-------. 1 root root 1131 Nov 18 08:30 vg_ossdb_00006-1987723911.vg
备注：使用vgcreate、vgreduce、vgremove、lvcreate、lvreduce、lvremove等命令时均会生成新的归档信息


4、利用vgcfgrestore命令恢复误删除的VG
[root@ol64 archive]# vgcfgrestore -f /etc/lvm/archive/vg_ossdb_00001-1635801039.vg vg_ossdb
Restored volume group vg_ossdb
[root@ol64 archive]# pvdisplay
--- Physical volume ---
PV Name /dev/sdb
VG Name vg_ossdb
PV Size 200.00 GiB / not usable 4.00 MiB
Allocatable yes
PE Size 4.00 MiB
Total PE 51199
Free PE 51199
Allocated PE 0
PV UUID Ui9wea-II1q-KOx0-96pA-4epf-9hlc-4NFDJF


--- Physical volume ---
PV Name /dev/sda2
VG Name vg_ol64
PV Size 199.51 GiB / not usable 3.00 MiB
Allocatable yes
PE Size 4.00 MiB
Total PE 51074
Free PE 33660
Allocated PE 17414
PV UUID 0dyB8L-p7ZM-Mkcw-76ae-DXPh-U6zg-9kIQ8z


"/dev/sdc" is a new physical volume of "200.00 GiB"
--- NEW Physical volume ---
PV Name /dev/sdc
VG Name
PV Size 200.00 GiB
Allocatable NO
PE Size 0
Total PE 0
Free PE 0
Allocated PE 0
PV UUID 4VKCJ9-G6kL-QJgg-Titf-UNA8-d3QZ-ZTES3P
上述操作发现只有/dev/sdb在vg_ossdb卷组中了，而/dev/sdc仍然不在vg_ossdb卷组中。这说明/etc/lvm/archive/vg_ossdb_00001-1635801039.vg归档文件较旧，还没有包括/dev/sdc在vg_ossdb卷组中，继续使用下一个归档文件恢复，直到/dev/sdb、/dev/sdc均在vg_ossdb卷组中，并且卷组中包含的LV个数正确。


[root@ol64 archive]# vgcfgrestore -f /etc/lvm/archive/vg_ossdb_00005-533258090.vg vg_ossdb
Restored volume group vg_ossdb
[root@ol64 archive]# vgchange -ay /dev/vg_ossdb
1 logical volume(s) in volume group "vg_ossdb" now active
[root@ol64 archive]# lvscan
ACTIVE '/dev/vg_ossdb/lvoradata' [200.00 GiB] inherit
ACTIVE '/dev/vg_ol64/lvopt' [10.01 GiB] inherit
ACTIVE '/dev/vg_ol64/lvroot' [40.01 GiB] inherit
ACTIVE '/dev/vg_ol64/lvswap' [8.00 GiB] inherit
ACTIVE '/dev/vg_ol64/lvhome' [10.01 GiB] inherit


[root@ol64 archive]# mount /dev/vg_ossdb/lvoradata /oradata/
[root@ol64 archive]# ls -l /oradata/ossdb/
total 1698340
-rwxrwxr-x. 1 Oracle oinstall 9748480 Nov 18 08:29 control01.ctl
-rwxrwxr-x. 1 oracle oinstall 1073742336 Nov 18 08:11 redo01.log
-rwxrwxr-x. 1 oracle oinstall 1073742336 Nov 18 08:11 redo02.log
-rwxrwxr-x. 1 oracle oinstall 1073742336 Nov 18 08:29 redo03.log
-rwxrwxr-x. 1 oracle oinstall 2147516416 Nov 18 08:29 sysaux01.dbf
-rwxrwxr-x. 1 oracle oinstall 2147516416 Nov 18 08:29 system01.dbf
-rwxrwxr-x. 1 oracle oinstall 8388640768 Nov 18 06:38 temp01.dbf
-rwxrwxr-x. 1 oracle oinstall 17179901952 Nov 18 08:29 undotbs01.dbf
-rwxrwxr-x. 1 oracle oinstall 17179901952 Nov 18 08:29 users01.dbf
//省略....
启动数据库，一切正常。

总结：在对VG做变更时，切记先用vgcfgbackup备份好VG的信息，避免意外。
[root@ol64 /]# vgcfgbackup -f /home/vg_ossdb.backup vg_ossdb
尽管本次操作并未造成损失，但惊醒了我。作为系统管理员，每一步操作都必须深思熟虑。切记，操作权限越大，承担责任也越大！

作者：firefly222