Mirroring the Boot disk with Disksuite on Glued Solaris Boxes

Notes on Mirroring the boot disk with DiskSuite on Glued Solaris Systems

The instructions assume we are going to mirror the system/boot disk (e.g. filesystems root (/), /usr, /var, /usr/vice/cache, and swap). It is fairly straight forward to adapt these instructions to mirror other filesystems (either separately or in conjunction with the above), and to deal with systems with more than 2 disks.

The instructions assume that we are enabling mirroring on a newly installed, non-production machine. These restrictions are not strictly required, but obviously enabling mirroring of the system disk on a system already in production runs some risks. The system WILL need to be rebooted at least once when mirroring the system disk; if a non-system disk is being mirrored you can probably get away with just umounting and remounting the partitions being mirrored at the appropriate times. NOTE: all filesystems being mirrored must be mounted using the single-ply metadisk mirror before attaching the second submirror to the mirror metadevice; if new data written to the filesystem will only be written to one submirror, and disksuite will get very upset upon attempt to reboot (as the two submirrors are listed as being in sync but are not); if you do this with root will probably not even be able to get to single-user mode.

1. Do a "normal" install of Glue onto the first of the two internal disks. The only thing special compared to any other Glue install is that you need to have an idea of where the metadb DB replicas are to go and make sure the partition table has room for them. See this section for more information about locating the DB replicas.
2. After Glue is installed, the initial rdist done, and the system is up and healthy, create the partition table on the second disk. I am not sure if this is strictly required, but if not it is strongly advised that all slices to be mirrored have the same slice number and be identical in the partition table (same starting and ending cylinder number, tags, flags, etc.). Indeed, since typically you will be mirroring all defined slices on the two disks, it is usually easiest to have identical partition tables, which you can do by copying the table between DISK1 and DISK2:
   prtvtoc /dev/rdsk/DISK1 | fmthard -s - /dev/rdsk/DISK2
3. Perform any preliminary steps required. In particular, for Solaris 2.7, a number of devices and files need to be created. You may also wish to review some of the references.
4. Decide on the location of the DB replicas (see here for help), and install the DB replicas. E.g., if you are installing to slice 7 of the two disks c0t0d0 and c0t1d0, you would use
   metadb -f -c3 -a /dev/dsk/c0t0d0s7
metadb -c3 -a /dev/dsk/c0t1d0s7
5. Decide on a metadevice naming scheme.
6. Create the metadevices for the two submirrors and for a single-ply mirror device for each partition to be mirrored. I find this easiest to do by changing the md.tab file (found in /etc/opt/SUNWmd on Solaris 2.7 systems, and /etc/lvm on Solaris 8 and 9 systems); this has the advantage of allowing comments and a more lasting record of what was done (especially if save a copy in the glue config tree).
   • For each submirror to be defined, there should be a line containing the metadevice name for the submirror, the number of stripes (should be 1), the number of slices making up the stripe (again, should be 1), and the physical device comprising the submirror. E.g., assuming the disks in the previous example and the Physics standard naming scheme, the submirrors for slice 5 (/var) would be defined with:
     /dev/md/dsk/d60 1 1 /dev/dsk/c0t0d0s5
/dev/md/dsk/d61 1 1 /dev/dsk/c0t1d0s5
   • For each mirror to be defined, define it as a single way mirror. I am assuming that the slice on disk1 contains a valid filesystem whose contents you wish to preserve (as would be the case when mirroring the existing /, /usr, and other system filesystems or swap). The mirror definition should consist of the metadevice name for the mirror, followed by the -m option and the name of the metadevice corresponding to the disk/slice that currently has data on it. The only submirror that should be mentioned in the mirror metadevice definitions in the md.tab file should be the one refering to the currently mounted physical device. Do not mention the other submirror in the mirror definition or there can be data loss on the currently mounted slice . Assuming we installed the system on the first disk from the previous examples, the mirror metadevice for /var (slice 5) would be defined with:
     /dev/md/dsk/d6 -m /dev/md/dsk/d60
     Note that we only mention one of the submirrors (the one which corresponds to the disk with real data on it).
   • The md.tab file does not actually define the metadevices, but stores options to be used by the metainit command when called with insufficient arguments, much like /etc/vfstab can be used to store options for mount. Because of the standard partitioning scheme in use by glue, these md.tab files are pretty standard, with usually just the names of the physicsal devices for the two disks needing to be changed. Review your md.tab file and make sure it is correct. In particular, make sure the mirror metadevice definitions refer to the submirror referring to the slice which has the good filesystem on it.
   • For each already existing filesystem that you want to mirror and have previously created a submirror metadevice for, issue the command
     metainit -f SUBMIRROR_NAME
     . E.g, using the same setup as previous examples, the current /var slice would have its submirror device initialized, retaining the current contents of /var, with the command:
     metainit -f d60
     The -f is required because the physical device referenced in the md.tab file for submirror d60 is currently mounted. Similar commands should be issued for each slice to mirror.
   • For each slice to be mirrored, initialize the submirror on the second (not currently used except for DB replicas) disk with the command
     metainit SUBMIRROR_NAME
     . Continuing along with the previous examples, the currently unused slice 5 of disk 2 would be initialized as a submirror (which will eventually mirror /var) with the command:
     metainit d61
     Note that no -f flag is used in this case. Similar commands should be issued for each slice to mirror.
   • For each slice to be mirrored, initialize the mirror metadevice with the single submirror refering to currently active/mounted physical device. This is done with the command
     metainit MIRROR_NAME
     Following with the previous examples, the mirror for /var would be initialized with
     metainit d6
     This will create a mirror metadevice with the single submirror d60 (because that is how d6 was defined in the md.tab file). Similar commands should be issued for each slice to mirror. Note: it is important that the single submirror be the submirror referring to the currently active/mounted disk/slice, otherwise data loss may occur. You may want to review your md.tab file again before issuing the commands to create the mirrors.
   • You can use the metastat command to see that all the submirrors and single-ply mirrors were set up correctly. Everything should be indicating a state of "Okay". Mirror metadevices should list a single submirror corresponding to the currently mounted device (if the other submirror is listed, correct it NOW to avoid data loss), and submirrors should list the physical device they are associated with and the mirror metadevice they are attached to. The unattached submirrors (i.e. those referring to slices not currently mounted) will show up as "Concat/Stripe" at this time. If any thing looks wrong or a mistake was made, FIX IT NOW to avoid data loss. You should be able to metadetach the incorrect mirror, and possibly metaclear the problematic mirror and submirrors, edit the md.tab file to correct things, and recreate the affected submirrors and mirror.
   • At this point, single-ply metadevice mirrors exist which are ready to be mounted. However, they are not mounted, the physical devices are. It is important that the mounted/active partitions are remounted with the metadevice devices; if you are mirroring filesystems other than those on the system disk, you may be able to just umount, edit vfstab, and remount. But most likely a reboot will be required after making the changes below.
     • If the root (/) filesystem is being mirrored, you MUST run the metaroot command. Usage is
       metaroot ROOT_MIRROR_METADEVICE
       . This command edits /etc/vfstab entry for root, but more importantly it modifies /etc/system so that the kernel knows where the root filesystem is. Failure to run this command might result in an inability to boot the system. Using the standard Physics naming scheme, the root device on slice 0 will get mirrored to the metadevice d1, and so would want to issue the command
       metaroot d1
       After running this, the /etc/vfstab file should list /dev/md/dsk/d1 as the device to mount on /. Also, the text file /etc/system should contain a new stanza with metadisk related stuff in it, something like
       * Begin MDD root info (do not edit)
forceload: misc/md_trans
forceload: misc/md_raid
forceload: misc/md_hotspares
forceload: misc/md_sp
forceload: misc/md_stripe
forceload: misc/md_mirror
forceload: drv/pcisch
forceload: drv/mpt
forceload: drv/sd
rootdev:/pseudo/md@0:0,1,blk
* End MDD root info (do not edit)

       At this point, you will likely see "WARNINGS" upon reboot about the forceloading of certain metadisk subsets (e.g. trans, raid, and hotspares) failing. Sun says these are harmless and can be ignored.
     • For any other filesystems being mirrored, you need to manually edit /etc/vfstab to have the filesystem mounted using the mirror metadevice. Yo can use the root partition as modified by metaroot as a guide. Following our previous examples, the line for /var would be so,ething like:
       /dev/md/dsk/d6 /dev/md/rdsk/d6 /var ufs 1 no logging
       Repeat for all filesystems to be mirrored.
     • At this point, we are ready to remount all the filesystems to be mirrored using the mirror metadevices. If you are only mirroring non-system partitions which you can dismount temporarily, you may be able to get away with just a dismount and remount. If you are mirroring root (/), /usr, or other critical system filesystems, you must reboot before proceeding further. The filesystems MUST be remounted using the mirror metadevices before attaching the second submirror to the mirror device, or serious problems (including inability to boot even to single user mode) can occur.
     • If you are mirroring the root disk, you may want to halt the system instead of rebooting it, and at the openboot prompt find the name for the second disk (the one which is going to be made into a mirror of the first), and create a device alias for it at the openboot prompt. E.g.,
       devalias altboot /pci@1f,4000/scsi@3/disk@1,0
       This is useful because openboot does not know about disksuite mirroring, and if the primary disk fails, when you reboot the disk you will need to tell openboot to boot from the second hard disk. If not done now, can be done later as well.
7. Issue the mount command and make sure all filesystems to be mirrored are listed as mounted under the metadisk mirror device, not the physical device. Do not proceed any further unless this is the case, as attaching the second submirror while the physical device for the first submirror is mounted will result in metadisk syncing the disks, but any writes to the file system go to one disk only, thereby breaking the synchronization. What is worse, is that metadisk thinks they are synchronized, and will fail hard. You might not even be able to get into single user on next reboot. Make sure all filesystems to be mirrored are mounted with the metadisk mirror device before attaching the other submirror. This generally means a reboot after editting vfstab as indicated in previous step.
8. At this point, the filesystems to be mirrored are using the mirrored metadevices, but we are not really mirroring yet because the mirrors all consist of only a single submirror. We can now attach the second submirror to each of the mirrors with a command like:
   metattach MIRROR SUBMIRROR2
   This causes SUBMIRROR2 to be attached to MIRROR, and will cause the newly attached submirror to be synchronized with the previous submirror, copying information from the old submirror to the new one. For the /var filesystem in our example, metattach d6 d61. Repeat this for all filesystems to mirror. Because the synchronization can take a while and keep the disks quite busy, you may want to allow each slice to finish synchronizing before starting the next on a production system (see the metastat command below). On non-production systems I usually get let them all sync in parallel.
9. You can run the metastat to see how things are going. All the mirrors you recently attached should be syncing up. This can take some time. The new submirrors should now show up as "Submirrors" and not as "Concat/Stripes". States for the newly attached submirrors will be "Resyncing" but should eventually change to OK when synchronized.
10. If the swap device was mirrored, you will need to change the crash dump device to the swap metadevice. This can be done with the command
    dumpadm -d `swap -l | tail -1 | awk '{print $1}'`
    It should report the new dump device as the swap mirror metadevice, e.g. /dev/md/desk/d2. You can verify this at a later point by issuing the command dumpadm without any arguments. Failure to do this may cause serious problems if a crash dump ever gets written, and may make system unbootable
11. If you are mirroring the boot disk (e.g. root (/), you need to make the mirror disk bootable. On SPARC systems, this can be done with the command
    installboot /usr/platform/`uname -i`/lib/fs/ufs/bootblk /dev/rdsk/ROOT2
    where ROOT2 is the root slice of the mirrored disk. E.g. c0t1d0s0.
12. If you are mirroring the boot disk you will probably want to create an alias for the alternate boot disk. Openboot does not know about the software mirrorings, and will only boot from physical disks. Normally it boots from the first disk, but it is helpful to define an alias for the mirrored disk so you can easily boot off of that disk if the primary disk fails. This can be done either from the shell or from openboot (as was mentioned when we suggested that you might want to halt instead of reboot after modifying the vfstab file.) If you did not do it then, you can do it now, as follows:
    1. Determine the physical device path of the mirror disk by examining the symbolic link for the device under /dev/dsk and extracting everything following the /devices part. E.g., if the mirrored root disk is c0t1d0s0, issue the command ls -l /dev/dsk/c0t1d0s0. This should return something like
       benfranklin:~# ls -l /dev/dsk/c1t1d0s0
lrwxrwxrwx 1 root root 70 May 4 2004 /dev/dsk/c1t1d0s0 -> ../../devices/pci@8,600000/SUNW,qlc@4/fp@0,0/ssd@w21000004cf8a6403,0:a

       so what you want is pci@8,600000/SUNW,qlc@4/fp@0,0/ssd@w21000004cf8a6403,0:a.
    2. If you are doing this from a system that is up in unix, first check for any previously existing device aliases with the command eeprom nvramrc. If it returns something like data not available, you can just go ahead. Otherwise you need to determine if the previous definitions should be kept or not. If you want to wipe them out, just proceed as if there were no previous definitions, otherwise you need to cut and paste the previous definitions into the next command.
    3. Create a device alias with an easily remembered name (e.g. "mirror") for this disk, and then issue the following commands from the unix prompt
       eeprom "nvramrc=OLDDEFS devalias mirror PHYSPATH
       where OLDDEFS are any old definitions in the NVRAM that you wish to keep, and PHYSPATH is the physical path to the secondary boot disk. For the above example, assuming no previous definitions exist (or want to keep), you would have:
       eeprom "nvramrc=devalias mirror /pci@8,600000/SUNW,qlc@4/fp@0,0/ssd@w21000004cf8a6403,0:a"
       You can, of course choose another name than "mirror" for the alias if desired. Alternatively, you can do this with the corresponding command at the openboot prompt, e.g.
       nvalias mirror /pci@8,600000/SUNW,qlc@4/fp@0,0/ssd@w21000004cf8a6403,0:a
       (Actually, you can save some typing by running the show-disks command, and selecting the device you want, and then typing ^Y (control-Y) instead of the long device name in the nvalias command).
    4. Make sure the NVRAM aliases are read at boot up. From Unix, use the command
       eeprom "use-nvram?=true"
       or from openboot prompt the command
       setenv use-nvramrc? true
    5. You can verify the settings with the commands
       eeprom "nvramrc"
       from Unix of
       devalias
       from openboot prompt.
13. If you are mirroring the boot disk you should consider configuring the system to ignore the standard behavior and boot anyway even if only half of the DB replicas are available. I believe the default behavior (to only allow booting if more than half of the DB replicas are present) is to prevent some pathological cases wherein one removes one of a mirrored pair of drives, boots the system, makes some changes, halts system, puts back the removed drive and removes the old drive, boots system, and makes some other changes to the filesystem. When booted then with both drives, the system will have 2 disks with different contents each claiming to hold the "true" data. It should be safe to disable that safeguard as long as you don't engage in the wierd behavior that leads to these pathological behaviors. You can disable it by adding the line
    set md:mirrored_root_flag=1
    in the file /etc/system. If you are primarily interested in a system that stays up as much as possible, and will reboot successfully if it goes down after a disk failure, you want this option set.