Please consider a donation to the Higher Intellect project. See https://preterhuman.net/donate.php or the Donate to Higher Intellect page for more info.

Backing up IRIX system

From Higher Intellect Wiki
Jump to navigation Jump to search

When I come to recover my data I'm getting "Magic String" errors. The backups are cpio format. I've two problems. I'm backing up over 250Gb's of data to DLT4000 and so I need a good backup system that won't screw up. I've been thinking of stopping spanning backups over multiple tapes. The other problem is how on earth do I recover my current backups since I've one tape I can't get back due to the magic string error.... I've dd'd it off to a file and tried all sorts on the file but no luck. What is everyone else using for backup on IRIX? Is it worth just backing up to another hard drive or something?


I have always used xfsdump. Apart from some early problems with the first O2 many years ago, it has always been my friend. (The troubles were related to not being able to dump across the network to a Solaris machine. Can't remember details.) Tape is superior to disk for backup, in that it allows for an easy method of rotation. I have a script which rotates a tape name list for me. It also uses chkconfig to decide if I want to do weekly backup (level 0) or daily backup (level 3). Each Monday it cycles through three weekly tapes, then a monthly one. Each night it does a daily dump. That way I have three monthly archives, three weekly archives, and a two weeks of daily. I am able to restore by day for two weeks, by week for four weeks, and by month for four months. Try that on a disk.

My recollection is that cpio is not going to label the tapes for you, or keep an inventory. The inventory is very handy for deciding exactly which tape you might need to refer to to retrieve the version of the file you have misplaced, deleted etc. Label the tapes physically, and via xfsdump, and it is hard to make mistakes. (Certainly not impossible, but you do have to try hard.) Xfsdump also has the concept of the dump level, which is vital. This enables incremental backups. You might need to restore from two or three different dumps to get back to where you want, but the saving in dump space and time is enormous.

You do have a lot of disk compared with the tape size. I still only use about 12 GB, and dump to a DDS-2 drive. Takes two tapes, and several hours. Presumably most of your 250 GB is static from week to week, and could be covered in a single archive session. Dump is all at level 0, then do subsequent weekly dumps at level 3, and daily at level 5. I like to leave a spare level, in case I need to squeeze something extra in...

As for recovering from the errors on the existing tapes... My recollection is that backup used to be a front end to tar. At some stage a while back this switched to being a front end for cpio, but either way it is just a raw dump of the disk contents onto the tape. If you seek a particular file, it is probably achievable to recover it from the tape, as raw data. If you want the entire disk system, that would seem like a large amount of effort.


We do a weekly level-0 dump and incrementals nightly to a pool of Magstar tapes in a 3494-library. We used to do dumps to another machine with this monstrosity

#!/usr/local/bin/perl5 -w
#
# $Header: /afs/.pdc.kth.se/sgi_63/usr/pdc/bin/RCS/daily.sysdump.pl,v 1.7 2001/08/01 14:28:53 pek Exp $

# Dump xfs filesystems

use English;
use POSIX;
use FileHandle;

$INPUT_RECORD_SEPARATOR = "\n";
autoflush STDERR;
autoflush STDOUT;

# Dump these filesystems (list the mount point names here without leading
# / except for the root fs which should be listed as "/")
my (@fslist) = ("/", "dmf/journals", "dmf/home");

# Specify which day to perform lvl 0 bup. First char must be capitalized.
my($lvl0day)    = "Thursday";

# Destination dirbase
my($bkdirbase)  = "/hsm/backup/innesi";

my($buphost)    = "foo.pdc.kth.se";

# Commands
my ($dump)      = "/usr/sbin/xfsdump";
my ($rsh)       = "/usr/athena/bin/rsh";
my ($date)      = "/sbin/date";
my ($tail)      = "/usr/bin/tail";
my ($wc)        = "/sbin/wc";
my ($df)        = "/bin/df";
my ($dmmigrate) = "/usr/dmf/etc/dmmigrate";

# Commands on remote host
my ($r_ls)      = "/bin/ls";
my ($r_mkdir)   = "/bin/mkdir";
my ($r_chmod)   = "/bin/chmod";
my ($r_awk)     = "/bin/awk";
my ($r_sort)    = "/bin/sort";
my ($r_tail)    = "/bin/tail";
my ($r_dd)      = "/usr/bin/dd";
my ($r_dsmmigrate) = "/usr/bin/dsmmigrate";

my ($dstr)      = `date +%A`;
my ($buplvl)    = 5;

chomp $dstr; # Eat trailing newline
if ($dstr eq $lvl0day) {
    print "Level 0 bup.\n";
    $buplvl = 0;
}

# Arguments : remote host (string)
#             command (string)
# Executes command on remote host using rsh
sub rexec ($$) {
    my ($host) = $ARG[0];
    my ($cmd)  = $ARG[1];
    my ($res);
    
#    print " $rsh $host $cmd\n";
    $res = `$rsh $host $cmd`;

    return $res;
}

##
# Perform backups of dmf filesystems.
# Full backup every $lvl0day, then a level five
# backup other days
#

print "### [" . localtime(time) . "] $PROGRAM_NAME: invoked\n";

foreach $fs (@fslist) {
    my ($lvl) = $buplvl;
    my ($execstr, $res);
    my ($zerobupdate) = "00000000";
    my ($fn, $dir, $bkfile, $fsdev);

    # Get the device name (which is what xfsdump wants)
    $res = `$df -n /$fs`;
    ($fsdev) = $res =~ /^(\S+).+$/;
    if ($fs eq "/") {
        $fs = "root";
    }
    # Change / to . in dump filename
    $fs =~ s/\//\./;
    # If incremental bup, find the lvl 0 to base it on
    if ($lvl != 0) {
        # This will (should!) extract the last lvl 0 date string
        # as a 8 char numeric (eg 20001016).
        # Escaping nightmare (defeat three levels of interpolation).
        $execstr = "$r_ls $bkdirbase/$fs.0/$fs.\\*.[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9] \\| $r_awk \\\'\{ NUM=split\\\(\\\$1,A,\\\"\\\.\\\"\\\)\\; print A\\\[NUM\\\],\\\$1\}\\\' \\| $r_sort \\| $r_tail -1 \\| $r_awk \\\'\{ print \\\$1\}\\\'";
        $zerobupdate = rexec($buphost, $execstr);
        chomp $zerobupdate;
    }
    if ($zerobupdate eq "") { # No prior lvl 0, revert to lvl 0 now
        $lvl = 0;
        $zerobupdate = "00000000";
        print "## Warning: Reverting to zero bup on $fs\n";
    }
    $fn = "$fs.$lvl";
    $dir= "$bkdirbase/$fn";
    rexec($buphost, "$r_mkdir -p $dir");
    rexec($buphost, "$r_chmod 700 $dir");
    if ($CHILD_ERROR != 0) {
        print "\n\nFATAL ERROR! Failed to create remote destination directory. Aborting.\n\n";
        exit 1;
    }
    $dstr = `$date +%Y%m%d`; chomp $dstr;
    $bkfile = "$dir/$fs.$zerobupdate.$dstr";
    $res = rexec($buphost, "$r_ls $bkfile");
    if ($res ne "") {
        print "### [" . localtime(time) . "] $PROGRAM_NAME: backup file exists, skipping backup of $fs lvl $lvl to $bkfile...\n";
    }
    else { # Perform dump
        print "### [" . localtime(time) . "] $PROGRAM_NAME: backing up $fs lvl $lvl to $bkfile...\n";
        $execstr = "$dump -l $lvl -v silent -F - $fsdev \| $rsh $buphost $r_dd of=$bkfile bs=64k";
#       print "$execstr\n";
        $res = `$execstr`;
#       print "  Dump result : \"$res\"\n";
    }
    # Check that the bup file was created
    $res = rexec($buphost, "ls $bkfile");
    chomp $res;
    $dstr = `$date`; chomp $dstr;
    if (!$res) {
        print "### [$dstr] $PROGRAM_NAME: Failed backing up $fs lvl $lvl to $bkfile - bup is missing\n";
    }
}
$dstr = `$date`; chomp $dstr;
print "### [$dstr] $PROGRAM_NAME: done\n";
exit 0;

I use 2 of these ACARD (I believe these cost arround $80 each) connected to 250GB WD2500JB EIDE drives. Each IDE drive has an ACARD adapter to connect to it's own Ultra2scsi buss connected to a Qlogic card in a pci card cage. With 2 parallel drives set up that way, I have arround 500GB that is readable/writable at arround 65MB/Sec average on large reads. These 2 drives have been connected to my Octane using XLV stripe (used as NFS server) for about 9 months now without any failures. It would be more reliable if it wasn't striped, but the performance would be less also, and I use the Octane for editing large audio files also. So, I beat on the drive a bit. I don't know if this is the case or not now, but I have been through several reliability problems with Maxtor drives, so I stick to Western Digital drives.

I use this space for backups from other systems, and all of my old backup tapes were extracted to this also. I also use it to serve my home directories (which is about 20GB) and I back up that on another machine from time to time.

When I fill it up with just backups, I plan on buying another pair of drives. It is a whole lot more usefull to have all of the backups online at a moments notice.

The whole price for the 2 drives with ACARD adapters was less than $1000. I find the convenience is well worth it. When the drives fill up and I need to replace them (and store these in a safe), I will pay less than $1/GB. Last I checked (which I admit has been years) DLTIV tapes were arround that price level.

Another advantage of using hard drives on a system is you can use much better compression than you could get on built in compression, and application specific compression schemes. I converted the old unix compress format tar archives to bzip2 saving a lot of space. I keep root xfsdumps in gzip format because it is faster, and about as efficient as bzip2, and most every SGI installs have gzip native. Just use a well used format that you would expect to find used in 10 years.

Another inexpensive way to do large backups is to use a cheap but reliable Linux box on the network. I would not suggest doing any striping or raid because there are so many ways to do it, and some of them are not reliable. Start with a 300GB hard drive, and add another one when you run out of space. You should be able to get started for under $600 in hardware, and from then on less then $1/GB. Make sure you use a reliable filesystem on your drives also.