Akamit Blog

Enterprise sysadmin's blog

  • You are here: 
  • Home
  • Tape Libraries

Connecting SL500 Library robot to emulex HBA port

Posted on October 25th, 2010

I was managed to setup netbackup server using Sun Enterprise 5120 and SL500 library. The problem was that the robot connection became broken after server reboot and I have to power cycle the library to get the robot appear again. After doing some investigation it was revealed that the robot connection is unstable due to automatic configuration of FC link with Emulex adapter port.
The port instance the robot connected to is 2. We configure the port to force loop topology and 2Gb/s port speed by specifying parameters in emulex driver configuration file /kernel/drv/emlxs.conf

emlxs2-topology=4;
emlxs2-link-speed=2

After reboot we got library connected and not loosing link.

Solaris version:

SunOS backup-machine 5.10 Generic_139555-08 sun4v sparc SUNW,SPARC-Enterprise-T5120

and FC port

HBA Port WWN: 10000000c978d630
        OS Device Name: /dev/cfg/c4
        Manufacturer: Emulex
        Model: LPe11002-S
        Firmware Version: 2.80a4 (Z3F2.80A4)
        FCode/BIOS Version: Boot:5.03a0 Fcode:3.01a1
        Serial Number: 0999BG0-08350000K2
        Driver Name: emlxs
        Driver Version: 2.31t (2009.04.05.14.10)
        Type: L-port
        State: online
        Supported Speeds: 1Gb 2Gb 4Gb 
        Current Speed: 2Gb 
        Node WWN: 20000000c978d630

Filed under Enterprise Backup, Solaris, Sun Hardware, Tape Libraries | No Comments »

tape operation: Netbackup runs out of tapes.

Posted on September 2nd, 2010

Recently I’ve faced with the problem that netbackup system runs out of tapes. To find out what happened, we used bpmedialist report.

To view summary report of media usage:

bpmedialist -summary
 
ACTIVE   FULL  SUSPENDED  FROZEN  IMPORTED
   39    155        0       51        0
 
Number of NON-ACTIVE media that:
       5 - are non-active and not written yet
            CLNU06 (FROZEN)
            CP9032 (FROZEN)
            FJQ884 (FROZEN)
            KR8006 (FROZEN)
            QXN569 (FROZEN)
      44 - are currently EXPIRED
            QXN523 expired 04/14/2007 00:00 (FROZEN)
            --- SKIPPED OUTPUT ---
            MF9228 expired 08/13/2010 08:00 (FROZEN)
      71 - will expire within 1 week
            KR8009 expires 09/02/2010 13:00
            -- SKIPPED OUTPUT --
            KI1998 expires 09/09/2010 02:00
       1 - will expire between 1 and 2 weeks
            QXN614 expires 09/12/2010 09:00
      85 - will expire in greater than three months
            CP9017 expires INFINITY
            -- SKIPPED OUTPUT --
            QXN611 expires INFINITY      
 
Number of ACTIVE media that, as of now:
      22 - will expire within 1 week
            QXN590 expires 09/06/2010 02:00
             -- SKIPPED OUTPUT --
            QXN559 expires 09/09/2010 10:06
       6 - will expire between 1 and 2 weeks
            MF9226 expires 09/13/2010 00:03
            -- SKIPPED OUTPUT --
            QXN606 expires 09/16/2010 00:03
      11 - will expire in greater than three months
            CP9027 expires INFINITY
            -- SKIPPED OUTPUT --
            QXN566 expires INFINITY      
 
Summary by retention level of ALL media
     Level   # Media        Megabytes
       0       127         80049825.8
       1        23          6964717.2
       9        95         40530208.3

Explanation (thanks to Wayne T Smith)
“Non-active” tapes are unavailable for backups. They are Full,
Suspended, and/or Frozen.

“Active” are all the rest of your tapes that are not empty. NetBackup
will use them for future backups (with the same retention period).

The “are currently EXPIRED” list is probably of interest. These are
tapes that have no useful data on them, but will not be used for
backups. They are probably “frozen”. Hopefully you recorded why the
tape is frozen (NetBackup doesn’t record this, except for the case of
the errors file (view with “less +G
/usr/openv/netbackup/db/media/errors”)) so can deal with it appropriately.

Tapes in a non-active category eventually have their data expire
(expirations of infinity take longer) and the tapes automatically become
available for backups (unless frozen .. see above).

Tapes that become empty (all data expires), and are not frozen,
disappear from this report. Whether these tapes stay in their current
Volume Pool or go to Scratch depends on other factors.

So, we are going to unfreeze some of the tapes and find out WTF the INFINITY backups occupied large number of tapes.

To determine the reason the tape was frozen look at logs:
1. /usr/openv/netbackup/db/media/errors
2. bptm logs

To unfreeze all the frozen media except these which caught write error during save.

grep WRITE_ERROR /usr/openv/netbackup/db/media/errors | awk '{print $3}' | sort -u > write_error.txt
bpmedialist -summary > bpmedialist-summary.txt
for i in `grep FROZEN bpmedialist-summary.txt | awk '{print $1}' | sort -u`
  do
    if /usr/xpg4/bin/grep -q $i write_error.txt; then
       continue
    else
       echo $i
   fi
  done > tounfreeze.txt
for i in `cat tounfreeze.txt`; do bpmedia -unfreeze -m $i; done

Next, we run bpimmedia -mediaid against INFINITY tape and revealed that there was a policy mistakenly configured with infinity retention period sitting on the tape. Policy name was oracle-archivelog.

To find backup ids of old backups:

/opt/openv/netbackup/bin/admincmd/bpimagelist -policy oracle-archivelog
s -d 01/01/2009 -e 06/01/2010 -idonly

Then use backup id for manually recalculate expiration date of the backup

bpexpdate -recalculate -backupid oracle_1248262567 -d 0
Are you SURE you want to recalculate expiration dates on all images
that meet the following criteria:
  backupid         oracle_1248262567  
 
to expire NOW
Continue?(y/n)y

To avoid asking question use -force option of bpexpdate.
You can batch expire the backups:

for i in `/opt/openv/netbackup/bin/admincmd/bpimagelist -policy oracle-archivelogs -d 01/01/2009 -e 06/01/2010 -idonly |  awk '{print $8}'`; do bpexpdate -recalculate -backupid $i -d 0 -force && echo $i expired; done

After last backup with infinity retention was expired the media contained it is moved to scratch pool and ready to use for backups.

Tape backup operation: swapping tapes

Posted on August 19th, 2010

Sometimes I need to pull large number of tapes from large tape library driven by Netbackup and replace them with fresh ones. Manual selection of media from Netbackup’s java GUI annoys me, so there is a small piece of shell code to batch eject of tapes from command line. Replace $HOST with your media server hostname.

#!/bin/sh
 
if [ $# -ne 1 ]; then
   echo Usage: $0 [number of tapes to eject]
   exit 1
fi
 
NUM=$1
MEDIA_LIST=
$HOST=backup-server
 
for  media in `vmquery -w -a |  \
awk '$3=="HCART2" && $7=="TLD" && $12 != "scratch" {print $1, $15}' \
| sort -k2 -n -r | awk '{print $1}' | head -$NUM`
do
if [ -z "$MEDIA_LIST" ]; then
        MEDIA_LIST=$media
else 
        MEDIA_LIST=$MEDIA_LIST:$media
fi
done
echo $MEDIA_LIST
vmchange -res -w -rh $HOST -rt tld -rn 0 -multi_eject -ml $MEDIA_LIST

After batch of tapes was loaded into your library CAP you will be prompted to empty the CAP and continue with eject.

*** Eject volume batch #1 is ready to be removed ***
        remove 10 volume(s) from media access port(s) of robot number 0 now
        (press <RETURN> to continue)

Filed under Tape Libraries, Veritas NetBackup | No Comments »

Sun L25 LTO-1 Tape Library Booting

Posted on February 25th, 2010

Sun L25 LTO-1 Tape Library Prism System controller boot messages.

Read the rest of this entry »

Filed under Tape Libraries | No Comments »

Strange tape alerts and crappy HP tapes.

Posted on January 29th, 2010

The problem began when we have bought HP LTO-2 tape cartridges. Every day at least 1 tape got frozen
by Netbackup software with the following message:

TapeAlert Code: 0x08, Type: Warning, Flag: NOT DATA GRADE,
from drive IBM.ULTRIUM-TD2.004 (index 4), Media Id KI8765.

It causes scratch pool to become empty someday.
If you in trouble like this, you should know that the problem is in these HP’s tapes.
Sun Storagetek and FUJIFILM’s tapes are ok.

Update: it was revealed that the problem actualy was because of drive firmware. Update to the latest version of firmware has fixed the problem completely.

Filed under Tape Libraries | No Comments »

Find free cells within tape library with Veritas Netbackup

Posted on January 20th, 2010

Suppose you have to find out which cells in your library are empty. It would be easy task when you have not too many cells. If you have library with 100 cells and more, this little script can simplify the task.
Read the rest of this entry »

Filed under Tape Libraries, Veritas NetBackup | No Comments »