Monitoring HDS AMS storage with nagios using SNMP protocol

Posted on June 16th, 2010

We are going to get rid of HDS proprietary hi-track software and use industry standard SNMP protocol to monitor Hitachi midrange storage systems. NMS of choice will be Nagios. There are 2 ways of snmp monitoring in general and both are supported by Nagios. Your required to configure your storage system’s SNMP agent and specify SNMP community and SNMP trap destination. When done, check if it is working by executing check_snmp plugin.

# ./check_snmp -H ams-ctl0 -C public -o sysDescr.0 -P 1
SNMP OK - HITACHI  DF600F           Ver 0781/A-M |

Ok, you done with SNMP agent configuration, lets begin configure SNMP manager, i.e. Nagios.

1.  Active monitoring using plugin executing SNMP GET request for some OID. Here is a small Hitachi AMS storage monitoring nagios plugin which will do the task. It is ready for run with nagios embedded perl interpreter (ePN).

2. Passive monitoring using SNMP traps handling.
First, we need to install NetSNMP project’s snmptrapd daemon and point it to the program which will be handling all the traps coming in. We choose to run snmptt on every trap event. snmptrapd configuration will look like:

traphandle default /usr/sbin/snmptt
disableAuthorization yes
donotlogtraps  yes

Next, we configure snmptt iteslf to give it some understanding of what to do on receiving traps. Open snmptt.ini config and create section [TrapFiles]:

[TrapFiles]
snmptt_conf_files = <<END
/usr/local/etc/snmptt.conf.AMS500
END

Next, create snmptt.conf.AMS500 by running snmpttconvertmib tool on dfraid.mib which resides on AMS500 SNMP CD.

# export PATH=$PATH:/usr/local/bin
# snmpttconvertmib --in=dfraid.mib \
> --out=/usr/local/etc/snmptt.conf.AMS500 \
> --exec='/usr/local/nagios/libexec/eventhandlers/submit_check_result $r TRAP 1'

snmpttconvertmib calls snmptranslate from NetSNMP package not using full path, so you should correct your path to include directory in which snmptranaslate resides. Next, we define TRAP service.
nagios templates.cfg:

define service {
   name                    snmptrap
   use                     generic-service
   register                0
   service_description     TRAP
   is_volatile             1
   max_check_attempts      1
   normal_check_interval   1
   retry_check_interval    1
   passive_checks_enabled  1
   check_period            none
   check_command           check-host-alive
   notification_interval   31536000
}

And we use this template when defining actual services like this:

define service{
        use                             snmptrap,alltime_sms
        host_name                   amsctl0
        }

alltime_sms is a host template with defined contact groups, having sms targets in it.

Summary: storage sends trap, snmptrapd daemon handle it by calling snmptt trap handler, snmptt then calls submit_check_result script to submit passive check result to nagios. Nagios dispatches this submission to corresponding host service and takes appropiate action.

Filed under Nagios, Plugins, Storage | No Comments »

HDS terminology

Posted on May 20th, 2010

HDS Terminology

• Array Group – A set of four physical disk drives installed as a group in the subsystem.
• BED – Back‐end Director
• CHA – Hitachi’s name for the FED.
• CMA – Cache Memory Adapter board
• Concatenated Parity Group ‐ A configuration where the VDEVs corresponding to a pair of RAID‐10 (2D+2D) or RAID‐5 (7D+1P) Parity Groups, or a quad of RAID‐5 (7D+1P) Parity Groups, are interleaved on a RAID stripe by RAID stripe, round robin basis on their underlying disk drives.
• CSW – Cache Switch board
• DKC – Hitachi’s name for the base control frame (or rack) where the Logic Box and disks are located.
• DKA – Hitachi’s name for the BED
• DPVOL – a Dynamic Provisioning (DP) Volume; the Virtual Volume from an HDP Pool.
• Feature – an installable hardware option that includes two PCBs
• FED – Front‐end Director feature. Used to attach hosts to the storage system.
• HDU (Hard Disk Unit) – the 64‐disk container in a frame that has 32 disk slots on the front side and 32 more on the back. The HDU is further split into left and right halves, each on separate power domains. There are up to four HDUs per expansion frame, and two in the control frame.
• LDEV (Logical Device) – A logical volume internal to the subsystem that can be used to contain customer data. LDEVs are identified within the subsystem using a six hex digit identifier in the form LDKC:CU:LDEV. LDEVs are carved from a VDEV. LDEVs may be mapped to a host as a LUN, either as a single LDEV, or as a set of up to 36 LDEVs combined in the form of a LUSE.
• LUN (Logical Unit Number) – the host‐visible identifier assigned by the user to an LDEV to make it visible on a host port.
• eLUN (external LUN) – an external LUN Is one which is located in another storage array that is attached via two or more ePorts on a Universal Storage Platform V and accessed by the host through a Universal Storage Platform V.
• ePort (external Port) – An external array connects via two or more FED ports on some HDS model arrays instead of a host. The FED ports used in this manner are changed from a host target into an initiator port (or external Port) by use of the Universal Volume Manager Software product.
• LUSE (Logical Unit Size Expansion) – A concatenation of LDEVs that is then presented to a host as a single LUN.
• MP (microprocessor) – the CPU used on the FEDs and BEDs. Also called CHP (FED) or DKP( BED)
• Parity Group – a set of one or two Array Groups (a set physical disk drives) formatted as a RAID level.
• PCB – printed circuit board; an installable board (adapter). There are two PCBs per “Feature”.
• PDEV (Physical DEVice) – a physical internal disk drive.
• SMA ‐ Shared Memory Adapter board
• SVP – Service Processor (normall a Windows PC) installed in the control rack
• VDEV – The logical container from which LDEVs are carved. There are four types of VDEVs – Internal VDEV, External storage VDEV, Copy On Write (CoW) VDEV, Dynamic Provisioning (DP) VDEV
• V‐VOL Group – the organizational container of either a Dynamic Provisioning VDEV or a Copy on Write VDEV. With Dynamic Provisioning, it is used to hold one or more DP‐Vols.

Tags: hds, hitachi
Filed under Storage | No Comments »

Grow and shrink HDS lun and VxVM disks online

Posted on May 2nd, 2010

There is an example of how to grow lun and Veritas volume on the fly. Example created using HDS AMS2500 storage, Solaris 10, VxVM 5.1. Veritas volume is mirrored using 2 LUNs of 300G each located on different enclusures, let’s name them AMS2500_DC1 and AMS2500_DC2. These disks has veritas disk media names oradg02 and oradg03.

First, let’s do resize on HDS lun.

# ./auluchgsize -unit AMS2500_DC1 -lu 3 -size 350g
Are you sure you want to grow the logical unit? (y/n [n]): y
The logical unit has been grown successfully.
 
# ./auluchgsize -unit AMS2500_DC2 -lu 10 -size 350g
Are you sure you want to grow the logical unit? (y/n [n]): y
The logical unit has been grown successfully.

Next we resize veritas disk media, so the change can be seen by veritas volume manager.

# vxdisk resize AMS2500_DC1_3
# vxdisk resize AMS2500_DC2_10

If we take a look on /var/adm/message we can see information about previous operation

WARNING: VxVM vxio V-5-0-1045 changing GUID for disk AMS2500_DC1_3
WARNING: VxVM vxio V-5-0-1045 changing GUID for disk AMS2500_DC2_10

Next, we do volume resize

# df -h /oracle/PRD/sapdata4/
Filesystem             size   used  avail capacity  Mounted on
/dev/vx/dsk/oradg/oracle_sapdata4
                       300G   134G   156G    47%    /oracle/PRD/sapdata4
# vxassist -g oradg maxgrow oracle_sapdata4 oradg02 oradg03
Volume oracle_sapdata4 can be extended by 104830976 to: 733795584 (358298Mb+1280 sectors)
# /etc/vx/bin/vxresize -g oradg oracle_sapdata4 733795584
 
# df -h /oracle/PRD/sapdata4/
Filesystem             size   used  avail capacity  Mounted on
/dev/vx/dsk/oradg/oracle_sapdata4
                       350G   134G   202G    40%    /oracle/PRD/sapdata4

Filed under Storage, Veritas SF | No Comments »

Installing Veritas SF 5.1 on SPARC

Posted on March 11th, 2010

After installing Veritas SFHA 5.1 on Sun SPARC machine masta apply the following patches.

142629-01 - VRTSvxvm patch
142634-01 - VRTSvxfs patch
143261-01 - VRTSllt patch
143263-01 - VRTSgab patch
143264-01 - VRTSvcs patch
143265-01 - VRTSvcsag patch
143271-01 - VRTSodm patch

If you getting message while booting the system

NOTICE: VxVM not started

then you need to run vxinstall command

Filed under Solaris, Storage | No Comments »

Making Stripped Volume with VxVM

Posted on February 27th, 2010

From time to time I need to create temporary volume for copying some large data files or something like that..
This is how to make veritas driven stripped volume from bunch of FC disks.

Read the rest of this entry »

Filed under Storage | No Comments »

Removing FC disks from Veritas VM control

Posted on February 11th, 2010

We need to destroy number of raid groups belonging to some array and remove disks from OS and Volume manager control. The following procedure is the note on how I did this with Hitachi AMS500 and Solaris 9.

Read the rest of this entry »

Filed under Solaris, Storage | No Comments »