May 07

Failed to open Data Protector GUI: Internal Error

I just came across a strange Data Protector User Interface error. After a fresh installation the DP GUI reported the following error during first start.

Internal Error
Non-recoverable error: (0)
Failed to initialize the working environment.
System error: Unknown

The debug.log created in the home directory of the user had the following error in it that pointed to the source of the error, wrong permissions on the OpenView registry key.

05.05.2014 16:01:09 MANAGER.3472.3460 [“/lib/cmn/win32_registry.c $Rev: 39353 $ $Date:: 2013-10-04 14:54:49”:200] A.08.10 b200

[RegOpenKey2] RegOpenKeyEx failed to open key! Errno: 5

05.05.2014 16:01:09 MANAGER.3472.3460 [“/lib/cmn/win32_registry.c $Rev: 39353 $ $Date:: 2013-10-04 14:54:49”:125] A.08.10 b200

Function RegReadDWORD – error details:

root = HKEY_LOCAL_MACHINE

keyName = SOFTWARE\Hewlett-Packard\OpenView\OmniBackII\Common\Parameters

valueName = OB2_ALIGNNMENTEXCEPTION

retval = 5 ([5] Access is denied. )

value = 1L

Using regedit I discovered other HP software products sharing the same Hewlett-Packard registry key. It seems that HP ProtectTools Security Manager was installed first and created the key Hewlett-Packard with some strange permissions and ownership.


The result was nobody was able to open the OpenView key and all elements below.

I had to change the owner of the key Hewlett-Packard from Administrators to SYSTEM, enabled inheritance and selected replace all child object permissions. As a security measure, I reinstalled the GUI and now everything was working as expected.


This happened with HP ProtectTools Security Manager 8.0.3.1345 (sp63727.exe) running Windows 8.1 installed on a HP ProBook 6570b.

Apr 25

Delete db40\dcbf folder after migration to DP 8.x

If you migrate from a previous version to Data Protector A.08.00 or later on Windows, you might come across an issue where you’re unable to remove the old db40\dcbf directories using GUI or omnidbutil -remove_dcdir. This is most likely caused by the used path separator inside the Internal Database. Since version 8 the path separator is a simple / for UNIX and Windows platforms.

General requirements to remove a DCBF directory

  • No active DCBF file should point to this directory: Move any actively used DCBF files to a different DCBF directory and run omnidbutil -remap_dcdir
  • Don’t move any non-active DCBF files (most likely created due to an crash or similar) to a new db80 DCBF directory: Best way is to move the DCBFs to a temporary directory first, run omnidbcheck -bf and move back only DCBFs that are marked with as ERROR
  • Keep in mind that the DCBF name is related directly to the media id, just replace underscores (_) with colons (:) and strip away the last section: you can use omnimm -media_info 9ccb19ac:5350e998:555a:0298 to get details for DCBF 9ccb19ac_5350e998_555a_0298_22EA5053.dat

How to remove an empty DCBF directory with wrong path separator inside the IDB

The idea here is not to follow the instructions from http://support.openview.hp.com/selfsolve/document/KM00481540, since omnidbutil -writedb is not always possible and might be dangerous. I just created a simple SQL script that will change the wrong path separator and make regular removal possible.

C:\>omnidbutil  -list_dcdirs

Configured DC directories:

Allocation Sequence
|        Maximum Usage in MB
|        |        Maximum Number of Files in Directory
|        |        |        Minimum Free Space [MB]
|        |        |        |        Directory
|        |        |        |        |
===========================================================================
0        204800   100000   2048     O:/OmniBack/db40/dcbf
4        204800   100000   2048     O:/OmniBack/server/db80/dcbf/dcbf4
3        204800   100000   2048     O:/OmniBack/server/db80/dcbf/dcbf3
2        204800   100000   2048     O:/OmniBack/server/db80/dcbf/dcbf2
1        204800   100000   2048     O:/OmniBack/server/db80/dcbf/dcbf1
0        204800   100000   2048     O:/OmniBack/server/db80/dcbf/dcbf0

DONE!

C:\>notepad C:\temp\dcbf_dir_fix.sql

–Displays DCBF directories before modification
SELECT dcbf_directory from dp_catalog_dcbf_directory;

–Fixes wrong path separator for db40 DCBF directories on Windows CM
UPDATE dp_catalog_dcbf_directory SET dcbf_directory = replace(dcbf_directory, ‘\’, ‘/’);
UPDATE dp_catalog_dcbf_directory SET dcbf_directory = replace(dcbf_directory, ‘\\’, ‘/’);

–Displays DCBF directories after change
SELECT dcbf_directory from dp_catalog_dcbf_directory;

C:\>omnidbutil -run_script C:\temp\dcbf_dir_fix.sql -detail
           dcbf_directory
————————————
 O:\OmniBack\db40\dcbf
 O:/OmniBack/server/db80/dcbf/dcbf4
 O:/OmniBack/server/db80/dcbf/dcbf3
 O:/OmniBack/server/db80/dcbf/dcbf2
 O:/OmniBack/server/db80/dcbf/dcbf1
 O:/OmniBack/server/db80/dcbf/dcbf0
(6 rows)
 
UPDATE 6
UPDATE 6
           dcbf_directory
————————————
 O:/OmniBack/db40/dcbf
 O:/OmniBack/server/db80/dcbf/dcbf4
 O:/OmniBack/server/db80/dcbf/dcbf3
 O:/OmniBack/server/db80/dcbf/dcbf2
 O:/OmniBack/server/db80/dcbf/dcbf1
 O:/OmniBack/server/db80/dcbf/dcbf0
(6 rows)
DONE!
C:\>omnidbutil -remove_dcdir O:/OmniBack/db40/dcbf
DONE!

C:\>omnidbutil  -list_dcdirs

Configured DC directories:

 Allocation Sequence
 |        Maximum Usage in MB
 |        |        Maximum Number of Files in Directory
 |        |        |        Minimum Free Space [MB]
 |        |        |        |        Directory
 |        |        |        |        |
===========================================================================
 4        204800   100000   2048     O:/OmniBack/server/db80/dcbf/dcbf4
 3        204800   100000   2048     O:/OmniBack/server/db80/dcbf/dcbf3
 2        204800   100000   2048     O:/OmniBack/server/db80/dcbf/dcbf2
 1        204800   100000   2048     O:/OmniBack/server/db80/dcbf/dcbf1
 0        204800   100000   2048     O:/OmniBack/server/db80/dcbf/dcbf0

DONE!

Sep 04

Data Protector Quick Fixes

There are situations in which you receive test modules or quick fixes from HP support to resolve a specific issue in your environment. Even if they provide instructions on how to apply them, I found the following general procedure working quite well for Linux clients and cell managers.

  • Upload the fix to the client system that need to be updated

decm01:/tmp/QCCR # ls -al

total 2132

drwxr-xr-x 2 root root 4096 Sep 4 19:29 .

drwxrwxrwt 6 root root 4096 Sep 4 19:28 ..

-rw-r–r– 1 root root 2170880 Sep 4 19:23 QCCR2A39087_TM4.tar

  • Extract the archive provided by HP support to a new directory and examine documentation included

decm01:/tmp/QCCR # /bin/mkdir QCCR2A39087_TM4

decm01:/tmp/QCCR # /bin/tar xf QCCR2A39087_TM4.tar -C QCCR2A39087_TM4

decm01:/tmp/QCCR # cd QCCR2A39087_TM4

decm01:/tmp/QCCR/QCCR2A39087_TM4 # ls -al

total 2132

drwxr-xr-x 2 root root 4096 Sep 4 19:29 .

drwxr-xr-x 3 root root 4096 Sep 4 19:29 ..

-rw-r–r– 1 27419 users 971 Aug 29 07:53 README.txt

-r-x—— 1 27419 users 2164336 Aug 29 07:45 csm

decm01:/tmp/QCCR/QCCR2A39087_TM4 # /usr/bin/less README.txt

  • Check if the new binaries are running on the system and reports a valid build number

decm01:/tmp/QCCR/QCCR2A39087_TM4 # /usr/bin/file csm

csm: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), for GNU/Linux 2.4.1, dynamically linked […]

decm01:/tmp/QCCR/QCCR2A39087_TM4 # /usr/bin/ldd csm

linux-vdso.so.1 => (0x00007fff04df9000)

libCrdm64.so => /opt/omni/lib/libCrdm64.so (0x00007f666845d000)

libCadm64.so => /opt/omni/lib/libCadm64.so (0x00007f6668355000)

libCrpc64.so => /opt/omni/lib/libCrpc64.so (0x00007f6668242000)

libCuapi64.so => /opt/omni/lib/libCuapi64.so (0x00007f6668140000)

libCrm64.so => /opt/omni/lib/libCrm64.so (0x00007f6668026000)

libCncp64.so => /opt/omni/lib/libCncp64.so (0x00007f6667f20000)

libm.so.6 => /lib64/libm.so.6 (0x00007f6667c7e000)

libdl.so.2 => /lib64/libdl.so.2 (0x00007f6667a79000)

libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x00007f666776f000)

libarm.so => /opt/omni/lib/arm/libarm.so (0x00007f666766e000)

libBrandChg_64bit.so => /opt/omni/lib/libBrandChg_64bit.so (0x00007f6667568000)

libc.so.6 => /lib64/libc.so.6 (0x00007f66671f4000)

libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f6666fd7000)

/lib64/ld-linux-x86-64.so.2 (0x00007f666856b000)

libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f6666dc0000)

decm01:/tmp/QCCR/QCCR2A39087_TM4 # ./csm

HP Data Protector A.07.00: CSM, internal build QCCR2A39087_TM4, built on […]

  • Locate the binaries to be replaced (only csm in that case), create a backup including permissions

decm01:/tmp/QCCR/QCCR2A39087_TM4 # which csm

/opt/omni/lbin/csm

decm01:/tmp/QCCR/QCCR2A39087_TM4 # /bin/cp -a /opt/omni/lbin/csm{,.`/bin/date +%Y%m%d`}

  • Replace the files and check if build number is reported correctly for each one and their backup. You might need to shutdown Data Protector services using omnisv stop in advance. Adjust permissions and ownership not matching the backup.

decm01:/tmp/QCCR/QCCR2A39087_TM4 # /bin/cp csm /opt/omni/lbin/csm

decm01:/tmp/QCCR/QCCR2A39087_TM4 # /bin/ls -al /opt/omni/lbin/csm*

-r-x—— 1 root sys 2164336 Sep 4 19:31 /opt/omni/lbin/csm

-r-x—— 1 root sys 1955592 Jul 22 21:23 /opt/omni/lbin/csm.20120904

decm01:/tmp/QCCR/QCCR2A39087_TM4 # /opt/omni/lbin/csm

HP Data Protector A.07.00: CSM, internal build QCCR2A39087_TM4, built on […]

decm01:/tmp/QCCR/QCCR2A39087_TM4 # /opt/omni/lbin/csm.20120904

HP Data Protector A.07.00: CSM, internal build 100, built on Tue Jul 24 09:51:39 2012

  • Start the services if stopped using omnisv start and check if the installed test module is working as expected. omnicheck -patches should list the non-standard binaries.

Keep in mind to track applied Quick Fixes if you’re going to install GR patches to your cell. Check the release notes of the fixes to make sure the issue has been resolved.

Aug 14

Data Protector Push installation for Debian clients

Integrating Debian or Debian-based Linux clients (e.g. Ubuntu) into a Data Protector cell could be a pain, if you have more than a bunch of them. Patch rollout and release upgrades will even take more time if you poke around with alien to convert the Data Protector rpm into deb format. There are some simple requirements that will allow you to leverage Data Protector Push installation method for Linux/UNIX even for Debian-based distributions:
 

  1. Name resolution work from client to IS and client to CM plus the other way around (DNS preferred)
  2. /etc/services must not contain any definition for 5555/tcp
  3. The packages xinetd and rpm must be installed
  4. root SSH login must be enabled on the client using public keys
  5. Use FQDN for all operations on CLI and GUI

 
With that information in mind, we can do the required modifications to our Debian 6.0 client and push the Data Protector Disk Agent (DA) to the system when ready.

  • Check for Debian and kernel version running (for your reference only)

root@decm01:~# /bin/cat /etc/debian_version

6.0.5

root@decm01:~# /bin/uname -a

Linux decm01 2.6.32-5-amd64 #1 SMP Sun May 6 04:00:17 UTC 2012 x86_64 GNU/Linux

  • Check if the port 5555/tcp is not blocked by any entry other than omni (looks good on Debian squeeze)

root@decm01:~# /bin/grep 5555 /etc/services

rplay 5555/udp # RPlay audio service

  • Installation of required packages for Data Protector push installation on Debian client

root@decm01:~# /usr/bin/apt-get install xinetd rpm

  • Allow password-less access to Debian system using SSH: command is run from prepared UNIX IS

[root@deis01 ~]# /usr/bin/ssh-copy-id -i /root/.ssh/id_dsa.pub decm01.syncer.de

21

The authenticity of host ‘decm01.syncer.de (192.168.25.150)’ can’t be established.

RSA key fingerprint is df:97:00:b2:62:33:1d:1e:fc:57:f7:94:bb:c5:f7:ba.

Are you sure you want to continue connecting (yes/no)? yes

Warning: Permanently added ‘decm01.syncer.de,192.168.25.150’ (RSA) to the list of known hosts.

root@decm01.syncer.de’s password:

Now try logging into the machine, with “ssh ‘decm01.syncer.de'”, and check in:

.ssh/authorized_keys

to make sure we haven’t added extra keys that you weren’t expecting.

  •  Now try the remote installation from UNIX IS to new client (DA only in my case)

[Normal] Starting installation session on Dienstag, 14. August 2012, 20:24:54…

[Normal] Getting list of clients for installation…

[Normal] Connecting to client decm01.syncer.de…

[Normal] <decm01.syncer.de> Checking for response from system decm01.syncer.de.

[Normal] <decm01.syncer.de> Using secure shell protocol to connect to clients …

[Normal] Client decm01.syncer.de (Linux 2.6.32-5-amd64) OK.

[Normal] <decm01.syncer.de> Starting installation/update for client system decm01.syncer.de at…

[Normal] <decm01.syncer.de> Installing Core Component

[Normal] <decm01.syncer.de> Installation of Core Component succeeded.

[Normal] <decm01.syncer.de> Installing Disk Agent

[Normal] <decm01.syncer.de> Installation of Disk Agent succeeded.

[Normal] <decm01.syncer.de> Update of client system decm01.syncer.de completed.

[Normal] Installation session finished on Dienstag, 14. August 2012, 20:25:02.

============================================================================

Session completed successfully!

============================================================================

  • If the client is imported to cell automatically, everything is fine. Verify if the rpm files are installed and the service responds to requests locally

root@decm01:~# /usr/bin/rpm -qa

OB2-DA-A.06.20-1.x86_64

OB2-CORE-A.06.20-1.x86_64

root@decm01:~# /usr/bin/telnet localhost 5555

Trying 127.0.0.1…

Connected to localhost.

Escape character is ‘^]’.

HP Data Protector A.06.20: INET, internal build 403, built on Fri Jun 15 23:33:35 2012

Connection closed by foreign host.

Aug 03

Restore from NDMP device mirror

I just came across an issue at customer site where Data Protector Device Mirror was used to duplicate NDMP backups. In general this is a great way to reduce the amount of time it takes to create separate sets of media at backup time. The worse thing is, this feature is not supported in Data Protector including version 7.01.

The backup will work as expected, but if you need to restore from the tape that was created using Mirror (not the original one) the session will just fail with the following error. Keep in mind, the restore from the original/primary media is working fine.

[Normal] From: RSM@decm01.syncer.de “” Time: 01.08.2012 13:56:35

Restore session 2012/08/01-50 started.

[Normal] From: RMA-NDMP@dema01.syncer.de “MSL8096_D2” Time: 01.08.2012 13:56:38

STARTING Media Agent “MSL8096_D4”

[Normal] From: RMA-NDMP@dema01.syncer.de “MSL8096_D2” Time: 01.08.2012 13:56:41

By: UMA@dema01.syncer.de@c32t0l0

Loading medium from slot 3 to device c32t0l0

[Major] From: RSM@decm01.syncer.de “” Time: 01.08.2012 13:57:27

Unknown internal error.

This looks pretty familiar, since there was the defect QCCR2A34355 that prevented NDMP restores from completing on 6.20 and 7.00.

DP 6.2 NDMP Restore with unknown error
http://support.openview.hp.com/selfsolve/document/KM1274902

Even with fixes/patches in place, the restore from Device Mirror (copy) will fail. This seems to be an issue with metadata not written correctly to DCBFs. omnidbcheck -sibf will show inconsistencies on all media used for device mirror.

I found a workaround for this issue that works pretty straight forward and allows you to restore the data.

  1. Identify the media(s) required for restore/restore chain using the restore dialog
  2. Load the tapes into library if not already present and perform barcode scan
  3. Check which session is stored on media(s)
  4. Remove data protection from session(s) in IDB that is used on that media to allow export (if not already expired)
  5. Export media(s) in question from IDB (right click on Slot -> Export)
  6. Import the media(s) in question back to IDB (right click on Slot -> Import) / make sure you select “Import Copy as Original” and “Log all”

Jul 22

Patch Data Protector standalone client

There are cases in with you need to apply GA patches to a UNIX client and you don’t have an Data Protector UNIX IS (Installation Server) available. This is worse, because the patches are designed to update an installation server only. The following procedure will help you out. Instead of applying the patches to an IS, we’ll just extract the desired target platform patch from the archive. I’ll demonstrate how this is done with DPLNX_00200.tar for DP 6.20.

  • Download the patches and extract the tarball

    /bin/tar xf DPLNX_00200.tar

  • Review the patch documentation and convert the IS RPM into CPIO archive

    /usr/bin/less DPLNX_00200.text
    /usr/bin/rpm2cpio DPLNX_00200.rpm > DPLNX_00200.cpio

  • Extract the CPIO archive into current directory

    /bin/cpio -dmi < DPLNX_00200.cpio

  • Copy the target platform patch out of the repository directory (the same structure can be found on any UNIX IS).

    /bin/cp opt/omni/databases/vendor/omnicf/gpl/x86_64/linux-x86-64/A.06.20/packet.Z \
    `file opt/omni/databases/vendor/omnicf/gpl/x86_64/linux-x86-64/A.06.20/packet.Z | awk '{ print $6 }'`.rpm

    /bin/cp opt/omni/databases/vendor/integ/gpl/x86_64/linux-x86-64/A.06.20/packet.Z \
    `file opt/omni/databases/vendor/integ/gpl/x86_64/linux-x86-64/A.06.20/packet.Z | awk '{ print $6 }'`.rpm

    Please note: The CORE patch is special, since it contains omnicf (CORE) and integ (Online Integration) patches. In case of Data Protector 6.11 and earlier, you need to gunzip the packet.Z before you can rename it. This has been changed with version 6.20 and later.

  • Apply the patch to the system. This does not even require the componet to be installed in advance.

    /bin/rpm -Uvh --replacefiles --replacepkgs OB2-CORE-A.06.20-1.rpm

Jul 21

Data Protector requirements for Linux CM / revised

Here are some required, but not well documented requirements for Data Protector Cell Manager (Cell Server) installation on the Linux platform. Update: This information has been revised for SLES11. These steps are not intended to replace the Installation and Licensing documentation shipped with the media kit. Rather they should help you to bring things up more quickly. As always, think before you type or copy & paste. Things might vary on your distribution. Please check directory structures and files before using the commands!

  • Make sure the port 5555 is not blocked by any conflicting entry / uncomment it if found

    /usr/bin/perl -p -i -e 's/personal-agent/\#personal-agent/' /etc/services
    /usr/bin/perl -p -i -e 's/rplay/\#rplay/' /etc/services

  • Make sure Korn shell (ksh) is installed / most likely missing on RHEL

    /bin/rpm -qa "ksh"
    /usr/bin/yum install ksh

  • Make sure compat-libstdc++ is installed. If not, install using distribution tools (yum in case of RHEL).

    /bin/rpm -qa | /usr/bin/awk '/compat-libstdc\+\+\-33|libstdc\+\+33/'
    /usr/bin/yum install compat-libstdc++-33

  • Make sure glibc.i686 is installed. This is required for the included 32 bit Perl to work on Linux x86_64.

    /bin/rpm -qa "glibc*" | /usr/bin/awk '/i686|32bit/'
    /usr/bin/yum install glibc.i686

  • Make Data Protector shared libraries available in LDPATH and update cache

    /bin/cat << EOF > /etc/ld.so.conf.d/omni.conf
    # /etc/ld.so.conf.d/omni.conf - Added for Data Protector Software
    /opt/omni/lib
    /opt/omni/lib/arm
    /opt/omni/java/server/bin
    EOF
    /sbin/ldconfig

  • Make Data Protector binaries available system wide using PATH variable

    /bin/cat << EOF > /etc/profile.d/omni.sh
    # /etc/profile.d/omni.sh - Added for Data Protector Software
    PATH=\$PATH:/opt/omni/bin:/opt/omni/lbin:/opt/omni/sbin
    export PATH
    EOF

  • Extend search path for man command to include Data Protector manpages. The /etc/manpath.config on SLES is adjusted by omnisetup.sh making those actions irrelevant.

    /bin/echo -e "\n# Added for Data Protector Software\nMANPATH\t\t/opt/omni/lib/man" >> /etc/man.config

Jul 06

WWN Converter 3.4.0

I just updated the WWN Converter to version 3.4.0. The changes based on the some feedback from the field. Thanks for that. If the included auto-update function fails, re-install the product from the ZIP file or via ClickOnce.

Changed things since version 3.0.0:

  • cleanup conversion dialog messages
  • updated OUI Database / 06.07.2012
  • minor GUI tweaks
  • rearrange of some menu items
  • added always on top function
  • make lowercase option

You can download the current version via ClickOnce or in a ZIP file. Please keep in mind, that this piece of software is shippted without any warranty.

Jan 01

Investigating NDMP performance

I spend some time on the Data Protector NDMP Integration, especially to compare it to CIFS/NFS backups available with the regular Disk Agent. Luckily an older EMC Celerra NS20 was available and I was able start immediately. A quick look at the NDMP support matrix revealed no restrictions since the unit was running T5.6.50.203.

To give you any idea on how the NS20 was configured, I just put together a quick drawing and some tables. This NS20 comes with one DAE4P used for FC and one used for SATAII drives. Redundant X-Blades (Data Movers) are configured as Active/Standby at the bottom provides NAS capability of the unit. Each X-Blade is equipped with two so called AUX ports used for tape connectivity. In my setup the first AUX port of each X-Blade was attached to the SAN infrastructure and zoned to an existing tape library mainly used for Object copies at the moment.


Read the rest of this entry »

Dec 30

Quiesced Snapshots for Exchange 2010 on vSphere 5.0

Due to an upgrade from vSphere 4.0 to 5.0 some of our virtual machines stopped working in VEPA/vStorage backup sessions. I’ve noticed that a quiesced snapshot using vCenter was not possible. I was able to fix the most of them by reinstalling VMware Tools to fix some VSS inconsistencies, but an Exchange 2010 VM caused some additional issues.

vCenter reported the following warnings during creation of a quiesced snapshot.

Create virtual machine snapshot / An error occurred while quiescing the virtual machine. See the virtual machine’s event log for details.

The guest OS has reported an error during quiescing. The error code was: 5 The error message was: ‘VssSyncStart’ operation failed: IDispatch error #8449 (0x80042301)

Check the output of vssadmin list writers to determine the status of available VSS writers. In our case the Exchange writer reports an error.

C:\>vssadmin list writers
vssadmin 1.1 – Volume Shadow Copy Service administrative command-line tool
(C) Copyright 2001-2005 Microsoft Corp.

[…]

Writer name: ‘Microsoft Exchange Writer’
Writer Id: {76fe1ac4-15f7-4bcd-987e-8e1acb462fb7}
Writer Instance Id: {10552a00-36bf-401a-9141-4436d61f0d2c}
State: Failed
Last error: Retryable error

[…]

The event logs of the Windows 2008 R2 VM had the following errors, pointing to a VSS Provider related issue. Strange but true, the Exchange 2010 VSS Backups using Data Protector work as expected without any problems.
Read the rest of this entry »

Older posts «

» Newer posts