Parm Patram
Phone : (404) 216-4283
Email: ppatram _at_ btsincorporated.com

He has proven ability to improve manageability, stability, security and performance. His analytical, troubleshooting, and creative abilities allow him to adapt to any circumstance quickly, accurately pinpoint root cause, and come up with unique solutions.

Skills Summary
* Oracle 10g/11g
* Recovery Manager (RMAN)
* User and Security Administration
* Export/Import (exp, expdp)
* Logminer
* Streams Replication
* Dataguard Standby Database
* Oracle Enterprise Manager (OEM)
* Oracle EBS
* Performance Tuning
* Materialized Views (Snapshots)
* PL/SQL, sqlplus
* Real Application Clusters (RAC)
* mySQL
* postgresql
* Unix Shell Scripting (bash, ksh)
* PHP/Perl, expect, C Programming
* Code Versioning (svn, fisheye)
* Linux/Solaris (kickstart, LVM/Disksuite RAID, NFS, samba)
* Installation/Configuration (rpm, up2date, yum, apt-get)
* Advanced Network Troubleshooting (ping, ipcalc, arping, tcpdump, nmap)
* Advanced Application Troubleshooting (strace/truss, netstat, ipcs, lsof, ps, jconsole)
* Monitoring (Hyperic, openNMS, SNMP)
* Migration (rsync)
* Change/Configuration Auditing (Tripwire, custom scripting)
* Change Control
* Automation (expect, Perl, C, shell)
* Apache/Tomcat

Work Employment History
Nov 2009 – Present: Linux/Solaris & Oracle Database Administrator at Barnhardt.
* Managed development, stage and production databases and application servers for large manufacturing company’s ERP.
* Install/configure Solaris 9 on Sunfire 280R. Developed method to clone the production server to backup/stage for consistency. Did some hardware work with RSC.
* Updated old documentation and wrote new articles in tech services wiki relating to Linux/Solaris/Oracle.
* Developed and implemented method to permit Oracle 10g Dataguard to compress archivelogs using SSH port forwarding (Linux).
* Wrote custom plugins in Perl/bash/ksh for Hyperic to monitor Oracle Streams, Oracle Dataguard, and certain specific metrics on the application server.
* Developed scripts using expect to test ERP functionality in an automated fashion.
* Helped establish Change Control process.
* Installed ERP on Linux (RHEL 5) for functional/development testing.

* Streamlined Dev refresh process by scripting much of the pieces and creating a checklist of the order to execute those scripts. Project time did not permit a fully automated approach, which was the next progression.
* Updated some of the scripts to use command-line arguments. This allowed the scripts to be exactly the same, but function differently based on arguments.
* Updated the ‘standby monitor’ to perform checks to eliminate false alerts.
* Developed ‘Tango’, which is a way to detect changes to an Oracle database’s settings or object properties. Tango consists of Oracle PL/SQL package and bash/Perl OS scripts.
* Developed method to resize the filesystem used for archive logging without needing to stop the database (Linux).
* Installed monitoring/auditing tools: Tripwire, openNMS, and Hyperic on RHEL 5 systems.
* Helped with securing the ERP platform.
* Identified/corrected performance problems resulting from over-sized SGA. Implemented SQL Profiling to improve performance of certain problem queries (Oracle).

Aug 2005 – Nov 2009: Senior Linux/Oracle Database Consultant at Business Technology Services, Inc.
* Specialized in RHEL, Oracle 10g and Oracle EBS.
* Managed company’s dedicated Linux server which services their website and email needs.
* Worked for many clients of BTS, Inc. in various industries.
* On call support.
* Oracle Dataguard standby database implementation, troubleshooting and tuning.
* Oracle Streams replication.
* General Troubleshooting and problem resolution.
* Custom monitoring for Linux OS and Oracle database.
* Oracle database performance tuning.
* Oracle RAC on Linux (RHEL) implementation and monitoring.
* Oracle EBS (11i and 12i) patch investigation and installation.
* Oracle EBS cloning.

Jan 2000 – August 2005: Senior Linux Administrator at Interland, Inc (now Peer1).
* Worked in Dedicated Server Solutions department which offered billable Admin services to company’s Dedicated Server customers.
* DSS also provided Data Center Operations support for company’s Shared Servers in addition to the Dedicated Servers (5000+ total).
* Tier III support for Dedicated Services (non-billable questions and issues).
* Assisted Engineering with build issues by writing work-around scripts and providing diagnostic data.
* Was the leading revenue earner in DSS.
* Some hardware work (HP/Compaq Proliant, Cobalt, white-box).
* Plesk Linux Control Panel SME. Wrote many scripts to manage and migrate Cobalt RAQ, systems with no control panel, and older versions of Plesk to the newest Plesk versions.
* Instituted Clean, Patch, &Lockdown service for customers whose servers had been hacked into.
* Invented ‘Backplane Sharing’ to retrieve data from RAID array on non-booting servers by using the dead system to power up the RAID array so it can be mounted to a running machine in order to perform a data copy.
* Developed unique method of reviving badly hacked servers that fail to boot by booting from CD, chroot, then manual services startup. Used as a stop-gap until a full migration to a new server could be effected.
* Provided excellent sales support for sizing requirements and other technical consulting required for completing the sale.

Miscellaneous Freelance Work
Omega Learning Centers (http://omegalearningcenter.com).
Administer company’s web server and email server.  Maintain and add new functionality to company’s suite of proprietary web-based interfaces used for running their business. Instrumental in scaling the products out to meet their need to franchise. Company now has 3 locations and growing.

3D Cellular (http://omegalearningcenter.com).
Administer company’s web and email server. Manage server firewall.  Responsible for the SSL certificates in use for both the website and email. Implemented VPN between LAN and Internet server. Designed custom scripts for backing up customer’s project files from their local area network to the server as well as designing and implementing an off-site backup plan for the server itself.

Kagan Creative (http://kagancreative.com).
Server administration and migration. Built and configured a dedicated mySQL server for client. Also responsible for about 10 SSL certificates for various websites running on the web server. Configured 3-node load-balanced web server for client.

HPP Christian Hosting  (http://hppchristianhosting.com).
Administered 3 high-traffic and heavily populated web servers totaling about 1000 different websites. Identify and upgrade old vulnerable applications running on the various websites that were being hacked into. Migrated the servers a few times to keep up with growing demand and the need for faster hardware. Assisted with some handover work when client sold the business in September 2007.

Certifications
Oracle Certified Professional (Database and Grids), 2009
Solaris Certified System Administrator, 2003
Microsoft Certified Systems Engineer, 1999

Education
1990, Associates of Arts (Data Processing)

References
Available Upon Request

Repairing Corruption on a DiskSuite File System

We have had a small number of cases in which file systems that are mirrored using DiskSuite have become corrupted. An explanation of how that arose is beyond the scope of this document. See notes in RT#56174 for background and for some messages from Sun about what is and is not safe to do in various circumstances. This is pretty important.

The procedure described here is provided by Sun in order to undo DiskSuite, restore affected file systems from backups, and redo DiskSuite. We adapt this procedure to run fsck safely on bad file systems.

Why is this needed? If a file system cannot be unmounted (e.g. root file system), but you need to fsck it, then you need to boot from CD-ROM. But the OS on the CD-ROM has no DiskSuite drivers, so it can’t understand DiskSuite metadevices. You should never fsck the underlying physical slices of a DiskSuite metadevice (e.g. mirrors round-robin, and fscking their underlying slices can put them out of synch). Thus, you must undo DiskSuite first before fscking such file systems, and re-do DiskSuite afterwards.

In our cases, the root file system was okay, but others were corrupted. To fix the corruption, the outline is:

• Boot from CD-ROM.
• Undo DiskSuite following Steps 5 through 7 of the following procedure from Sun.
• Fsck the underlying slices on whichever of the mirrored disks you’re going to consider to be your “good” one
• Repeat fsck until all slices fsck without errors. If fsck finds a problem and repairs it, fsck again until no errors are found!
• Continue with the procedure from Sun at Step 8 to re-do DiskSuite.

NOTES:

• The following document is unaltered except for formatting.
• At Step (8), boot single user first, and set a root password. Recall that our standard practice is to have no root password in /etc/passwd or /etc/shadow, but you won’t be able to become root using suw after logging in as yourself when you’re booting single-user and leaving other file systems (e.g. /u) unmounted. So you must set a root password and then log in as root.
• Step (10) talks about restoring contents of md.tab. This file contains only comments, even on live systems. And the example they show is not actually md.tab; it matches with what is found in md.cf.
• Rather than using md.cf and running metainit to automatically recreate all the DiskSuite metadevices as per Step 11, I recreated all the DiskSuite metadevices (including metadb replicas) “by hand” using documentation we typically stash in the local_<hostname>.uwaterloo.ca package about the machine’s DiskSuite configuration. So I can’t confirm whether or not the automatic approach works as documented.

Looks like this might be internal Sun tech document 1-25-14650-1.
E.g. try this URL: http://sunsolve.central.sun.com/search/document.do?assetkey=1-25-14650-1

* Resolution *  	*Top*
http://sunsolve.central.sun.com/search/document.do?assetkey=1-25-14650-1

NOTE: In order to restore the machine exactly the way it was, it is necessary
to have a record of the locations of the state databases, and also the
configuration of the metadevices.   These are *not* held in user-readable form
on the root filesystem, so it is useful to get the information BEFORE a crash
with the commands:

  # metadb -i
  This command shows the condition and location of DiskSuite's state
  database replicas. It is useful to have this information in case their
  recreation is necessary.

  # metastat -p
  You will redirect this output to a file for safekeeping. This
  information is most often sent to the /etc/opt/SUNWmd/md.tab file for
  disaster recovery purposes (see step 6 below). Check the md.tab file prior
  to overwriting it, though. If a valid one exists, redirect the metastat -p
  to another filename and run a 'diff' on it to check for inconsistencies.
  The metastat -p, however, will reflect the most current, known, configuration.

Of course, you will need to create a new root disk and restore the last
full backup to that new disk.   It is expected that you will partition
the disk in much the same manner as you had it before.   Remember that
if there was a small partition on your original disk for a state
replica that you do the same thing for this disk.

Once the disk has been created, restore your root filesystems from
backups.   However, before attempting to boot the system after the
restore, we must make some necessary changes to let Solaris know that
the root disk is no longer mirrored.   To do this, the /etc/system and
the /etc/vfstab files must be modified.                                                                                         

1) Boot cdrom, and mount the root filesystem to /a.

     ok boot cdrom -s
           # mount /dev/dsk/c#t#d#s# /a

2) Restore the root filesystem from backup tape into /a and initialize
   the boot block using

       # installboot /usr/platform//lib/fs/ufs/bootblk /dev/rdsk/c#t#d#s#

3) Mount and restore any other critical filesystems such as /usr /var /opt etc.

4) If the replacement machine does not have it disks connected via the same
   paths as the original then you may have to follow the procedure documented in
   SRDB 15010 to rebuild the /devices and /dev structures.

5) vi /a/etc/system (before using vi you may need to set the terminal
   type e.g. TERM=sun; export TERM)

     Remove ALL lines between the "MDD root info" lines as well as those
     between the "MDD database info" lines.   In the following example
     file, all these lines would be removed from the file:

       -------------------------------------------------------------
       * Begin MDD root info (do not edit)
       forceload: misc/md_trans

       forceload: misc/md_raid
       forceload: misc/md_hotspares
       forceload: misc/md_stripe
       forceload: misc/md_mirror
       forceload: drv/sd
       forceload: drv/esp
       forceload: drv/espdma
       forceload: drv/sbus
       forceload: drv/iommu
       rootdev:/pseudo/md@0:0,3,blk
       * End MDD root info (do not edit)* Begin MDD database info (do not edit)
       set md:mddb_bootlist1="sd:14:16 sd:15:16"
       * End MDD database info (do not edit)
       -------------------------------------------------------------

6) vi /a/etc/vfstab

     Change all metadevices for the root filesystems (root, usr, var, and
     opt) back to regular slices (/dev/dsk/c#t#d#s#).   Comment out all
     the other metadevices for the time being.   For example:

   ** EXAMPLE FILE BEFORE:
   ---------------------------------------------------------------------------
   #device         device           mount         FS    fsck mount     mount
   #to mount       to fsck          point         type  pass at boot options
   #
   /proc           -                /proc         proc  -    no          -
   fd              -                /dev/fd       fd    -    no          -
   swap            -                /tmp          tmpfs -    yes         -
   /dev/md/dsk/d1  -                -             swap  -    no          -
   /dev/md/dsk/d0  /dev/md/rdsk/d0  /             ufs   1    no          -
   /dev/md/dsk/d2  /dev/md/rdsk/d2  /usr          ufs   1    no          -
   /dev/md/dsk/d3  /dev/md/rdsk/d3  /var          ufs   2    yes         -
   /dev/md/dsk/d4  /dev/md/rdsk/d4  /opt          ufs   2    yes         -
   /dev/md/dsk/d5  /dev/md/rdsk/d5  /export/home  ufs   3    yes         quota
   /dev/md/dsk/d6  /dev/md/rdsk/d6  /export/home1 ufs   3    yes         quota

   ---------------------------------------------------------------------------

   ** EXAMPLE FILE AFTER:
   ---------------------------------------------------------------------------
   #device           device             mount         FS    fsck mount    mount
   #to mount         to fsck            point         type  pass at boot options
   #
   /proc             -                  /proc         proc  -    no      -
   fd                -                  /dev/fd       fd    -    no      -
   swap              -                  /tmp          tmpfs -    yes     -
   /dev/dsk/c0t0d0s1 -                  -             swap  -    no      -
   /dev/dsk/c0t0d0s0 /dev/rdsk/c0t0d0s0 /             ufs   1    no      -
   /dev/dsk/c0t0d0s6 /dev/rdsk/c0t0d0s6 /usr          ufs   1    no      -
   /dev/dsk/c0t0d0s7 /dev/rdsk/c0t0d0s7 /var          ufs   2    yes     -
   /dev/dsk/c0t0d0s4 /dev/rdsk/c0t0d0s4 /opt          ufs   2    yes     -
   #/dev/md/dsk/d5   /dev/md/rdsk/d5    /export/home  ufs   3    yes     quota
   #/dev/md/dsk/d6   /dev/md/rdsk/d6    /export/home1 ufs   3    yes     quota
   ---------------------------------------------------------------------------

7) Remove all lines (except for the 2 comment lines at the top) from
     the "mddb.cf" file.   This file exists either in the /etc/opt/SUNWmd
     or /etc/lvm directory, depending on the version of DiskSuite you are
     running.

  Boot the system from the newly restored boot disk.   When the system
     comes up, only root (and /usr, /var, and /opt, if they exist) will be
     mounted using the slices on the new root disk.                                                                       

9) Re-add the state databases with the 'metadb' command.   Use the
     output of the 'metadb' command you saved to get the locations.

               metadb -a -f
               metadb -a -f
               metadb -a -f 

10) Recreate the "md.tab" file from the 'metastat' output you saved (or
     from memory), except make sure that all mirrors to be one-way
     mirrors and all RAID5 devices contain the "-k" option.   For the boot
     disk, ensure that the one-way mirrors refer to the side which has
     been restored.   Make sure the order is correct so that mirrors aren't
     created until the submirrors already exist. See below the necessary
     changes that must be made to ensure one-way mirrors are used.

     The "md.tab" file exists either in the /etc/opt/SUNWmd or /etc/lvm
     directory, depending on the version of DiskSuite you are running.

     For example:

           vi md.tab

           ** EXAMPLE FILE BEFORE:
           -------------------------------------------------
           d0 -m d10 d20                       <-- see the two-way mirror!
           d10 1 1 c0t0d0s0
           d20 1 1 c1t0d0s0
           d1 -m d11 d21
           d11 1 1 c0t0d0s1
           d21 1 1 c1t0d0s1
           d2 -m d12 d22
           d12 1 1 c0t0d0s6
           d22 1 1 c1t0d0s6
           d3 -m d13 d23
           d13 1 1 c0t0d0s7
           d23 1 1 c1t0d0s7
           d4 -m d14 d24
           d14 1 1 c0t0d0s4
           d24 1 1 c1t0d0s4
           d5 -m d15 d25
           d15 4 c0t1d0s2 c0t2d0s2 c1t1d0s2 c1t2d0s2
           d25 4 c0t3d0s2 c1t3d0s2 c1t4d0s2 c1t5d0s2
           d6 -r c1t6d0s2 c1t8d0s2 c1t9d0s2
           -------------------------------------------------

           ** EXAMPLE FILE AFTER:
           -------------------------------------------------
           d10 1 1 c0t0d0s0
           d20 1 1 c1t0d0s0
           d0 -m d10                               <-- see the one-way mirror!
           d11 1 1 c0t0d0s1
           d21 1 1 c1t0d0s1
           d1 -m d11
           d12 1 1 c0t0d0s6
           d22 1 1 c1t0d0s6
           d2 -m d12
           d13 1 1 c0t0d0s7
           d23 1 1 c1t0d0s7
           d3 -m d13
           d14 1 1 c0t0d0s4
           d24 1 1 c1t0d0s4
           d4 -m d14
           d15 4 c0t1d0s2 c0t2d0s2 c1t1d0s2 c1t2d0s2
           d25 4 c0t3d0s2 c1t3d0s2 c1t4d0s2 c1t5d0s2
           d5 -m d15
           d6 -r c1t6d0s2 c1t8d0s2 c1t9d0s2 -k
           -------------------------------------------------

11) Run the 'metainit' command to create all the metadevices that are
     listed in the "md.tab" file.

               # metainit -f -a

12) Run the 'metaroot' command to set the metadevice as a root device.

               # metaroot d0

13) Add in the other metadevices into the /etc/vfstab file for the
     other root filesystems as well as the other metadevices that you
     created in step 7.

14) Reboot.

15) You will now attach all the second mirrors to all the mirrored
    metadevices. For example:
	# metattach d0 d20