http://www.virtualizationadmin.com/articles-tutorials/vmware-esx-articles/general/how-to-access-the-vmware-esxi-hidden-console.html

Good video on administering ESX from service console:

http://www.ditii.com/2010/02/25/10-basics-of-linux-to-manage-vmware-esx-through-service-console-webcast/

CCNA labs (with topology diagrams): http://www.howtonetwork.net/public/department140.cfm

cdrecord -v –dev=4,1,0 driveropts=burnfree -dao

run

cdrecord –scanbus to get the address of the CD/DVD drive.

Chorus:
Oh, the precious blood
That flowed from Mercy’s side
Washed away my sin
When Christ my Savior died
Oh, the precious blood
Of Christ the crucified
It speaks for me before Your throne
Where I stand justified

Verse 2:
And who am I that I should know
This treasure of such worth
My Savior’s pure atoning blood
Shed for the wrath I’d earned
For sin has stained my every deed
My every word and thought
What wondrous love that makes me one
Your priceless blood has bought

Bridge:
A crown of thorns, pierced hands and feet
A body bruised, and Mercy’s plea

Let’s face it, your system will become unpredictable at some point. How the system is handled in this state makes a huge difference in my mind. Many common practices I have witnessed in use actually reduce predictability. It is generally not a good idea to make ANY changes to a system that is acting in an unpredictable manner. The fact that it is doing something unexpected means that something is happening under the hood that you don’t yet understand. Computers have no ‘mind of their own’ and definitely do not have ‘mood swings’. A lack of understanding can lead to fear and superstition, and will usually end up in the practice ’sysadmin voodoo’.

The ‘uninstall/reinstall’ ritual is often practiced. It may work on some operating systems, but for applications that run on UNIX this usually causes more problems than it fixes and puts the system into a state that is even harder to understand.  How would you like if your doctor uninstalled you in attempt to cure some disease that he didn’t have the right information in order to diagnose? Well, you wouldn’t care because you would be dead. Although there is not an equivalent of dead for a computer system, you can see that this method leads to an unnecessary amount of work, keeps the system in a down state longer, and the reinstall may actually result in a configuration that is different from what was there before. Why would anyone do this?

Another voodoo ritual is the the ‘wheel-of-fixes’ . This involves making a best guess as to what will correct the problem based on the symptoms. Would you like to be operated on that way? Oh, I will remove your tonsils now because your throat is sore. This approach can end up breaking something that was working, introduce new problems, and may not even solve the original issue.

So how you you handle an unpredictable system? Troubleshooting. There is a saying, “Troubleshooting is not so much about knowing what it is you need to fix as it is about knowing what NOT TO FOOL WITH!”. It sounds silly at first, but if you think about the question it raises it starts to make sense. How can you tell what to fix and what not to fool with? Well, it means you must have the right information at your disposal when you are troubleshooting. Doctors use very specialized equipment and methods to troubleshoot, and so it should be with sysadmins. Without the proper tools and methods your job will be a nightmare and you, too, might resort to occult practices.

Over the years I have come up with a system of tools and methods that WORKS, and works well.
I call it VIDCAPS (Visibility, Intelligence, Documentation, Communication, Automation, Processes, and Standards):
V: visibility. Do you know the current running state of your systems? Let’s face it, you won’t drive your car if the instrument cluster was not there, would you? You would not be able to tell how fast you were going, if you were about to run out of gas, if the engine was overheating, etc, etc. I will call anything that provides real-time information to you a dashboard, so ‘top’ is a dashboard for the purposes of this discussion. With proper visibility you will be able to better predict what your system would NOT do next. I find it interesting that anyone would operate any type of this equipment without the proper instrumentation.

One of the things on your car dash is an odometer. You look at this to tell when to change the oil, although the newer models will tell you when service is due. BTW, I want automakers to design the thing to drive into the shop by itself and have a robot service it With the proper visibility you will be able to know when certain maintenance tasks need to be performed.

One challenge of using the built-in dashboards is that they are generally on the machine itself. What if you have bunches of servers? Who has the time to read an email from every system every morning, or login to each control panel to see what is going on?
Another problem is that the dashboards provide general information and they may not be customizable. Besides, it is not a good practice to mess with system-supplied tools. You may at some point want to see how many of a certain process or network connection type are active.
Some cars have a tachometer which is nice, but most drivers don’t really care about engine RPMS. The existing dashboards might be ‘cluttered’ with stuff you really don’t care about.
Also, most system supplied dashboards give point-in-time data only. What if you wanted to see a graph of your /data mountpoint over the last month?

There are often business-specific metrics that you may want to have available so I look for every opportunity to add a new gadget to my dashboard, especially when I’m troubleshooting a certain problem and I’m looking for a certain piece of information that is not already on a dashboard.

The answer is Monitoring. For cost-conscious businesses, there are several free monitoring tools. Hyperic is one I have used and it works very well. They have an Enterprise version that offers better features like the ability to alert on a combination of metrics, but at least try the free one to see how much of your needs it will meet. Another thing I like about Hyperic is that it is very customizable. You will most likely have very business-specific metrics that you need to monitor. I will be writing a separate article on custom plugins for the Hyperic server.

An import part of monitoring is Change Assessment. Do you know what files are being changed on your system on a day-to-day basis? Tripwire Enterprise is well worth the money, because on production systems you want to know exactly what has changed. If setup correctly, Tripwire will also ‘version’ the changed files, so you will not only be able to see what about a file has changed, you will have the option to put it back the way it was before. Change Assessment and Change Control (to be discussed in the Communication section) need to tie together so you can ‘reconcile’ the changes. Before making any changes, submit a Change Request. Somewhere in the Change Request should have a list of files that you intend to modify, and, if it is possible for you to predict, the MD5 or SHA1 hash on that file after it is deployed to the server. Next day when the change report comes in from Tripwire you can match what the servers saw to what you intended to do.

Although more related to Documentation (discussed below), make sure Tripwire is saving the content of your production-tested configurations. Out of the box, Tripwire may not be collecting and versioning the information you may need to quickly restore your system back to the Last Known Good state. Tripwire has a database module that can version database objects, and it is really not that expensive. I highly recommend this add-on for any production database.

There is an open source version of Tripwire, but for production systems I recommend using the real deal.

I: intelligence. So you have a dashboard in your car. There is a little needle thingie that gives the engine temerature. It is supposed to stay under the red band. My wife has driven her car until smoke started pouring out from under the hood, because she did not notice that needle thingie creeping into the red. Well, it is because her car does not have any other means of communicating engine temperature. In her defense, I did once run out of gas at 4AM! Most of the new models will start ringing a dingbell and lights will start flashing on the dashboard. This is Intelligence. Hyperic allows you to set thresholds on a metric, and can generate an alert when that metric is exceeded. Even the free version permits very sophisticated rule creation, although it lacks the ability to set an alert based on a combination of metrics.

This will improve both stability and performance because you will be able to head off problems before they bring the system to a crawl, or worse yet, causes the system to crash.

D: documentation. What is documentation? Is it just having an article describing something, or does the accuracy of that description make it Documentation? When performing a build or install, keep notes as you go along. When you’re done, create a document listing the steps, then  go back and perform the build again, using that document. This way you can be certain that doc is valid. Although existing documentation may not be 100% accurate, it is still a good thing. Try as much as possible to validate any existing documents, updating as necessary.

Code Versioning falls into this category. I use subversion and websvn. Although there are enterprise code visualization tools, websvn works well for small code files, and it is free. I always put the SVN checkout link in my scripts as well, so when someone else sees it on the server they know where to find it in the repository to make updates if necessary. It would also be a good thing to create a process (to be discussed shortly) whereby the scripts are not modified on the system itself, but checked out of SVN, modified, tested and committed, then deployed on the system.

C: communication. Have you ever had someone do something that you wish they had asked you about first? Make sure that all persons responsible for a system, and all users of that system are informed of ANY changes you are going to make. Change Control handles this piece of the puzzle. I found an excellent free web-based tool Brage that works very well.

I have a habit of running ‘w’ as soon as I log into a system anyway. I want to see who else is there. I can communicate with other users on the system using ‘write’ so we don’t step on each others’ toes.

Communication to me also means doing what you say, when you promised, and in EXACTLY the same way you told your users. I hate unpleasant surprises, and for that reason I do not impose it on others.

A: automation. I tell folks that I’m a UNIX Systems Administrator and they say, “so you work on computers”. I don’t see myself as someone who “works on computers”. I have computers that work for me. I look for every opportunity to script things so I don’t have to do them. There are several benefits of automation. Firstly, I’m lazy. I prefer not to work, so if I can make the system take care of itself, I can sit back and drink coffee (just kidding ). Secondly, I make mistakes, so I don’t really trust myself to perform a task that might involve several commands in a row. What if I miss a step or mistype a command or argument? Third is that I get bored and tired easily. I don’t really care to do the same thing over and over. Fourth is I need to sleep. Who wants to be working at 3AM? Fifth is that I forget things often. Didn’t remember your wife’s birthday anyone?

By automating tasks that can be automated you will dramatically improve manageability by having more stuff run in maintenance windows, and of course, get more sleep

Automation, however, has its perils. Understand first what is required, create a plan to achieve that goal, keeping in mind the things that can go wrong, then start to script. Always test the script thoroughly by simulating every condition that will cause it to branch

P: Processes. There are some things that you just can’t automate, so a person has to do it. Well, do you do it the same way Joe does? How about Harry? Before you create a process, you should ask, “Is there a recommended way of doing this?” I will address this next in talking about Standards.

Using the same methodology as automation, write out a ’script’ for the process. Create a checklist for this process. This will ensure that everyone does the same task the same way every time. This will improve the manageability of your systems because every piece of every system that is supposed to be the same will be the same, and various iterations of that task on the same system will result in a predictable outcome.

S: Standards and processes are somewhat alike, but different enough in my mind to warrant a different discussion. Standards, Best Practices, and Vendor Recommendations are all lumped together here. The opposite to this approach, doing-it-your-way puts the systems in a state where it is difficult for anyone to understand. Although this practice may not be meant to cause harm, it usually does. I spoke on this above: don’t create a process for which there is already a well established standard.

Besides the manageability problems, following the established standards will help the security and stability of the systems. It is often the case that businesses have ’strange’ problems on their systems that can easily be corrected by following one or more established standards.

Conclusion:
So why did I not mention backups? Well, this was a discussion on manageability, security, stability, and performance.  Besides, if you are not already performing backups you must not care enough about your system, so VIDCAPS is going to be a waste of your time!

Look out for an upcoming articles on Hyperic installation, setting up alerts, and writing custom plugins.

External Links
Human Side of System Administration

Sysadmin Voodoo

Parm Patram is a 10 year Linux/Solaris veteran with a proven track record of improving manageability, stability, security and performance.

He has proven ability to improve manageability, stability, security and performance. His analytical, troubleshooting, and creative abilities allow him to adapt to any circumstance quickly, accurately pinpoint root cause, and come up with unique solutions.

Skills Summary
* Oracle 10g/11g
* Recovery Manager (RMAN)
* User and Security Administration
* Export/Import (exp, expdp)
* Logminer
* Streams Replication
* Dataguard Standby Database
* Oracle Enterprise Manager (OEM)
* Oracle EBS
* Performance Tuning
* Materialized Views (Snapshots)
* PL/SQL, sqlplus
* Real Application Clusters (RAC)
* mySQL
* postgresql
* Unix Shell Scripting (bash, ksh)
* PHP/Perl, expect, C Programming
* Code Versioning (svn, fisheye)
* Linux/Solaris (kickstart, LVM/Disksuite RAID, NFS, samba)
* Installation/Configuration (rpm, up2date, yum, apt-get)
* Advanced Network Troubleshooting (ping, ipcalc, arping, tcpdump, nmap)
* Advanced Application Troubleshooting (strace/truss, netstat, ipcs, lsof, ps, jconsole)
* Monitoring (Hyperic, openNMS, SNMP)
* Migration (rsync)
* Change/Configuration Auditing (Tripwire, custom scripting)
* Change Control
* Automation (expect, Perl, C, shell)
* Apache/Tomcat

Work Employment History
Nov 2009 – Present: Linux/Solaris & Oracle Database Administrator at Barnhardt.
* Managed development, stage and production databases and application servers for large manufacturing company’s ERP.
* Install/configure Solaris 9 on Sunfire 280R. Developed method to clone the production server to backup/stage for consistency. Did some hardware work with RSC.
* Updated old documentation and wrote new articles in tech services wiki relating to Linux/Solaris/Oracle.
* Developed and implemented method to permit Oracle 10g Dataguard to compress archivelogs using SSH port forwarding (Linux).
* Wrote custom plugins in Perl/bash/ksh for Hyperic to monitor Oracle Streams, Oracle Dataguard, and certain specific metrics on the application server.
* Developed scripts using expect to test ERP functionality in an automated fashion.
* Helped establish Change Control process.
* Installed ERP on Linux (RHEL 5) for functional/development testing.

* Streamlined Dev refresh process by scripting much of the pieces and creating a checklist of the order to execute those scripts. Project time did not permit a fully automated approach, which was the next progression.
* Updated some of the scripts to use command-line arguments. This allowed the scripts to be exactly the same, but function differently based on arguments.
* Updated the ‘standby monitor’ to perform checks to eliminate false alerts.
* Developed ‘Tango’, which is a way to detect changes to an Oracle database’s settings or object properties. Tango consists of Oracle PL/SQL package and bash/Perl OS scripts.
* Developed method to resize the filesystem used for archive logging without needing to stop the database (Linux).
* Installed monitoring/auditing tools: Tripwire, openNMS, and Hyperic on RHEL 5 systems.
* Helped with securing the ERP platform.
* Identified/corrected performance problems resulting from over-sized SGA. Implemented SQL Profiling to improve performance of certain problem queries (Oracle).

Aug 2005 – Nov 2009: Senior Linux/Oracle Database Consultant at Business Technology Services, Inc.
* Specialized in RHEL, Oracle 10g and Oracle EBS.
* Managed company’s dedicated Linux server which services their website and email needs.
* Worked for many clients of BTS, Inc. in various industries.
* On call support.
* Oracle Dataguard standby database implementation, troubleshooting and tuning.
* Oracle Streams replication.
* General Troubleshooting and problem resolution.
* Custom monitoring for Linux OS and Oracle database.
* Oracle database performance tuning.
* Oracle RAC on Linux (RHEL) implementation and monitoring.
* Oracle EBS (11i and 12i) patch investigation and installation.
* Oracle EBS cloning.

Jan 2000 – August 2005: Senior Linux Administrator at Interland, Inc (now Peer1).
* Worked in Dedicated Server Solutions department which offered billable Admin services to company’s Dedicated Server customers.
* DSS also provided Data Center Operations support for company’s Shared Servers in addition to the Dedicated Servers (5000+ total).
* Tier III support for Dedicated Services (non-billable questions and issues).
* Assisted Engineering with build issues by writing work-around scripts and providing diagnostic data.
* Was the leading revenue earner in DSS.
* Some hardware work (HP/Compaq Proliant, Cobalt, white-box).
* Plesk Linux Control Panel SME. Wrote many scripts to manage and migrate Cobalt RAQ, systems with no control panel, and older versions of Plesk to the newest Plesk versions.
* Instituted Clean, Patch, &Lockdown service for customers whose servers had been hacked into.
* Invented ‘Backplane Sharing’ to retrieve data from RAID array on non-booting servers by using the dead system to power up the RAID array so it can be mounted to a running machine in order to perform a data copy.
* Developed unique method of reviving badly hacked servers that fail to boot by booting from CD, chroot, then manual services startup. Used as a stop-gap until a full migration to a new server could be effected.
* Provided excellent sales support for sizing requirements and other technical consulting required for completing the sale.

Miscellaneous Freelance Work
Omega Learning Centers (http://omegalearningcenter.com).
Administer company’s web server and email server.  Maintain and add new functionality to company’s suite of proprietary web-based interfaces used for running their business. Instrumental in scaling the products out to meet their need to franchise. Company now has 3 locations and growing.

3D Cellular (http://omegalearningcenter.com).
Administer company’s web and email server. Manage server firewall.  Responsible for the SSL certificates in use for both the website and email. Implemented VPN between LAN and Internet server. Designed custom scripts for backing up customer’s project files from their local area network to the server as well as designing and implementing an off-site backup plan for the server itself.

Kagan Creative (http://kagancreative.com).
Server administration and migration. Built and configured a dedicated mySQL server for client. Also responsible for about 10 SSL certificates for various websites running on the web server. Configured 3-node load-balanced web server for client.

HPP Christian Hosting  (http://hppchristianhosting.com).
Administered 3 high-traffic and heavily populated web servers totaling about 1000 different websites. Identify and upgrade old vulnerable applications running on the various websites that were being hacked into. Migrated the servers a few times to keep up with growing demand and the need for faster hardware. Assisted with some handover work when client sold the business in September 2007.

Certifications
Oracle Certified Professional (Database and Grids), 2009
Solaris Certified System Administrator, 2003
Microsoft Certified Systems Engineer, 1999

Education
1990, Associates of Arts (Data Processing)

References
Available Upon Request