Pages

Showing posts with label ILOM. Show all posts
Showing posts with label ILOM. Show all posts

Tuesday, November 6, 2018

Remotely Login & Run Commands on ILOMs

Logging into Oracle's Integrated Lights Out Manager (ILOM) to get info can be a real pain, so I wrote this script to do it for me. Normally one would use use Simple Network Management Protocol (SNMP) or Intelligent Platform Management Interface (IPMI), but due to security concerns I was not able to use either of these options. Even with the latest firmware installed the ILOMs would not support modern security practices. So I was forced to find anther way. I needed to write a script that would wait for a prompt and then fill it in for me. Expect an extension to the Tcl scripting language is great for this kind of stuff, but I decided to use HERE which is even easier.

In order to make this work I created the user mancnt on the local system and on all the ILOMs. I also created a SSH key and setup an SSH agent on the local system and then I copied the key over to the ILOMs. If you don't know how to setup SSH keys check out my last post on how to do it  "A Better Way to Setup SSH Keys". You will also need a file containing the hostnames of the ILOMs you want access. In the example script below I use two such files, lsILOMb and lsILOMc, one for the blades and one for the chassis.


#!/bin/bash
#
# This section is for the ILOM blades
 HERE-ILOM(){
ssh $1 2>/dev/null <show /SP/network macaddress
HERE
}
# This section is for the ILOM Chassis
HERE-ILOMc(){
ssh $1 2>/dev/null <show /CMM/network macaddress
HERE


# To get IP address from hostname
Ping-to-IP(){
ping -c1 $1 |grep PING|awk '{print $3}'|sed -e 's/(//' -e 's/)//'
}

# Main section
ps aux|grep manacnt|grep -v grep |grep agent &>/dev/null || echo "Need to have an agent running"

# Section for ILOMs on Oracle Blades
for s in $(cat lsILOMb)
do echo -e "$(Ping-to-IP $s),$(HERE-ILOM $s),Embedded Linux,$s"
done

# Section for ILOMs on Oracle Chassis
for s in $(cat lsILOMc)
do echo -e "$(Ping-to-IP $s),$(HERE-ILOMc $s),Embedded Linux,$s,FALSE,ILOM,N611"
done

So the script generates a comma-separated values (CVS) file, which contains the IP address, MAC address, OS, and hostname. I then give this file to the network security people.

Example output: 10.0.1.20,00:10:e0:40:c2,Embedded Linux,server-ilom

If you have any questions feel free to ask them below.


Friday, December 12, 2014

Clear fmadm or FMA fault logs

If the System Administrator doesn't have log rotation setup, logs fill up on the server from time to time. Usually the system or audit logs are the root cause, but sometimes the Fault Management Architecture (FMA) logs are the cause. On Solaris or if installed on Linux, there is a utility called fmd (Fault Manager Daemon) that checks for issues with the system hardware. This program is managed by the fmadm program. Follow the steps below to clear out the logs.

Clear error fmadm reports
root@earth> fmadm repair  UUID
The UUID is the event code that is shown when you run the fmadm faulty command.

* Note: If the above command doesn't clear the error then, there is a real issue hardware that needs to be addressed.

Clear reports and resource cache
root@earth> cd /var/fm/fmd
root@earth> rm e* f* c*/eft/* r*/*

Clearing out FMA files without rebooting.
root@earth> svcadm disable -s svc:/system/fmd:default
root@earth> cd /var/fm/fmd
root@earth> find /var/fm/fmd -type f -exec ls {} \;
root@earth> find /var/fm/fmd -type f -exec rm {} \;
root@earth> svcadm enable -s svc:/system/fmd:default


Reference
blogs.oracle.com: How to clear fmadm log or FMA faults log
Fault Management Architecture

Manpage:
fmadm(1M)
fma

Tuesday, April 8, 2014

Setting up Email Alerts for an ILOM

This is how to setup email alerts an Integrated Lights Out Manager (ILOM). This is a two set process,  first step is setup the SMTP client and then step up the Alerts.

Part I Setting up SMTP 

The SMTP client sends the email. There are two ways to get there in the web ILOM.
1) Go to the configuration tab and then the SMTP sub tab, to get to the SMTP setting.
2) On newer ILOMS go go to menu on the right and expand the ILOM Administration menu. Then click on Notifications. In the main area click on SMTP Client.


Make sure that the SMTP State is checked and add the IP or hostname of the SMTP server.
Put something meaning full in the Custom Sender field. This field sets the from field in the email that is sent to you. In the example above, if the servers name is mars the the email will come from alert@mars. Don't forget to save.

Not shown above, enter your email into the last field and press send to send a test email. Make adjustments as you see fit. This set is optional and can be skipped.


Part II Setting the Alert

Click on the alert tab to get to the alert area.


Click on the radio button and click on the Edit button to edit or create a rule. A pop up window will appear similar to one below.



Change the Level to Major or Minor. Set the Type drop down to Email. Put in your email in the Email Address field and your done. None of the other fields are needed.

Please comment below if you have anything to add.



Wednesday, December 4, 2013

How to get an ILOM Commmnad Line Console

I had an issue the other day on an Oracle x86 blade server. I wanted to access the the system command line system console on a x86 blade. There are two ways to access the console on a ILOM. You can use the Java web console or the command line console. Now on the SPARC systems could always use both consoles, but the x86 servers could only use the Java web console.  So we where forced to web console on the x86 servers. I could just use the ILOM's Java web console, but not all servers have a web browser and Java installed. Below, I have posted my notes on how to make the command line console work for the x86 ILOM.

Run the eeprom command to see your current settings.  If the console setting is set to text then you must change it to ttya.
root@earth> eeprom
ata-dma-enabled=1
atapi-cd-dma-enabled=0
ttyb-rts-dtr-off=false
ttyb-ignore-cd=true
ttya-rts-dtr-off=false
ttya-ignore-cd=true
ttyb-mode=9600,8,n,1,-
ttya-mode=9600,8,n,1,-
lba-access-ok=1
prealloc-chunk-size=0x2000
keyboard-layout=US-English
console=text

Change the setting with the command below.
root@earth> eeprom console=ttya

The server must be rebooted for the change to take affect.

After the reboot test to see if it works.
root@earth> ssh root@ILOM
-> start /SP/console


Reference
Document ID 1448462.1 on the Oracle Support Site.
My Oracle Support
 

Friday, November 1, 2013

Show Faulted Hardware in ILOM

Here, I will go over my notes on how to identify and clear hardware faults, in an ILOM (Integrated Lights Out Manager). On this page I will use the example of a chassis fan module error. If you follow my notes and the error clears Then you didn't have a real issue. On the other hand, If after following my notes you can't clear the error. Then you have a real hardware issue. You can't clear errors if the error is still an issue.

This is how you login to the command line interface for the ILOM.
man@earth> ssh root@ilom

The command below is one way to show system faults. The only target you should see is shell. If you see anything other then shell it is a fault. In the example below, the ILOM shows a bad system fan. Shown as 0 (/SYS/FMO).
--> show /SP/faultmgmt

/SP/faultmgmt
     Targets:
          shell
          0 (/SYS/FM0)

      Properties:

      Commands:
          cd
          show

Using the show faulty command is anther way to see the system faults. This command shows a lot more detail. If you have a support contract with Oracle, you will want to paste the output of this command into the ticket, you submit to MOS. The show faulty command can be used without any paths, which will be extra useful if are coming in from a chassis ILOM.
--> show faulty
Target                    | Property                   | Value
-----------------------+--------------------------+---------------------------------
/SP/faultmgmt/0    | fru                            | /SYS/FM0
/SP/faultmgmt/0/   | class                         | fault.chassis.device.fan.fail
faults/0                  |                                  |
/SP/faultmgmt/0/   | sunw-msg-id            | SPX86-8X00-33
faults/0                  |                                  |
/SP/faultmgmt/0/   | component               | /SYS/FM0
faults/0                   |                                 |
/SP/faultmgmt/0/   | uuid                          | 8692c3e4-G481-635e-f8e2-f3f215d1
faults/0                   |                                 | 13f0
/SP/faultmgmt/0/   | timestamp                | 2013-10-02/12:10:43
faults/0                   |                                 |
/SP/faultmgmt/0/   | detector                   | /SYS/FM0/ERR
faults/0                   |                                  |
/SP/faultmgmt/0/   | product_serial_number | 1203FMM107
faults/0                   |                                  |

The command below shows the event log, which will also contain the system hardware errors.
--> show /SP/logs/event/list

To clear the hardware fault from the logs run the command below.
--> show /SP/logs/event/ clear=true

Run this command to clear the fan error.
--> set /SYS/FM0 clear_fault_action=true
Try to clear the hardware fault. If the hardware is really having an issue, the hardware fault will come back. In about a minute or less. If you can't clear the error and you have a support contract then this is when you summit your ticket.

If you have any questions or I missed something let me know.

Thursday, August 1, 2013

Java PATH need to run the ILOM Remote console

From time to time there is something you only set once. For example Oracle's Integrated Lights Out Manager (ILOM) has a web interface. Which has remote console feature. This feature uses Java running in a web browser. The first time it asks for the path to launch the program . Below I have displayed  the path the Java needs for the remote console feature.


For UNIX & Linux computers:  /usr/bin/javaws
For Windows computers: C:\Program Files\Java\jre\bin\javaws 

I hope this helps someone.

Monday, April 15, 2013

Make Firefox load ILOM pages, Part III

This yet anther way to make Firefox load the ILOM web interface properly. Posted below is a script my co-worker wrote. Basically it adds the content to the userContent.css file via this script. This way you don't have to edit the file manually, like you had to in my other post "Make Firefaox load ILOM pages".




export PROFILE_IDZ=$(grep Path= $HOME/.mozilla/firefox/profiles.ini | awk -F={`print $2`})
export FILE4FIXZ-"~/.mozilla/firefox/${PROFILE_IDZ}/chrome"

mkdir -p ${FILE4FIXZ}
touch ${FILE4FIXZ}/userContent.css

echo "@media print {" > ${FILE4FIXZ}/userContent.css
echo "}" >> ${FILE4FIXZ}/userContent.css
echo " " >>  ${FILE4FIXZ}/userContent.css
echo "@namespace url (https:www.w3.org/1999/xhtml);" >>  ${FILE4FIXZ}/userContent.css
echo "#mainpage { visibility: visible !important; }" >>  ${FILE4FIXZ}/userContent.css

cat  ${FILE4FIXZ}/userContent.css


If you have comments please post below.

Thursday, March 28, 2013

Change the ILOM IP address

From time to time I have to change the IP addresses on the ILOM (Integrated Lights Out Manager). So I decided to post my notes on how to do this and maybe someone will find it useful. There is also anther older hardware manger called ALOM (Advanced Lights Out Manager), this not the same. They both the provide console access to your server. For these don't know, console access is when you get remote access to a server, as if you where right next to it. So basically you see everything as if you where physically there. Note that this post changes the IP address for the MGT (management) port. This doesn't change the servers IP address for the OS installed on the hardware.

If this is the first time anyone has logged in then the ILOM will be set to the defaults. The default username is root and the default password is changeme.
# ssh root@server-ILOM
Are you sure you want to continue connecting (yes/no)? yes
Password:

For most severs with ILOM just put in the commands like you see below in the example. -> cd SP/network
-> set pendingipaddress=10.1.0.10
-> set pendingipgatway=0.0.0.0
-> set pendingipnetmask=255.0.0.0
-> set pendingipdiscovery=static
-> set commitpending=true

To check you work or see what your IP settings are, type ls or use the command below
You can see your IP address from any place in the ILOM by running the command below. -> show /SP/network 
To change the IP address of the chassis or CMM that hold the blade servers. Login and go to /CMM/network like in the example below and run the same commands as above. -> cd CMM/network
You can change the IP to the blades from the chassis ILOM as well. -> cd CH/BL#/SP/network

If you have any comments or questions please post them below.

Friday, January 4, 2013

Make Firefox load ILOM pages, part II

Fig 1. ILOM Certificate error
This is part two of my Make Firefox load ILOM pages. The first post was how to get Integrated Lights Out Manager (ILOM) to load right using Firefox. Part one is focused on getting the ILOM to display properly on a blade or a normal server. This post focuses on getting the ILOM page for the chassis to work.


The issue is when you have logged into the ILOM on the chassis and then try to connect to a individual blade. Figure one shows the error that comes up when you try to connect. Basically the certificate is not excepted by Firefox because it is self-signed.

Fig 2. The fix


This happens when you connect to the blade directly, via the ILOM interface. The issue here is that the button that usually appears that allows you to except the certificate is not there. To get it to where you can load the certificate you need to load the the page outside of the frame. You do this by right clicking on the frame. In the menu select. This Frame and click on Show Only this Frame.


At this point the frame will appear outside the frame. Accept the certificate to load the blades LOM page. To get the frame back go back 2 pages in Firefox. The ILOM page should look like figure 3 when your done. Do this for each blade you need access too.
Fig 3. Finished product

Why does this matter? If you can access the blades ILOM directly then accessing them though the chassis may not be an issue for you. But lets say you don't have your blade cabled up. Many work places don't allow you to connect servers to the network that have not been hardened first. By having the a way to get to the server remotely via the ILOM chassis you can build the server in place and then connect the server to the network after you harden the blade. It is also a good practice to have the ILOMs on there own network.

I can't take credit for this fix. I got this fix from My Oracle Support which means that this is the oracle supported fix. I hope this helps someone. Please feel free to comment below.




Thursday, October 11, 2012

Make Firefox load ILOM pages

I had this issue where Firefox wouldn't load ILOM pages right. I've also noticed that I'm not the only one with this issue. So I'm posting this little how-to for people who are still struggling with this. Basically the content section of the ILOM webpage will not display.

For those who don't know ILOM stands for Integrated Lights Out Manager. It is a web interface that helps you remotely manage servers. This interface is good for monitoring hardware issues and can can send out SNMP traffic. The ILOM can also give you console access. Meaning that you get a console or window that stays connected even during a reboot. Basically it is as if you are physically standing in front of the server with a keyboard and monitor.

Each user will have to add the following file to their home directory.
In ~/.mozilla/firefox/profile_id.default/chrome add a file called userContent.css@media print {
}

@namespace url(http:www.w3.org/1999/xhtml);
#mainpage { visibility: visible !important; }

Note - The profile_id.default will be the only file with .default at the end in the firefox directory. You may have to create the chrome directory.

I originally posted this fix at the forum linked to below, under the user name cyberninja.
https://forums.oracle.com/forums/thread.jspa?messageID=10283552
I also provided this fix to Oracle tech support and if you put in a trouble ticket to My Oracle Support this is the solution they will provide you. So in other words, this is the Oracle supported fix.

I have a second part to this post, where I fix a connection issue between the chassis and their blades.