Core file administration

This is the defaults:

     global core file pattern:
     global core file content: default
     init core file pattern: core
     init core file content: default
     global core dumps: disabled
     per-process core dumps: enabled
     global setid core dumps: disabled
     per-process setid core dumps: disabled
     global core dump logging: disabled

This is what I like to set it to:

     global core file pattern: /var/cores/core_%n_%f_%u_%g_%t_%p
     global core file content: default
     init core file pattern: core
     init core file content: default
     global core dumps: enabled
     per-process core dumps: disabled
     global setid core dumps: enabled
     per-process setid core dumps: disabled
     global core dump logging: enabled

Reasoning:
1. I don't like my core files all over the place
2. Easier to find and clean up.
3. I only enable per-process core file dump when a user needs it (on a zone)

VALUE		HOW TO SET IT
global core file pattern	Specifies name and location of the global core files (A leading "/" specifes an absolute path)	coreadm -g /var/cores/core_%n_%f_%u_%g_%t_%p
global core file content	Defines content of global files
init core file pattern	Specifies name and location of the global per-process files (No leading "/" makes it relative to working directory)	coreadm -p core
init core file content	Defines content of per-process files
global core dumps	Enable/disables global core-dumps	coreadm -e global
per-process core dumps	Enable/disables per-process core-dumps	coreadm -d process
global setid core dumps	Enable/disables global setid core-dumps	coreadm -e global-setid
per-process setid core dumps	Enable/disables per-process setid core-dumps	coreadm -d proc-setid
global core dump logging	Enable/disables logging to syslog	coreadm -e log

Installing ASR

Auto Service Request (ASR) is a secure, scalable, customer-installable software feature of Oracle Premier Support for Systems and Oracle/Sun Limited Warranty support that provides auto-case generation when specific hardware faults occur. ASR is designed to enable faster problem resolution by eliminating the need to initiate contact with Oracle for hardware failures, reducing both the number of phone calls needed and overall phone time required. ASR also simplifies support operations by utilizing electronic diagnostic data. If your qualified system is under warranty or covered by a service plan, you are entitled to ASR at no charge.

 I've been using ASR on my SUN storage for years. And it works brilliantly. Great to come into work in the morning and receive emails that a disk failed and also that a replacement disk is on its way. That ASR worked a little different - the new implementation needs an ASR server installed. Here are the steps I followed to install ASR:

1. Created a zone for ASR - I put everything in zones. Bitches love zones. I had a warm standby machine in my DMZ so I used that.
2. Download the software logging into support.oracle.com and searching for Oracle Document : 1185493.1. To install ASR, you need to install OASM (Oracle Automated Service Manager) first. This doc will give you the link to both ASR and OASM.
3. Install OASM
• unzip Oracle_OASM_150_SOLARIS64.zip
• pkgadd -d SUNWsasm-1.5.0-20130731121217.pkg
• (The next two steps I did because of errors messages during the install so maybe do them before the install.)
• pkg install SUNWcar SUNWkvm
• pkg install pkg:/system/management/service-tag@1.1.5-0.175.1.0.0.23.0
4. Install ASR
• unzip Oracle_ASR_481_SOLARIS64.zip
• pkgadd -d SUNWswasr-4.8.1-20140415212449.pkg
• export PATH=$PATH:/opt/SUNWswasr/bin:/opt/SUNWsasm/bin
• put line above in .profile
• Run: asr
• asr> register
• Options are all all easy - just make sure your machine can get to transport.oracle.com
5. Make sure you other machines can talk to ASR server
1. First way of doing this:
• asr> enable_http_receiver -p 8234
2. Second way of doing this:
• Edit /var/opt/SUNWsasm/configuration/config.ini
  jetty.enable=true
  jetty.host=<ipaddress> (Don't use "localhost")
  jetty.http.port=8234
• Restart sasm: svcadm restart sasm
6. Edit /var/opt/SUNWsasm/configuration/config.ini and set java.exec property to point to your java binary. And then restart sasm with svcadm restart sasm.
7. Check that it all works with:
• asr> show_http_receiver
8. To connect Solaris 11 machines to your new ASR server, run:
• asradm register -e http://<ip_of_your_server>:8234/asr

And that's it! Next you'll want to log into support.oracle.com and make sure that all your servers' details are correct and what not.

Rolling back a Solaris boot environment

Boot environments have been with us since Solaris 10. In my experience it started out pretty buggy but in the last year or so its been stable. It really has revolutionized patching for me in terms of cutting down risk,time and effort.

In Solaris 11, boot environments are pretty much built into how you do patching. ("Patching" is a bit of a misnomer in Solaris 11 - "updating/upgrading" are more appropriate terms.)

Recently I had to roll back to a previous boot environment two weeks after the implementation. Rolling back is pretty easy (activating the previous boot environment and rebooting), however, you do lose changes you've made to the OS in the meantime. You can mitigate this risk by mounting the other boot environment and comparing files between the two.

Areas you should check are:

1. /etc/system and any other config files in /etc
2. Crons
3. Zone configurations
4. Files in root's home directory and other home directories.

These are just the basics, hopefully if you've made any major changes since then, you've documented it well enough to figure out how to re-implement then if needed.

Solaris Crash Dumps and Basic Analysis

If your Solaris system panics and reboots, it'll probably create a crash dump in /var/crash. You can also force a crash dump either online (using "savecore -L") or as part of a reboot (using "reboot -d").

Normally this is where I stop and upload the /var/crash/vmdump.0 file up to Oracle to find out what the problem was. However, you can do some basic investigations yourself using the following steps:

# savecore –f vmdump.0 /somedirectory

# cd /somedirectory

# mdb *0

mdb> ::status

mdb> ::panicinfo

mdb> ::stack

mdb> ::msgbuf

mdb> ::cpuinfo

mdb> ::ps

mdb> ::arc

mdb> ::memstat

(*If the vmdump file is called vmdump.1 then use 1 instead of 0 in the above steps)

Honestly, most of the output is Greek to me but it's nice to know in the off chance something makes sense.

Sys-Unconfig on Solaris 10 Branded Zone Doesn't Give Option to Change Hostname

As the title says, I was trying to do a sys-unconfig on a Solaris 10 branded zone and when I reboot and go enter the new config info, the hostname doesn't get asked to be changed. The system automatically configures the zonename as the hostname.

Some things I tried that didn't work:

1. Configure a NIC that doesn't exist
   Since most of the googled results pointed to pulling out the network cable, I tried the logical equivalent for a zone. However, configuring a non-existent NIC doesn't work, because the a zone won't boot when configured with a NIC that doesn't exist.  
2. Choose a NIC that does exist but is offline
   Same reasoning at point 1, still doesn't work though.
3. Configure an IP that doesn't exist on our LAN
   Same reasoning at point 1, still doesn't work though.
4. Do it with no NIC configured
   This actually works, the system asks me for a new hostname during the sysconfig. Unfortunately, it doesn't ask me for any of the subsequent network info. So when I add the NIC, I'll have to then do the subnet and DNS configuration manually. No thank you.  
5. Search for files that contain the hostname and delete manually
   Just in case the sys-unconfig was not deleting these files. Didn't work.
6. Set the bootargs to "noauto"
   Got this from an Oracle SR for when the hostname is being retrieved from the jumpstart server (which I don't have anyway). Didn't work.

What worked:

1. After the sys-unconfig and subsequent sysconfig completed, I edited the following files and rebooted the zone:
• /etc/inet/hosts
• /etc/nodename
• /etc/hostname.net0 (or whatever's relevant)

Empty root folder on Solaris10 branded zone

If you have a Solaris10 branded zone, and it is in an "unavailable" state, the root folder in the zone path will look empty. Scared the living daylights out of me! Just do an attach and the folder's contents will appear.

It might just be my understanding of how Solaris 11 does zones though. It looks like during the attach, there is another filesystem that gets mounted on root. I'll have to do some testing and prodding and understand what's going on.

SSH Issue on Solaris 10 Branded Zones

So suddenly one night I get a call from the night operators telling me that they can't ssh to a particular zone. Their ssh sessions are even dying. When I log in (via the global zone), the message log has loads of sshd core dumps!:

Jul 11 20:41:08 hostname genunix: [ID 603404 kern.notice] NOTICE: core_log: ssh[2791] core dumped: /var/core/core_hostname_ssh_14247_103_1373568067_2791
Jul 11 20:41:15 hostname genunix: [ID 603404 kern.notice] NOTICE: core_log: ssh[2813] core dumped: /var/core/core_hostname_ssh_14247_103_1373568074_2813
Jul 11 20:41:26 hostname genunix: [ID 603404 kern.notice] NOTICE: core_log: ssh[2889] core dumped: /var/core/core_hostname_ssh_14247_103_1373568085_2889
Jul 11 20:44:37 hostname genunix: [ID 603404 kern.notice] NOTICE: core_log: ssh[6711] core dumped: /var/core/core_hostname_ssh_14247_103_1373568276_6711
Jul 11 20:47:54 hostname genunix: [ID 603404 kern.notice] NOTICE: core_log: ssh[11121] core dumped: /var/core/core_hostname_ssh_14247_103_1373568473_11121
Jul 11 20:59:34 hostname genunix: [ID 603404 kern.notice] NOTICE: core_log: ssh[25061] core dumped: /var/core/core_hostname_ssh_14247_103_1373569173_25061

I try a couple of things, none of which seem to particular help but the problem goes away after about half an hour. Then it comes back a couple of days later. And then again. And then it happens on some other containers.

Of course by this time, my call logged with Oracle has been escalated to the highest level. They come back with this:

It seems at this point that you have bin hit by known issue.
Bug 15781192 - SUNBT7156478-SOLARIS_11U1 double free in kernelSlottable.c kernel_slottable_ini
This was fixed in the S11u1 release .. but now we have started a backport CR for S10. At this point the only workaround is to disable pkcs11 engine in the sshd_conf and restart ssh.

And then gave the complete workaround:

The complete workaround requires three steps to be executed inside the Solaris 10 branded zone: 1) Uninstall the pkcs11 kernel provider:  # cryptoadm uninstall provider='/usr/lib/security/$ISA/pkcs11_kernel.so' 2) Disable the pkcs11 engine for sshd  # vi /etc/ssh/sshd_config add the line "UseOpenSSLEngine no" to this file (without the quotes) 3) Restart the ssh service to pickup the change:  # svcadm restart ssh

EDIT: I updated to the latest patches. I'll have to take some time to reverse these workarounds and see if the problem has been fixed. I've been it told it has but I'll have to confirm for myself.