Rolling back a Solaris boot environment

Boot environments have been with us since Solaris 10. In my experience it started out pretty buggy but in the last year or so its been stable. It really has revolutionized patching for me in terms of cutting down risk,time and effort.

In Solaris 11, boot environments are pretty much built into how you do patching. ("Patching" is a bit of a misnomer in Solaris 11 - "updating/upgrading" are more appropriate terms.)

Recently I had to roll back to a previous boot environment two weeks after the implementation. Rolling back is pretty easy (activating the previous boot environment and rebooting), however, you do lose changes you've made to the OS in the meantime. You can mitigate this risk by mounting the other boot environment and comparing files between the two.

Areas you should check are:

1. /etc/system and any other config files in /etc
2. Crons
3. Zone configurations
4. Files in root's home directory and other home directories.

These are just the basics, hopefully if you've made any major changes since then, you've documented it well enough to figure out how to re-implement then if needed.

Solaris Crash Dumps and Basic Analysis

If your Solaris system panics and reboots, it'll probably create a crash dump in /var/crash. You can also force a crash dump either online (using "savecore -L") or as part of a reboot (using "reboot -d").

Normally this is where I stop and upload the /var/crash/vmdump.0 file up to Oracle to find out what the problem was. However, you can do some basic investigations yourself using the following steps:

# savecore –f vmdump.0 /somedirectory

# cd /somedirectory

# mdb *0

mdb> ::status

mdb> ::panicinfo

mdb> ::stack

mdb> ::msgbuf

mdb> ::cpuinfo

mdb> ::ps

mdb> ::arc

mdb> ::memstat

(*If the vmdump file is called vmdump.1 then use 1 instead of 0 in the above steps)

Honestly, most of the output is Greek to me but it's nice to know in the off chance something makes sense.

Sys-Unconfig on Solaris 10 Branded Zone Doesn't Give Option to Change Hostname

As the title says, I was trying to do a sys-unconfig on a Solaris 10 branded zone and when I reboot and go enter the new config info, the hostname doesn't get asked to be changed. The system automatically configures the zonename as the hostname.

Some things I tried that didn't work:

1. Configure a NIC that doesn't exist
   Since most of the googled results pointed to pulling out the network cable, I tried the logical equivalent for a zone. However, configuring a non-existent NIC doesn't work, because the a zone won't boot when configured with a NIC that doesn't exist.  
2. Choose a NIC that does exist but is offline
   Same reasoning at point 1, still doesn't work though.
3. Configure an IP that doesn't exist on our LAN
   Same reasoning at point 1, still doesn't work though.
4. Do it with no NIC configured
   This actually works, the system asks me for a new hostname during the sysconfig. Unfortunately, it doesn't ask me for any of the subsequent network info. So when I add the NIC, I'll have to then do the subnet and DNS configuration manually. No thank you.  
5. Search for files that contain the hostname and delete manually
   Just in case the sys-unconfig was not deleting these files. Didn't work.
6. Set the bootargs to "noauto"
   Got this from an Oracle SR for when the hostname is being retrieved from the jumpstart server (which I don't have anyway). Didn't work.

What worked:

1. After the sys-unconfig and subsequent sysconfig completed, I edited the following files and rebooted the zone:
• /etc/inet/hosts
• /etc/nodename
• /etc/hostname.net0 (or whatever's relevant)

Empty root folder on Solaris10 branded zone

If you have a Solaris10 branded zone, and it is in an "unavailable" state, the root folder in the zone path will look empty. Scared the living daylights out of me! Just do an attach and the folder's contents will appear.

It might just be my understanding of how Solaris 11 does zones though. It looks like during the attach, there is another filesystem that gets mounted on root. I'll have to do some testing and prodding and understand what's going on.

SSH Issue on Solaris 10 Branded Zones

So suddenly one night I get a call from the night operators telling me that they can't ssh to a particular zone. Their ssh sessions are even dying. When I log in (via the global zone), the message log has loads of sshd core dumps!:

Jul 11 20:41:08 hostname genunix: [ID 603404 kern.notice] NOTICE: core_log: ssh[2791] core dumped: /var/core/core_hostname_ssh_14247_103_1373568067_2791
Jul 11 20:41:15 hostname genunix: [ID 603404 kern.notice] NOTICE: core_log: ssh[2813] core dumped: /var/core/core_hostname_ssh_14247_103_1373568074_2813
Jul 11 20:41:26 hostname genunix: [ID 603404 kern.notice] NOTICE: core_log: ssh[2889] core dumped: /var/core/core_hostname_ssh_14247_103_1373568085_2889
Jul 11 20:44:37 hostname genunix: [ID 603404 kern.notice] NOTICE: core_log: ssh[6711] core dumped: /var/core/core_hostname_ssh_14247_103_1373568276_6711
Jul 11 20:47:54 hostname genunix: [ID 603404 kern.notice] NOTICE: core_log: ssh[11121] core dumped: /var/core/core_hostname_ssh_14247_103_1373568473_11121
Jul 11 20:59:34 hostname genunix: [ID 603404 kern.notice] NOTICE: core_log: ssh[25061] core dumped: /var/core/core_hostname_ssh_14247_103_1373569173_25061

I try a couple of things, none of which seem to particular help but the problem goes away after about half an hour. Then it comes back a couple of days later. And then again. And then it happens on some other containers.

Of course by this time, my call logged with Oracle has been escalated to the highest level. They come back with this:

It seems at this point that you have bin hit by known issue.
Bug 15781192 - SUNBT7156478-SOLARIS_11U1 double free in kernelSlottable.c kernel_slottable_ini
This was fixed in the S11u1 release .. but now we have started a backport CR for S10. At this point the only workaround is to disable pkcs11 engine in the sshd_conf and restart ssh.

And then gave the complete workaround:

The complete workaround requires three steps to be executed inside the Solaris 10 branded zone: 1) Uninstall the pkcs11 kernel provider:  # cryptoadm uninstall provider='/usr/lib/security/$ISA/pkcs11_kernel.so' 2) Disable the pkcs11 engine for sshd  # vi /etc/ssh/sshd_config add the line "UseOpenSSLEngine no" to this file (without the quotes) 3) Restart the ssh service to pickup the change:  # svcadm restart ssh

EDIT: I updated to the latest patches. I'll have to take some time to reverse these workarounds and see if the problem has been fixed. I've been it told it has but I'll have to confirm for myself.

Automated Snapshots

My test systems don't get backed up or snapped via the SAN. So I figured I'd create zfs snapshots on a regular basis just in case.

For Solaris 10 I had a script that did this for me (including sending it to a remote machine if need be) but it was really complex and a bit of a mission for anyone other than me to figure out. For Solaris 11, there is a services that takes care of the snapshotting and scheduling for you (without the option of sending to a remote site).

It's pretty easy to install and configure as well:

1. Install the package:
• pkg install time-slider
2. Start the services
• svcadm restart dbus
• svcadm enable time-slider
3. Choose which filesystems it snaps (properties should be inherited by child filesytems)
• zfs set com.sun:auto-snapshot=true rpool/export
4. Manually exclude certain filesystems
• zfs set com.sun:auto-snapshot=false rpool/swap1
5. Enable the snap schedules you need:
• svcadm enable auto-snapshot:hourly
• svcadm enable auto-snapshot:daily

Source: http://vartmpnotes.blogspot.com/2013/03/automatic-zfs-snapshots-on-solaris-11.html

and it includes the steps for checking and modifying the frequency of the snaps, as well as how many it will keep.

I just wish there was a text only version of time-slider so that I don't have to install all the Gnome packages I'm never going to use.

Hanging Out - Not in a good way

I was configuring a new Solaris 11 zone the other day when I started getting some performance problems. Notably things would just hang. Command like prstat and top and even an ls would hang or take a very long time to complete.

The real worry was that it not only affected the zone, but the global zone too!

I could log in to extra sessions with no problem but as soon as I ran a command it would just hang. And since there was only this zone on this freshly installed global zone, I was really really worried that I had some problem with the hardware.

Luckily I did some checking first before having a well-deserved nervous breakdown (I still had 8 other zones to configure and the server is going live on Saturday!). The template I had used for creating the Solaris 11 zone was a current Solaris 10 zone template that I had modified.

More luck than reasoning made the following two settings on the zone config stand out:
limitpriv: default
scheduling-class: FSS

I cleared both settings and restarted my zone and the intermittent problems went away.


*Note to self: Investigate at a later point to understand fully.