Empty root folder on Solaris10 branded zone

If you have a Solaris10 branded zone, and it is in an "unavailable" state, the root folder in the zone path will look empty. Scared the living daylights out of me! Just do an attach and the folder's contents will appear.

It might just be my understanding of how Solaris 11 does zones though. It looks like during the attach, there is another filesystem that gets mounted on root. I'll have to do some testing and prodding and understand what's going on.

SSH Issue on Solaris 10 Branded Zones

So suddenly one night I get a call from the night operators telling me that they can't ssh to a particular zone. Their ssh sessions are even dying. When I log in (via the global zone), the message log has loads of sshd core dumps!:

Jul 11 20:41:08 hostname genunix: [ID 603404 kern.notice] NOTICE: core_log: ssh[2791] core dumped: /var/core/core_hostname_ssh_14247_103_1373568067_2791
Jul 11 20:41:15 hostname genunix: [ID 603404 kern.notice] NOTICE: core_log: ssh[2813] core dumped: /var/core/core_hostname_ssh_14247_103_1373568074_2813
Jul 11 20:41:26 hostname genunix: [ID 603404 kern.notice] NOTICE: core_log: ssh[2889] core dumped: /var/core/core_hostname_ssh_14247_103_1373568085_2889
Jul 11 20:44:37 hostname genunix: [ID 603404 kern.notice] NOTICE: core_log: ssh[6711] core dumped: /var/core/core_hostname_ssh_14247_103_1373568276_6711
Jul 11 20:47:54 hostname genunix: [ID 603404 kern.notice] NOTICE: core_log: ssh[11121] core dumped: /var/core/core_hostname_ssh_14247_103_1373568473_11121
Jul 11 20:59:34 hostname genunix: [ID 603404 kern.notice] NOTICE: core_log: ssh[25061] core dumped: /var/core/core_hostname_ssh_14247_103_1373569173_25061

I try a couple of things, none of which seem to particular help but the problem goes away after about half an hour. Then it comes back a couple of days later. And then again. And then it happens on some other containers.

Of course by this time, my call logged with Oracle has been escalated to the highest level. They come back with this:

It seems at this point that you have bin hit by known issue.
Bug 15781192 - SUNBT7156478-SOLARIS_11U1 double free in kernelSlottable.c kernel_slottable_ini
This was fixed in the S11u1 release .. but now we have started a backport CR for S10. At this point the only workaround is to disable pkcs11 engine in the sshd_conf and restart ssh.

And then gave the complete workaround:

The complete workaround requires three steps to be executed inside the Solaris 10 branded zone: 1) Uninstall the pkcs11 kernel provider:  # cryptoadm uninstall provider='/usr/lib/security/$ISA/pkcs11_kernel.so' 2) Disable the pkcs11 engine for sshd  # vi /etc/ssh/sshd_config add the line "UseOpenSSLEngine no" to this file (without the quotes) 3) Restart the ssh service to pickup the change:  # svcadm restart ssh

EDIT: I updated to the latest patches. I'll have to take some time to reverse these workarounds and see if the problem has been fixed. I've been it told it has but I'll have to confirm for myself.

Automated Snapshots

My test systems don't get backed up or snapped via the SAN. So I figured I'd create zfs snapshots on a regular basis just in case.

For Solaris 10 I had a script that did this for me (including sending it to a remote machine if need be) but it was really complex and a bit of a mission for anyone other than me to figure out. For Solaris 11, there is a services that takes care of the snapshotting and scheduling for you (without the option of sending to a remote site).

It's pretty easy to install and configure as well:

1. Install the package:
• pkg install time-slider
2. Start the services
• svcadm restart dbus
• svcadm enable time-slider
3. Choose which filesystems it snaps (properties should be inherited by child filesytems)
• zfs set com.sun:auto-snapshot=true rpool/export
4. Manually exclude certain filesystems
• zfs set com.sun:auto-snapshot=false rpool/swap1
5. Enable the snap schedules you need:
• svcadm enable auto-snapshot:hourly
• svcadm enable auto-snapshot:daily

Source: http://vartmpnotes.blogspot.com/2013/03/automatic-zfs-snapshots-on-solaris-11.html

and it includes the steps for checking and modifying the frequency of the snaps, as well as how many it will keep.

I just wish there was a text only version of time-slider so that I don't have to install all the Gnome packages I'm never going to use.

Hanging Out - Not in a good way

I was configuring a new Solaris 11 zone the other day when I started getting some performance problems. Notably things would just hang. Command like prstat and top and even an ls would hang or take a very long time to complete.

The real worry was that it not only affected the zone, but the global zone too!

I could log in to extra sessions with no problem but as soon as I ran a command it would just hang. And since there was only this zone on this freshly installed global zone, I was really really worried that I had some problem with the hardware.

Luckily I did some checking first before having a well-deserved nervous breakdown (I still had 8 other zones to configure and the server is going live on Saturday!). The template I had used for creating the Solaris 11 zone was a current Solaris 10 zone template that I had modified.

More luck than reasoning made the following two settings on the zone config stand out:
limitpriv: default
scheduling-class: FSS

I cleared both settings and restarted my zone and the intermittent problems went away.


*Note to self: Investigate at a later point to understand fully.

Migrating a Solaris 10 zone to a Solaris 10 branded zone on Solaris 11

So there's an official Oracle procedure somewhere. Mine just goes into a little more detail and putting in workarounds for some of the bugs I found. 

*Edit: Points 5 and 11 is not necessary with the latest Solaris packages installed.

Zone name: zone1

On old (Solaris 10) global zone:

1. zoneadm -z zone1 ready
2. cd /zone/path
3.  find root -print | cpio -oP@ | gzip >/dumps/zone1.cpio.gz
4. zonecfg -z zone1 export -f /dumps/zone1.cfg
5. Copy config and dump across to new global zone
6. zfs send and recv any extra filesystems across to new global zone

On new (Solaris 11) global zone

1. Notes:
1.  I create a zpool with the name “zone”where all my zones data will sit on
2.  I create a zfs filesystem zone/roots where all the zone paths will be in
3.  Each zone gets a get a zfs filesystem off /zone where its mounted filesystems stem off from. e.g /zone/zone1 with /zone/zone1/home
2. vi zone1.cfg*
1. Change IP if needed
2. Correct attached filesystems path if needed
3. Set brand=Solaris10
4. Set ip-type=exclusive
5. Change  from net to anet
3. zonecfg -z  zone1 -f zone1.cfg
4. zoneadm -z zone1 attach -a /dump/zone1.cpio.gz
5. Make sure the NIC gets configured on boot - fixes this
1. vi /zone/path/zone1/root/etc/rc3.d/S99sol11networkaround
• #This is a workaround for Sol10 zones on Sol11
• # Till the bug gets fixed
• ifconfig net0 `cat /etc/hostname.net0`
• sleep 3
• svcadm clear svc:/network/physical:default
2. vi /zone/path/zone1/root/etc/hostname.net0
• zone1 netmask 255.255.255.0 up
6. Change root's home from /export/home/root to /root - might not be needed in your environment
1. vi /zone/path/zone1/root/etc/passwd
2. mv /zone/zone1/home/root /zone/roots/zone1/root/
7. If IP is to change, vi /zone/roots/zone1/root/etc/hosts
8. Boot zone1 and zlogin
9. ifconfig plumb net0
10. vi /etc/default/nfs and change: LOCKD_SERVERS=1024 - fixes this
11. vi /etc/defaultrouter
12. Reboot zone and test
13. If you're changing the hostname, you'll have to do a sys-unconfigure and don't forget to update:
1. /etc/hosts
2. /etc/nsswitch.conf
3. /etc/samba/smb.conf
4. /etc/hostname.net0
5. and you'll probably have to do a final reboot.

----

*Example zone1.cfg

create -b

set brand=solaris10

set zonepath=/zone/roots/zone1

set autoboot=false

set bootargs=”-m verbose”

set ip-type=exclusive

add fs

set dir=/export/home

set special=/zone/zone1/home

set type=lofs

end

add fs

set dir=/oracle

set special=/zone/zone1/oracle

add anet

set linkname=net0

set lower-link=aggr0

set allowed-address=192.12.23.52/24

set configure-allowed-address=true

set defrouter=192.12.23.1

set link-protection=mac-nospoof

set mac-address=random

end

add capped-memory

set physical=2G

end
----

Time command Solaris 11

So all of the scripts so far that I've taken across from Solaris 10 to Solaris 11 have worked. Which is no surprise since I generally use the Bourne shell with the idea that it makes my scripts more acceptable in other environments.

One of my scripts didn't work though. It's a simple little script that writes a test file to the current directory and tells you how long it took.
#!/bin/sh

# This is a quick test of write speed.
# The filesize to write can be specified in gigabytes as a parameter.
# Doubt whether this script works if the test file takes more than 59m to write.
tempfile=gigfile.tmp
if [ $# = 1 ]; then
  filesizeGb=$1
else
  filesizeGb=1
fi
filesizeMb=`expr $filesizeGb \* 1024` || exit 1
# Find out the time taken to write the file
sync
timeforwrite=`time dd bs=1048576 count=$filesizeMb if=/dev/zero of=$tempfile 2>&1 | grep real | awk '{ print $NF }'`
timeforsync=`time sync 2>&1 | grep real | awk '{ print $NF }'`
#Take into account if time for write took more than a minute
if [ "`echo $timeforwrite | grep ':'`" != "" ]; then
  seconds=`echo $timeforwrite | awk -F':' '{ print $NF }'`
  minutes=`echo $timeforwrite | awk -F':' '{ print $1 }'`
  min2sec=`expr $minutes \* 60`
  timeforwrite=`echo "scalar=4;$min2sec+$seconds" | bc`
fi
# Calculate the speed
timetaken=`echo "scalar=4;$timeforwrite+$timeforsync" | bc`
writespeed=`echo "scalar=2;$filesizeMb/$timetaken" | bc`
# Do some cleaning up
[ -f "$tempfile" ] && rm "$tempfile"
echo A "$filesizeGb"Gb file was written in $timetaken seconds at a speed of approximately $writespeed"Mb/s."
exit 0

The error message when running it on Solaris 11 is not important because it incorrectly pointed out "bc" - i.e. my calculator. Looking into the script, I could see that my script wasn't giving bc the correct variables to add - none in fact.

Rather than talk you through everything, here's the conclusion. The "time" command works differently in Solaris 11 in two ways:

1. It always outputs the time in ##m##s format, similar to the way it did in bash in Solaris 10 but not in the Bourne shell.
2. It doesn't pipe into standard error as neatly as it did before. I'll give an example. If I wanted to store in a test file how long it takes the system to echo "hello world", previously I would run command like this: time echo "hello world" 2>output.txt. This doesn't work in Solaris 11, I need to run it like this: (time echo "hello world") 2>output.txt

Login Security Part 2 - Setting up a Solaris11 to authenticate to AD using SAMBA

Setting up a Solaris11 to authenticate to AD using SAMBA:

1. Add to /etc/system and reboot (This is once off on the global zone only)
2. vi /etc/samba/smb.conf*
3. mv /etc/pam.conf /etc/pam.conf.bak
4. mv /etc/pam.conf-winbind /etc/pam.conf
5. svccfg -s name-service/switch
   > setprop config/password = "files winbind"
   > setprop config/group = "files winbind"
   > exit
6. svcadm refresh name-service/switch
7. net join -U ADUserThatCanAddToDomain -S ADDomainControllerName
8. svcadm enable samba winbind
9. getent passwd

*Truncated smb.conf:

[global]
        workgroup = <HELLO>
        #realm = <HELLO.COM>
        encrypt passwords = yes
        netbios aliases = <hostname>
        server string = <hostname>
        security = DOMAIN
        auth methods = winbind
        password server = <ADDomainControllerIP>
        unix password sync = Yes
        log level = 2 vfs:3
        syslog = 2
        log file = /var/log/samba/smb-%U-%M.log
        max xmit = 65535
        name resolve order = host bcast
        deadtime = 15
        socket options = TCP_NODELAY IPTOS_LOWDELAY
        load printers = No
        disable spoolss = Yes
        show add printer wizard = No
        preferred master = No
        local master = No
        domain master = No
        dns proxy = No
        ldap ssl = no
        socket address =
        idmap uid = 10000-20000
        idmap gid = 10000-20000
        winbind enum users = Yes
        winbind enum groups = Yes
        winbind use default domain = yes
        hide special files = Yes
        hide unreadable = Yes
        veto files = /lost+found/samba_recycle_bin/