Solaris 11.2 breaks my zpool import of clone with missing cache disk

1.       Assign disks to machine

root@prodmachine:~# sanlun lun show |grep zpooltest

zamgnasvm01          /vol/unixprod_vol99_zpooltest/unixprod_vol99_zpooltest_data01  /dev/rdsk/c0t600A0980383034716124465434593156d0s2 qlc3       FCP        1g      C

zamgnasvm01          /vol/unixprod_vol99_zpooltest/unixprod_vol99_zpooltest_data02  /dev/rdsk/c0t600A0980383034716124465434593157d0s2 qlc2       FCP        1g      C

zamgnasvm01          /vol/unixprod_vol99_zpooltest/unixprod_vol99_zpooltest_log01   /dev/rdsk/c0t600A0980383034716124465434593158d0s2 qlc2       FCP        1g      C

zamgnasvm01          /vol/unixprod_vol99_zpooltest/unixprod_vol99_zpooltest_cache01 /dev/rdsk/c0t600A0980383034716124465434593159d0s2 qlc1       FCP        1g      C

root@prodmachine:~#

2.       Create a zpool

root@prodmachine:~# zpool status zpooltest

  pool: zpooltest

 state: ONLINE

  scan: none requested

config:

        NAME                                       STATE     READ WRITE CKSUM

        zpooltest                                  ONLINE       0     0     0

          mirror-0                                 ONLINE       0     0     0

            c0t600A0980383034716124465434593156d0  ONLINE       0     0     0

            c0t600A0980383034716124465434593157d0  ONLINE       0     0     0

        logs

          c0t600A0980383034716124465434593158d0    ONLINE       0     0     0

        cache

          c0t600A0980383034716124465434593159d0    ONLINE       0     0     0

errors: No known data errors

root@prodmachine:~#

3.       Use SAN replication to replicate disks to remote machine

4.       Check remote machine’s BE

root@drmachine:~# beadm list

BE                 Active Mountpoint Space  Policy Created

--                 ------ ---------- -----  ------ -------

solaris-5          -      -          53.83M static 2014-07-08 12:43

solaris-5-backup-1 NR     /          11.78G static 2014-10-13 14:34

solaris-7          -      -          15.05G static 2015-02-26 14:43

solaris-8          -      -          1.77G  static 2015-04-08 10:39

root@drmachine:~# pkg list entire

NAME (PUBLISHER)                                  VERSION                    IFO

entire                                            0.5.11-0.175.1.19.0.6.0    i--

root@drmachine:~#

5.       Only assign the data lun (i.e. no mirror, no log, no cache)

root@drmachine:~# sanlun lun show |grep zpooltest

zamgnasvmdr01        /vol/zamgnasvm01_unixprod_vol99_zpooltest_mirror_CLONE/unixprod_vol99_zpooltest_data01 /dev/rdsk/c0t600A0980443355634D3F46644573306Fd0s2 qlc2       FCP        1g      C

root@drmachine:~#

6.       Import the pool (with –f and –m options) RESULT: SUCCESS

root@drmachine:~# zpool import -R / -f -m zpooltest

root@drmachine:~# zpool status zpooltest

  pool: zpooltest

 state: DEGRADED

status: One or more devices are unavailable in response to persistent errors.

        Sufficient replicas exist for the pool to continue functioning in a

        degraded state.

action: Determine if the device needs to be replaced, and clear the errors

        using 'zpool clear' or 'fmadm repaired', or replace the device

        with 'zpool replace'.

        Run 'zpool status -v' to see device specific details.

  scan: none requested

config:

        NAME                                       STATE     READ WRITE CKSUM

        zpooltest                                  DEGRADED     0     0     0

          mirror-0                                 DEGRADED     0     0     0

            c0t600A0980443355634D3F46644573306Fd0  ONLINE       0     0     0

            4167588681570226322                    UNAVAIL      0     0     0

        logs

          16316075685438834122                     UNAVAIL      0     0     0

        cache

          c0t600A0980383034716124465434593159d0    UNAVAIL      0     0     0

errors: No known data errors

root@drmachine:~#

7.       Change to different BE and reboot

root@drmachine:~# beadm activate solaris-7

root@drmachine:~# init 6

…

root@drmachine:~# beadm list

BE                 Active Mountpoint Space   Policy Created

--                 ------ ---------- -----   ------ -------

solaris-5          -      -          53.83M  static 2014-07-08 12:43

solaris-5-backup-1 -      -          1.45G   static 2014-10-13 14:34

solaris-7          NR     /          25.94G  static 2015-02-26 14:43

solaris-8          -      -          1.77G   static 2015-04-08 10:39

root@drmachine:~# pkg list entire

NAME (PUBLISHER)                                  VERSION                    IFO

entire                                            0.5.11-0.175.2.5.0.5.0     i--

root@drmachine:~#

8.       Import the pool (with –f and –m) RESULT: FAILURE

root@drmachine:~# zpool import -R / -f -m zpooltest

cannot import 'zpooltest': one or more devices is currently unavailable

root@drmachine:~#

9.       Assign the mirror disk to the machine and retry import RESULT: FAILURE

root@drmachine:~# sanlun lun show

controller(7mode)/                                                                                          device                                            host                  lun

vserver(Cmode)       lun-pathname                                                                           filename                                          adapter    protocol   size    mode

------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

zamgnasvmdr01        /vol/zamgnasvm01_unixprod_vol99_zpooltest_mirror_CLONE/unixprod_vol99_zpooltest_data01 /dev/rdsk/c0t600A0980443355634D3F46644573306Fd0s2 qlc2       FCP        1g      C

zamgnasvmdr01        /vol/zamgnasvm01_unixprod_vol99_zpooltest_mirror_CLONE/unixprod_vol99_zpooltest_data02 /dev/rdsk/c0t600A0980443355634D3F466445733070d0s2 qlc0       FCP        1g      C

root@drmachine:~# zpool import -R / -f -m zpooltest

cannot import 'zpooltest': one or more devices is currently unavailable

root@drmachine:~#

10.   Assign the log device disk to the machine and retry import RESULT: SUCCESS

root@drmachine:~# sanlun lun show

controller(7mode)/                                                                                          device                                            host                  lun

vserver(Cmode)       lun-pathname                                                                           filename                                          adapter    protocol   size    mode

------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

zamgnasvmdr01        /vol/zamgnasvm01_unixprod_vol99_zpooltest_mirror_CLONE/unixprod_vol99_zpooltest_data01 /dev/rdsk/c0t600A0980443355634D3F46644573306Fd0s2 qlc2       FCP        1g      C

zamgnasvmdr01        /vol/zamgnasvm01_unixprod_vol99_zpooltest_mirror_CLONE/unixprod_vol99_zpooltest_data02 /dev/rdsk/c0t600A0980443355634D3F466445733070d0s2 qlc0       FCP        1g      C

zamgnasvmdr01        /vol/zamgnasvm01_unixprod_vol99_zpooltest_mirror_CLONE/unixprod_vol99_zpooltest_log01  /dev/rdsk/c0t600A0980443355634D3F466445733071d0s2 qlc0       FCP        1g      C

root@drmachine:~# zpool import -R / -f -m zpooltest

root@drmachine:~#

11.    Export  the zpool, unassign the log device and assign the cache device disk to the machine and retry import. RESULT: FAILURE

root@drmachine:~# sanlun lun show

controller(7mode)/                                                                                           device                                            host                  lun

vserver(Cmode)       lun-pathname                                                                            filename                                          adapter    protocol   size    mode

-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

zamgnasvmdr01        /vol/zamgnasvm01_unixprod_vol99_zpooltest_mirror_CLONE/unixprod_vol99_zpooltest_data01  /dev/rdsk/c0t600A0980443355634D3F46644573306Fd0s2 qlc2       FCP        1g      C

zamgnasvmdr01        /vol/zamgnasvm01_unixprod_vol99_zpooltest_mirror_CLONE/unixprod_vol99_zpooltest_data02  /dev/rdsk/c0t600A0980443355634D3F466445733070d0s2 qlc0       FCP        1g      C

zamgnasvmdr01        /vol/zamgnasvm01_unixprod_vol99_zpooltest_mirror_CLONE/unixprod_vol99_zpooltest_cache01 /dev/rdsk/c0t600A0980443355634D3F466445733072d0s2 qlc0       FCP        1g      C

root@drmachine:~# zpool import -R / -f -m zpooltest

cannot import 'zpooltest': one or more devices is currently unavailable

root@drmachine:~#

Got a call logged with Oracle to sort this out. Fixed with Solaris 11.3.

Core file administration

This is the defaults:

     global core file pattern:
     global core file content: default
     init core file pattern: core
     init core file content: default
     global core dumps: disabled
     per-process core dumps: enabled
     global setid core dumps: disabled
     per-process setid core dumps: disabled
     global core dump logging: disabled

This is what I like to set it to:

     global core file pattern: /var/cores/core_%n_%f_%u_%g_%t_%p
     global core file content: default
     init core file pattern: core
     init core file content: default
     global core dumps: enabled
     per-process core dumps: disabled
     global setid core dumps: enabled
     per-process setid core dumps: disabled
     global core dump logging: enabled

Reasoning:
1. I don't like my core files all over the place
2. Easier to find and clean up.
3. I only enable per-process core file dump when a user needs it (on a zone)

VALUE		HOW TO SET IT
global core file pattern	Specifies name and location of the global core files (A leading "/" specifes an absolute path)	coreadm -g /var/cores/core_%n_%f_%u_%g_%t_%p
global core file content	Defines content of global files
init core file pattern	Specifies name and location of the global per-process files (No leading "/" makes it relative to working directory)	coreadm -p core
init core file content	Defines content of per-process files
global core dumps	Enable/disables global core-dumps	coreadm -e global
per-process core dumps	Enable/disables per-process core-dumps	coreadm -d process
global setid core dumps	Enable/disables global setid core-dumps	coreadm -e global-setid
per-process setid core dumps	Enable/disables per-process setid core-dumps	coreadm -d proc-setid
global core dump logging	Enable/disables logging to syslog	coreadm -e log

Installing ASR

Auto Service Request (ASR) is a secure, scalable, customer-installable software feature of Oracle Premier Support for Systems and Oracle/Sun Limited Warranty support that provides auto-case generation when specific hardware faults occur. ASR is designed to enable faster problem resolution by eliminating the need to initiate contact with Oracle for hardware failures, reducing both the number of phone calls needed and overall phone time required. ASR also simplifies support operations by utilizing electronic diagnostic data. If your qualified system is under warranty or covered by a service plan, you are entitled to ASR at no charge.

 I've been using ASR on my SUN storage for years. And it works brilliantly. Great to come into work in the morning and receive emails that a disk failed and also that a replacement disk is on its way. That ASR worked a little different - the new implementation needs an ASR server installed. Here are the steps I followed to install ASR:

1. Created a zone for ASR - I put everything in zones. Bitches love zones. I had a warm standby machine in my DMZ so I used that.
2. Download the software logging into support.oracle.com and searching for Oracle Document : 1185493.1. To install ASR, you need to install OASM (Oracle Automated Service Manager) first. This doc will give you the link to both ASR and OASM.
3. Install OASM
• unzip Oracle_OASM_150_SOLARIS64.zip
• pkgadd -d SUNWsasm-1.5.0-20130731121217.pkg
• (The next two steps I did because of errors messages during the install so maybe do them before the install.)
• pkg install SUNWcar SUNWkvm
• pkg install pkg:/system/management/service-tag@1.1.5-0.175.1.0.0.23.0
4. Install ASR
• unzip Oracle_ASR_481_SOLARIS64.zip
• pkgadd -d SUNWswasr-4.8.1-20140415212449.pkg
• export PATH=$PATH:/opt/SUNWswasr/bin:/opt/SUNWsasm/bin
• put line above in .profile
• Run: asr
• asr> register
• Options are all all easy - just make sure your machine can get to transport.oracle.com
5. Make sure you other machines can talk to ASR server
1. First way of doing this:
• asr> enable_http_receiver -p 8234
2. Second way of doing this:
• Edit /var/opt/SUNWsasm/configuration/config.ini
  jetty.enable=true
  jetty.host=<ipaddress> (Don't use "localhost")
  jetty.http.port=8234
• Restart sasm: svcadm restart sasm
6. Edit /var/opt/SUNWsasm/configuration/config.ini and set java.exec property to point to your java binary. And then restart sasm with svcadm restart sasm.
7. Check that it all works with:
• asr> show_http_receiver
8. To connect Solaris 11 machines to your new ASR server, run:
• asradm register -e http://<ip_of_your_server>:8234/asr

And that's it! Next you'll want to log into support.oracle.com and make sure that all your servers' details are correct and what not.

Rolling back a Solaris boot environment

Boot environments have been with us since Solaris 10. In my experience it started out pretty buggy but in the last year or so its been stable. It really has revolutionized patching for me in terms of cutting down risk,time and effort.

In Solaris 11, boot environments are pretty much built into how you do patching. ("Patching" is a bit of a misnomer in Solaris 11 - "updating/upgrading" are more appropriate terms.)

Recently I had to roll back to a previous boot environment two weeks after the implementation. Rolling back is pretty easy (activating the previous boot environment and rebooting), however, you do lose changes you've made to the OS in the meantime. You can mitigate this risk by mounting the other boot environment and comparing files between the two.

Areas you should check are:

1. /etc/system and any other config files in /etc
2. Crons
3. Zone configurations
4. Files in root's home directory and other home directories.

These are just the basics, hopefully if you've made any major changes since then, you've documented it well enough to figure out how to re-implement then if needed.

Solaris Crash Dumps and Basic Analysis

If your Solaris system panics and reboots, it'll probably create a crash dump in /var/crash. You can also force a crash dump either online (using "savecore -L") or as part of a reboot (using "reboot -d").

Normally this is where I stop and upload the /var/crash/vmdump.0 file up to Oracle to find out what the problem was. However, you can do some basic investigations yourself using the following steps:

# savecore –f vmdump.0 /somedirectory

# cd /somedirectory

# mdb *0

mdb> ::status

mdb> ::panicinfo

mdb> ::stack

mdb> ::msgbuf

mdb> ::cpuinfo

mdb> ::ps

mdb> ::arc

mdb> ::memstat

(*If the vmdump file is called vmdump.1 then use 1 instead of 0 in the above steps)

Honestly, most of the output is Greek to me but it's nice to know in the off chance something makes sense.

Sys-Unconfig on Solaris 10 Branded Zone Doesn't Give Option to Change Hostname

As the title says, I was trying to do a sys-unconfig on a Solaris 10 branded zone and when I reboot and go enter the new config info, the hostname doesn't get asked to be changed. The system automatically configures the zonename as the hostname.

Some things I tried that didn't work:

1. Configure a NIC that doesn't exist
   Since most of the googled results pointed to pulling out the network cable, I tried the logical equivalent for a zone. However, configuring a non-existent NIC doesn't work, because the a zone won't boot when configured with a NIC that doesn't exist.  
2. Choose a NIC that does exist but is offline
   Same reasoning at point 1, still doesn't work though.
3. Configure an IP that doesn't exist on our LAN
   Same reasoning at point 1, still doesn't work though.
4. Do it with no NIC configured
   This actually works, the system asks me for a new hostname during the sysconfig. Unfortunately, it doesn't ask me for any of the subsequent network info. So when I add the NIC, I'll have to then do the subnet and DNS configuration manually. No thank you.  
5. Search for files that contain the hostname and delete manually
   Just in case the sys-unconfig was not deleting these files. Didn't work.
6. Set the bootargs to "noauto"
   Got this from an Oracle SR for when the hostname is being retrieved from the jumpstart server (which I don't have anyway). Didn't work.

What worked:

1. After the sys-unconfig and subsequent sysconfig completed, I edited the following files and rebooted the zone:
• /etc/inet/hosts
• /etc/nodename
• /etc/hostname.net0 (or whatever's relevant)

Empty root folder on Solaris10 branded zone

If you have a Solaris10 branded zone, and it is in an "unavailable" state, the root folder in the zone path will look empty. Scared the living daylights out of me! Just do an attach and the folder's contents will appear.

It might just be my understanding of how Solaris 11 does zones though. It looks like during the attach, there is another filesystem that gets mounted on root. I'll have to do some testing and prodding and understand what's going on.