Changing _keepalive_interval doesn’t seem to take
I recently had cause to need to change the keepalive time on a Solaris 11.4 box from the default of 7200 seconds back to 900 seconds. Don’t ask why; just accept that I had the need. I’m writing this blog entry as I could find nothing out there that talked sensibly about this problem.
So, the way to do this under Solaris 11.4 is
# ipadm set-prop -p _keepalive_interval=900000 tcp# ipadm show-prop -p _keepalive_interval tcpPROTO PROPERTY PERM CURRENT PERSISTENT DEFAULT POSSIBLEtcp _keepalive_interval rw 900000 900000 7200000 10000-864000000
The problem was that the processes that I was wanting this to apply to were spawned from inetd and keepalive was still happening at 7200 seconds.
First off, I need to note that the Solaris 11.4 function tcp_init_values() has changed to what is currently in Illumos, so Illumos became only the vaguest of guides for the DTrace.
Rather than go through the DTrace and mdb hoops to work this out, it turns out tat the processes spawned by inetd, inherit the TCP defaults of the existing listeners.
Restarting the inetd service updates the listeners to the new defaults.
# svcadm restart inetd
I knew it would be something simple and silly.
Solaris Auto-Installer Ignores Swap and Dump Manifest Entries in a UAR installation
Well that was a “fun” couple of hours chasing down.
I have been trying to use a clone UAR with auto-installer to help me do a system migration to new hardware. As archiveadm(1M) specifically excludes the swap and dump zvols, I tried to specify them in the installation manifest.
<logical noswap="false" nodump="false">
<zpool name="rpool" action="create" is_root="true">
<vdev name="vdev1" redundancy="mirror"/>
<zvol name="swap" action="create" use="swap">
<size val="32gb" />
</zvol>
<zvol name="dump" action="create" use="dump">
<size val="64gb" />
</zvol>
</zpool>
</logical>
However, I was finding that while the installer correctly parsed the above:
-> [logical] (Logical: noswap=False; nodump=False) -> [rpool] (Zpool: name=rpool; action=create; is_root=True; is_boot=False; vdev_list=[None]) -> [vdev1] (Vdev: name=vdev1; redundancy=mirror) -> [swap] (Zvol: name=swap; action=create; use=swap; size=32gb) -> [dump] (Zvol: name=dump; action=create; use=dump; size=64gb)
It then discarded it
2026-09-07 01:03:12,376 InstallationLogger.target-selection DEBUG Deleting Zvols from manifest: [Zvol: name=swap; action=create; use=swap; size=32gb, Zvol: name=dump; action=create; use=dump; size=64gb]
This is in _prune_manifest() in /usr/lib/python3.11/vendor-packages/solaris_installall/client/checkpoints/archive_target_selection.py.
The comment in this function is
Removes unwelcome guests from the manifest such as BEs and datasets
that would conflict with path hierarchies received from the
received archive streams
I am assuming that this function is only called when doing a UAR installation so as not to conflict with anything incoming from the rpool stream.
As archiveadm(8) specifically excludes swap and dump zvols when creating a UAR archive, it then goes to auto-determine the sizes to use, which will inevitably differ from what may have been provided in the manifest.
The upshot is that it is NOT currently possible to declare swap and dump zvols in an installation manifest. As far as I am concerned, this is a bug and I have asked the support engineer on the MOS ticket that I have on this to log a defect appropriately.
Explorer generates error messages after 25.2.25.04.15.
We have a few Solaris systems that run “explorer -q -P” from cron. Since we upgraded I’m seeing these “quiet” runs generate root email containing
Your "cron" job on <hostname>
/usr/lib/explorer/bin/explorer -q -P
* *For more information on the error below, run rda.sh -E RDA-00013
* For more information on the error below, run rda.sh -E RDA-20421
RDA-20421: SDCL execution error
* For more information on the error below, run rda.sh -E RDA-07605 in
RDA:MCend near line 82
RDA-07605: Error encountered in method
"RDA::Object::Report::write_cmd_rpts"
* For more information on the error below, run rda.sh -E RDA-22817
RDA-22817: Invalid path "/usr/bin/ls -ld /var/adm"
* For more information on the error below, run rda.sh -E RDA-22223
RDA-22223: Error encountered in the block called
I was unable to find anything helpful on the net (and indeed “Doc ID 1492341.1 Troubleshooting of Oracle Explorer Data Collector as part of Oracle Services Tools Bundle (STB)” is WOEFULLY out of date). As such I logged a ticket.
It appears that this is a known bug that will be fixed in RDA 25.4 (whenever that is released).
Bug 38053397 Running Explorer VAR module - RDA-22817: Invalid path "/usr/bin/ls -ld /var/adm"
The error is informational and the explorer is still collected, though I’d be interested in knowing exactly what this error stops from running. I’m assuming that it’s a test just before it does something with /var/adm but have really not had a chance to look too deeply in to it.
I’ve asked when we might see RDA 25.4 and whether or not it will be able to be upgraded without installing the full SRU.
In the mean time I guess I just need to live with a bunch of extra emails.
Processes are cheap (but not free)
I did a talk at a Sun conference wit this title with a slant to pointing out how a single threaded shell script could bring an F15k to it’s knees because of the ecache flushing going on in a process knockdown, because of lots of short lived processes going on. That was actually what prompted me to look at the whole shell DTrace provider thing that ended up in an appendix in Brendan’s DTrace book.
In my current role one of the things I am currently doing is going through cron jobs that are being run on a number of systems under my purview, and I came across a monitoring script that is being run every five minutes on some systems and every minute on others. This script looks at the output of a “netstat -n” command, specifically using a particular interface, gathering the local port number and remote address. It takes this list and merges it with the existing list. I have no idea of the history of why this is here, but am assuming it is actually serving a purpose for the time being.
#!/bin/ksh
TF1=/tmp/netstat.tmp
TF2=/tmp/netstat2.tmp
> $TF2
OUTFILE=/export/corp/logs/netlist
netstat -n | grep -v stream-ord | grep "10." | grep -v 9100 > /tmp/netstat.tmp
cat $TF1 | sed 's/ */\ /g' | while read line
do
PORT=`echo ${line} | cut -d'.' -f5 | cut -d' ' -f1`
ADDR=`echo ${line} | cut -d' ' -f2`
N1=`echo ${ADDR} | cut -d '.' -f1`
N2=`echo ${ADDR} | cut -d '.' -f2`
N3=`echo ${ADDR} | cut -d '.' -f3`
N4=`echo ${ADDR} | cut -d '.' -f4`
typeset -i OP=`echo ${ADDR} | cut -d '.' -f5`
typeset -i PCOUNT=`echo ${PORT} | wc -c`
typeset -i PTI=`echo ${PORT}`
if [ ${PCOUNT} -lt 6 -a ${PTI} -ne 23 -a ${PTI} -ne 1400 ]; then
echo "${N1}.${N2}.${N3}.${N4}:${PORT}" >> $TF2
fi
done
cat $OUTFILE >> $TF2
cat $TF2 | sort | uniq > $OUTFILE
Yes, I shuddered when I saw it.
On one of our S7-2s, this script takes about a minute to run (which is problematical on those systems running it every minute), spawns a bit over 30,000 processes.
I could have simply moved to using ksh internals to do pretty much all of this work (e.g. the maths, and string manipulation), but I took a step further back and realised there was a better way to do this. I also moved to coding the intent, rather than the actuality, by comparing the actual port I was interested in, rather than just grepping it out.
I replaced the main loop with a bit of awk, giving us the following.
#!/bin/ksh
TF1=/tmp/netstat.tmp
trap "rm -f ${TF1}" 0
OUTFILE=/export/corp/logs/netlist
(netstat -n | awk -F'[. ]' '$1 == 10 {
port = $5
if (port<10000 && port != 23 && port != 1400 && port != 9100){
printf("%d.%d.%d.%d:%d\n", $6, $7, $8, $9, port)}}'; \
cat ${OUTFILE}) > ${TF1}
sort -u ${TF1} > ${OUTFILE}
It’s dropped now to only five spawned processes and runs in less than a second.
It’s all very well to know that you can use spawned processes to do things, but you also need to understand the overheads involved when you start putting them in tight loops.
For those interested, the DTrace for counting the spawned processes was:
# dtrace -qn '
BEGIN {c = 0}
proc:::create /pid == $target/{c++}
END {printf("%d spawned processes\n", c)}' -c ./monnet.12.new
5 spawned processes
As an aside, for shits and giggles, I redid the script in python. The python script runs in 0.1 – 0.2 seconds. I’m going with the python. I also added some code to the python so I could place the log file under logadm control. If the LOCKFILE exists, it simply sleeps for a second and tries again (it really shouldn’t need to go through the loop more than once, but to be sure I’ll make it exit after ten fails and syslog something.
Building Chef Inspec from git
So, I’m looking at using inspec for some checking on solaris systems, but I actually want to build it from gitlab, as the community version for down load is sufficiently old that it does not work with our current sshd configurations. Specifically, I’d have to add in
HostKeyAlgorithms +ssh-rsa
to my port 22 sshd_config, and I don’t really want to do that.
I set up WSL on my work desktop, as I already had some scripting for doing things across multiple hosts and besides, you all know I’m a UNIX guy from way back and I’m much more comfortable in at least a UNIX-like environment. As an aside, I did get it to build on my local solaris 11.4 x86 virtual box, but I don’t have one of those on the appropriate work networks.
The “Install it from source” instructions in the README.md at https://github.com/inspec/inspec/tree/main are mostly good. Unfortunately they leave out four gem commands. I’ll likely do a fuller explanation of the build, but after the “bundle install”, you need to run the following before it will let you do the final gem install that is listed.
$ cd inspec-bin
$ gem build inspec-bin.gemspec
$ gem install inspec-bin-6.8.38.gem
$ cd ..
$ gem build inspec-core.gemspec
$ gem install inspec-core-6.8.38.gem
before you can run the
$ gem build inspec.gemspec
$ gem install inspec-6.8.38.gem
Note that I’m also specifically listing the version in there (that’s where the main branch is at the time of writing), as I’d been experimenting with other version and the .gem files for those were still lying around and using ‘*’, like the documentation says would have picked them up too.
So far I’ve managed to do some libraries up for coreadm and logadm (Hmmm, I didn’t think to check if there is already a module for that) checking and I’m making use of the ‘sshd_config’ plugin for checking various the ‘sshd_config’s that we have.
Openat(2) failing with EINTR
I’ve been dealing with a curious issue over the last week or so. We had an application attempting to open a zvol, but it was returning with an error, only on one of the attempts to open it. This had some flow on consequences for the run of the application.
Note that I’m changing the names of the application and other stuff so as to NOT identify the application.
I ran something similar to the following Dtrace to watch what the kernel was doing for the failed openat() system call.
#!/sbin/dtrace -Fs
BEGIN {printf("Monitoring, ...\n");}
syscall::openat:entry /execname == "appname" && copyinstr(arg1) == "/dev/zvol/dsk/rpool/volume"/ {
printf("%s [%d] Opening /dev/zvol/dsk/rpool/volume\n",
execname, pid);
self->interest = 1
}
fbt:::entry /self->interest/ {}
fbt:::return /self->interest/ { printf("rc=%d errno=%d", arg1, errno); }
syscall::openat:return /self->interest && errno == 4/ {exit(0);}
syscall::openat:return /self->interest/ { self->interest = 0; }
Which gave us the following (trimmed for brevity):
6 <- zfsdev_dispatch rc=4 errno=0 retaddr=zfsdev_dispatch+0x18c
…
6 <- zfsdev_ioctl rc=4 errno=0 retaddr=zfsdev_ioctl+0x46c
…
6 <- spec_cb_ioctl rc=4 errno=0 retaddr=spec_cb_ioctl+0x8c
…
6 <- ldi_ioctl rc=4 errno=0 retaddr=ldi_ioctl+0x160
…
6 <- devzvol_handle_ioctl rc=4 errno=0 retaddr=devzvol_handle_ioctl+0x150
…
6 <- devzvol_objset_check rc=4 errno=0 retaddr=devzvol_objset_check+0x124
…
6 <- devzvol_validate rc=4 errno=0 retaddr=devzvol_validate+0x1fc
…
6 <- devname_lookup_func rc=4 errno=0 retaddr=devname_lookup_func+0x5fc
…
6 <- devzvol_lookup rc=4 errno=0 retaddr=devzvol_lookup+0x3e4
6 <- fop_lookup rc=4 errno=0 retaddr=fop_lookup+0x304
…
0 <- lookuppnvp rc=4 errno=0
0 <- lookuppnatcred rc=4 errno=0
0 <- lookupnameatcred rc=4 errno=0
0 <- lookupnameat rc=4 errno=0
0 <- vn_openat rc=4 errno=0
We see that during the call we get a return code of 4 from call to zfsdev_dispatch(). In earlier code (looking at the opensolaris code), devzvol_lookup() ignored an upper return code of 4.
In Solaris 11.4 sru 24 the following change is listed.
30675505 devzvol_lookup should just pass EINTRs along
which explains why it passes it back up the call stack, resulting in the failure with an EINTR. The return code of 4 is an indication that a signal has been received while processing the openat() system call.
We now need to determine what the signal is and where it came from.
Inside the zfs_dispatch() call we get a signal so we end up in issig_forreal() to deal with it.
6 -> issig_forreal
6 -> schedctl_finish_sigblock
6 <- schedctl_finish_sigblock rc=426610995637088 errno=0 retaddr=schedctl_finish_sigblock+0x4c
6 -> fsig
6 -> sigdiffset
6 <- sigdiffset rc=2890567951920 errno=0 retaddr=sigdiffset+0x2c
6 <- fsig rc=0 errno=0 retaddr=fsig+0x150
6 -> fsig
6 -> sigdiffset
6 <- sigdiffset rc=2890567951920 errno=0 retaddr=sigdiffset+0x2c
6 -> lowbit
6 <- lowbit rc=18 errno=0 retaddr=lowbit+0x84
6 <- fsig rc=18 errno=0 retaddr=fsig+0x170
6 -> sigdeq
6 <- sigdeq rc=426611868820408 errno=0 retaddr=sigdeq+0x138
6 -> isjobstop
6 <- isjobstop rc=0 errno=0 retaddr=isjobstop+0x124
If we look at the return code of fsig() we are told the signal that we received. It’s 18, or a SIGCHLD, which is
#define SIGCLD 18 /* child status change */
This means that we’ve had a child process exit and notify us of its exit. So, what was it?
With some updated DTrace we see the following (column 1 is a nanosecond timestamp) during one of the failed runs. Note that in this run application process that we are interested in is PID 4624.
1733723047273935775 application [4624] openat called
1733723047273935775 application [4624] Created PID 4626
libc.so.1`__forkx+0xc
...
application`main+0xc
application`_start+0x64
1733723047273935775 application [4626] exec /etc/application_parallel_startup
1733723047273935775 application [4626] exec /usr/xpg4/bin/sh
1733723047283936690 sh [4628] exec /usr/bin/date
1733723047283936690 sh [4626] exit status 1 ppid=4624
1733723047283936690 sh [4626] post SIGCLD to 4624 (application) *****
1733723047283936690 application [4624] openat returns
We see that we’ve started up “/etc/application_parallel_startup”. Seeing it exec the pathname and then exec /bin/sh shows that it’s a shell script. We see it run a date and then exit.
Having a look at the script, that is basically what it does. It’s there for local customisation of other stuff we may want to do at the time we start the application.
It is noteworthy that the application has used functionality that it has to start longer running daemons to run this, and does NOT do a waitpid() on it.
It looks like we are simply unfortunate enough that about three times out of ten we get that SIGCLD in that 1.5ms window of the openat().
As a workaround, I’ve added a “sleep 5” to the end of that script so it finishes much later (I could probably have made it smaller, or even just run another command to give us a context switch, which would add up to 10ms), but this looks to work.
Thoughts
I think that it would be better practice for the application to actually do a waitpid() on this short lived process. That would definitely prevent this problem occurring.
After a little more thought, I am also concerned that this change to devzvol_lookup() has introduced a bug.
In the normal course of events, if the application was executing in user space, the receipt of a SIGCLD with the default actions would not impact the process. This change to devzvol_lookup() means tat if we happen to be in a system call that it DOES impact the process, and in this instance impacted it pretty badly.
My feeling is that the behaviour in devzvol_lookup() of signal handling should have the same results as handling that signal in user space; but we’ll have to see how far that opinion gets me with Solaris Engineering.
Updating Ops Center Agents
It has been my (unfortunate) experience that after applying the Final Aggregate Patch for Ops Center, which delivers the v12.4.0.3201 Agent, that
- We don’t always get the option to upgrade the target in the Web UI
- Running “upgrade_all_agents” on the Ops Center CLI does not always do all of the targets
- some of the targets have the agent in an unclear status
I found two documents particularly helpful here:
- Doc ID 1991863.1 OPS Center Agent Configuration Fails with: “Cannot register SMF service for instance: [scn-agent]” – https://support.oracle.com/epmos/faces/SearchDocDisplay?_afrLoop=430249586014284&_afrWindowMode=0&_adf.ctrl-state=dbni93kzd_4
- Chapter 2 “Manage Assets” of the Ops Center 12c Release 4 “Enterprise Manager Ops Center Configuration Reference” – https://docs.oracle.com/cd/ops-center-12.4/doc.1240/e59970/GUID-17FB2DF7-D3C0-4132-A65B-FBE1B557AFD6.htm#GUID-CBEC20F8-E69B-4ED6-B174-C4909DB2A2B3
It should be noted that the previous Doc points to the v12.2 version of this page.
In Short, I created a tarball (/tmp/OC.tar.gz) containing all of the following in a directory called “OC”
- mytoken – The token on the last line of “/var/opt/sun/xvm/persistence/scn-proxy/connection.properties”, with
- backslash characters removed
- all text up to and including the first ‘=’ removed
- The extracted zip file from “
/var/opt/sun/xvm/images/agent/OpsCenterAgent.SolarisIPS.all.12.4.0.3201.zip”
Given a list of targets hosts in ${LDOMS} and the knowledge that the admin account on these targets can perform “svcadm” functions with “pfexec”, and that I have a script that can be used to disable puppet in the global zone and any non-global-zones (our puppet agents run every hour and do a few pkg commands that could interfere with running pkg updates in the global zone), we run
for ldom in ${LDOMS}; do echo === ${ldom}; scp /tmp/OC.tar.gz ${ldom}:/var/tmp; ssh ${ldom} pfexec puppetsvc.sh -d; done
Then we ssh to each LDOM and as root run
OPSCENTER_IP={IP Address of Ops Center Server}
cd /var/tmp ; tar zvxpf OC.tar.gz ; cd OC ;\
cacaoadm prepare-uninstall ;\
/opt/SUNWxvmoc/bin/agentadm unconfigure ;\
OpsCenterAgent/install -p ${OPSCENTER_IP} \;
/opt/SUNWxvmoc/bin/agentadm configure -t /var/tmp/OC/mytoken -x ${OPSCENTER_IP}
Answering any questions that may be posed. When done, clean up after ourselves and restart puppet.
for ldom in ${LDOMS}; do echo === ${ldom}; ssh ${ldom} 'pfexec puppetsvc.sh -e ; cd /var/tmp ; rm -rf OC OC.tar.gz' ; done
The above seems to have a 100% success rate for the targets that had not been updated by other means.
Ops Center – Recovering Credential Information
I recently found myself in the position of needing to remove and reinstall Ops Center. While gathering information that I would need to recreate everything, I got stuck on some credentials (specifically IPMI and SNMPv3).
I can’t be the only one who has needed to try to pull this kind of information back out of the Ops Center database.
Now there is obviously no way to do this in the Ops Center BUI, so enter the Ops Center CLI.
# /opt/SUNWoccli/bin/oc
Oracle Enterprise Manager Ops Center
Copyright (c) 2007, 2019 Oracle and/or its affiliates. All rights reserved.
Use is subject to license terms.
Use the -connect- command to get started
The tab key will always show the available command set
xvmSh > connect
localhost > credentials
localhost/credentials > list -l
Driver Credentials:
ID | Name | Type | Description | Attributes |
-----------------------------------------------------------------------------------------------------------------------------------------------------------------
166404852| OCopscenter | SNMPV3 | None |snmpUserName=OCopscenter,authPassword=*****,authProtocol=MD5,privPassword=*****,privProtocol=DES
14351 | SNMP1 | SNMPV3 | SNMPv3 Credentials |snmpUserName=OCipmi,authPassword=*****,authProtocol=SHA,privPassword=*****,privProtocol=AES
166404851| Prod IPMI | IPMI |Production IPMI Credentials| login=root,sharedSecret=*****
Now we see the sensitive information blanked out. The python code that does this is in /opt/SUNWoccli/share/lib/occli/xvm_discovery.py in the code:
137 _hide_data=('protocol','id')
138 _sensitive_data_display_value="*****"
139
140 def strip_secure_data(key, value):
141 """
142 returns (boolean,string) True if value should be shown
143 """
144 if key in _hide_data:
145 return (False,"")
146
147 if not value:
148 #nothing to be stripped
149 return (True,value)
150
151 from com.oracle.sysman.services.discovery import DriverCredentials
152 if key in DriverCredentials.SENSITIVE_DATA:
153 return (True,_sensitive_data_display_value)
154 return (True, value)
Now, if we comment out that conditional
152 #if key in DriverCredentials.SENSITIVE_DATA:
153 # return (True,_sensitive_data_display_value)
154 return (True, value)
Then we will get the hidden information
localhost/credentials > list -l
Driver Credentials:
ID | Name | Type | Description | Attributes |
-----------------------------------------------------------------------------------------------------------------------------------------------------------------
166404852| OCopscenter | SNMPV3 | None |snmpUserName=OCopscenter,authPassword=PasswordInClearText,authProtocol=MD5,privPassword=PasswordInClearText,privProtocol=DES
14351 | SNMP1 | SNMPV3 | SNMPv3 Credentials |snmpUserName=OCipmi,authPassword=PasswordInClearText,authProtocol=SHA,privPassword=PasswordInClearText,privProtocol=AES
166404851| Prod IPMI | IPMI |Production IPMI Credentials| login=root,sharedSecret=PasswordInClearText
Don’t forget to change it back.
How I addressed a geo-infrastructure geocluster failure
Description of Issue
We are running a Two Cluster geocluster; ProdNFS and DRNFS. On the production side we can no longer run any of the commands like geoadm or geopg. The geo-infrastructure resource group is in “Pending_online”. the geo-failovercontrol resource is faulted.
Investigation
It looks like we are stuck in the “infinite loop” in /usr/cluster/lib/rgm/rt/hamasa/cmas_service_ctrl_start with “cacaoadm list-trusted certs” returning status 11.
39 CLASSPATH=${CMASSPATH}:${CACAOPATH}:${JDMKPATH}:${JMXPATH}:${OPENJDMKPATH}
40 CACAOADM=${ROOT_DIR}/usr/sbin/cacaoadm
41
42 # is cacao ready?
43 if [ ! -x ${CACAOADM} ]
44 then
45 status=100 # exit err
46 else
47 ${CACAOADM} list-trusted-certs > /dev/null 2>&1
48 status=$?
49
50 if [ $status -ne 0 ]
51 then
52
53 if [ $status -eq 3 ] # cacao not running
54 then
55 ${CACAOADM} enable > /dev/null 2>&1
56 ${CACAOADM} start > /dev/null 2>&1
57 fi
58
59 # infinite loop OK; rgm will kill us if we can't exit on our own
60 while [ $status -ne 0 ]
61 do
62 /usr/bin/sleep 5
63 ${CACAOADM} list-trusted-certs > /dev/null 2>&1
64 status=$?
65 done
66
67 fi
68
69 java $JAVA_OPTS -classpath ${CLASSPATH} ServiceControl $AGENT $COMMAND $FAILOVER_GROUP
70 status=$?
71 fi
Specifically, we are timing out in the infinite loop in lines 60-65. the cacaoadm command is returning status 13, for which the man page is not particularly helpful. I also notice if I run “cacaoadm list-trusted-certs -v” on the other side of the geocluster that I see we have the certificates from PRODNFS being listed as having expired mid last month, as well as the actual signing cacao certificate being expired. This last turns out to be our issue.
root@*****:/etc/cacao/instances/default/security/jsse# keytool -keystore /etc/cacao/instances/default/security/jsse/keystore -list -v Enter keystore password: Keystore type: jks Keystore provider: SUN Your keystore contains 1 entry Alias name: cacao_agent Creation date: 16/07/2020 Entry type: PrivateKeyEntry Certificate chain length: 2 Certificate[1]: Owner: CN=*****_agent Issuer: CN=*****_ca Serial number: 3a4c1ff9 Valid from: Tue Jun 16 20:52:46 PGT 2020 until: Fri Sep 16 20:52:46 PGT 2022 Certificate fingerprints: SHA1: 13:BE:B5:FC:BE:E6:69:98:ED:76:F3:41:C0:16:0F:09:30:01:FF:19 SHA256: F4:B2:BF:EF:39:D8:17:2A:69:3E:D0:72:6C:FC:9D:30:6A:7B:00:7B:B3:35:F6:80:90:4F:95:1F:77:7F:34:8C Signature algorithm name: SHA256withRSA Subject Public Key Algorithm: 2048-bit RSA key Version: 3 Extensions: #1: ObjectId: 2.5.29.19 Criticality=false BasicConstraints:[ CA:false PathLen: undefined ] #2: ObjectId: 2.5.29.14 Criticality=false SubjectKeyIdentifier [ KeyIdentifier [ 0000: 1A 7A 19 7B 64 66 90 C8 A3 69 05 3D 16 57 59 27 .z..df...i.=.WY' 0010: 07 04 A2 BB .... ] ] Certificate[2]: Owner: CN=*****_ca Issuer: CN=*****_ca Serial number: 6997e07d Valid from: Tue Jun 16 20:52:46 PGT 2020 until: Fri Sep 16 20:52:46 PGT 2022 Certificate fingerprints: SHA1: 84:DE:5D:CD:D9:99:2A:F3:81:75:7C:D0:0A:0E:0B:07:EC:73:36:D6 SHA256: 8C:47:32:FA:0E:2F:9A:48:8A:C8:59:C4:8B:16:89:34:63:DA:19:D1:32:3A:28:30:4E:2D:33:7D:FD:74:86:A4 Signature algorithm name: SHA256withRSA Subject Public Key Algorithm: 2048-bit RSA key Version: 3 Extensions: #1: ObjectId: 2.5.29.19 Criticality=true BasicConstraints:[ CA:true PathLen:1 ] ******************************************* ******************************************* These are the certificates generated when we initialised this cacao instance. They appear to have been generated with a two year lifetime, which is a known issue with Oracle support. There are documents available to address the issue with Ops Center, but not with this Solaris Cluster and Geocluster.
Solution
We can actually kill two birds with one stone here. The original process that I got from Oracle had us using geops to remove trust and then use geops add-trust to regenerate the certificates. The problem with this approach is that given the cacao_agent certificate has also expired, so the geocluster commands can’t communicate with it.
If, however, we stop the cacao daemons and generate new certificates then we
- Have a new agent certificate with a 20 year expiry
- Remove the expired geocluster certificates from the trust store
This process is going to describe replacing all of the certificates as I note that the ones on DRNFS are due to expire in the next month or so as well.
On each side of the geocluster we need to
- On the node that we originally had the certificates generated, shut down cacao, regenerate the agent certificate and restart it
# cacaoadm stop # cacaoadm create-keys -f # cacaoadm start
- Make a tarball of /etc/cacao/instances/default/security and copy it to the other node of this cluster
# cd /etc/cacao/instances/default/security # tar cf /tmp/cacao.tar * # scp /tmp/cacao.tar othernode://tmp/cacao.tar
- On the other node, stop cacao, extract the tarball and restart it.
# cacaoadm stop # cd /etc/cacao/instances/default/security # tar xpf /tmp/cacao.tar # cacaoadm start
Once this has been done on each geocluster we can add trust again.
On the production geocluster (ProdNFS)
# geops add-trust -c DRNFS
On the DR geocluster (DRNFS)
# geops add-trust -c ProdNFS
ProdNFS and DRNFS are the names of the respective geoclusters, you’d need to replace those with the ones that you chose.
Once I had done this, I could again run the geocluster command and control programs.
OBDX 18.0.0 Installer and Solaris 11.4 beyond sru 21
This one took us a couple of weeks to work out and I’m writing this to save others the pain we’ve just been through.
For whatever reason our customer is using this particular version of Oracle OBDX. The installer code is written in python 2.7.
We recently had the non-production systems upgraded to Solaris 11.4 sru 32 taking us to something much more current, with a view to pushing this out to pre-prod and then pro/dr and moving to more regular updates.
We noticed the installer failing, being unable to find the cx_Oracle packages for python 2.7.
To cut a long story short, in 11.4 sru21 we had the following package changes:
- library/python/cx_oracle
- Among other things, a dependency on runtime/python-27 was removed
- library/python/cx_oracle-27
- This package delivered the cx_Oracle.so python module and also had a requirement of developer/oracle/odpi-23. The files were removed from the package
- developer/oracle/odpi-23
- Files required by cx_Oracle.so were removed and this became an empty package.
I can only assume that there was not a PSARC license between Solaris and the folks looking after this product installer or appropriate notifications would have occurred before the removal.
Because of the way that dependencies have been done, it is not possible to just re-install the packages.
It is possible to work around this issue by installing the following missing files on the system we wish to do the install on (and probably removing them afterwards as they are not being tracked in the packaging system). In my case I gathered them from a system running Solaris 11.4 SRU 20.
usr/lib/python2.7/vendor-packages/64/cx_Oracle.so usr/lib/python2.7/vendor-packages/cx_Oracle.so usr/lib/libodpic.so.1 usr/lib/sparcv9/libodpic.so.1
A cleaner solution(and the one I took as I’d like to have these tracked by IPS) is to take the files that constitute the old packages and create a new IPS package. I ended up creating a local package called “obdx-install-prereq” that contains:
usr/include/odpi-230/dpi.h usr/lib/libodpic.so.1 usr/lib/python2.7/vendor-packages/64/cx_Oracle.so usr/lib/python2.7/vendor-packages/cx_Oracle-6.3-py2.7.egg-info/PKG-INFO usr/lib/python2.7/vendor-packages/cx_Oracle-6.3-py2.7.egg-info/SOURCES.txt usr/lib/python2.7/vendor-packages/cx_Oracle-6.3-py2.7.egg-info/dependency_links.txt usr/lib/python2.7/vendor-packages/cx_Oracle-6.3-py2.7.egg-info/top_level.txt usr/lib/python2.7/vendor-packages/cx_Oracle.so usr/lib/sparcv9/libodpic.so.1 usr/share/man/man3lib/libodpic-230.3lib
We currently have a support call open with Oracle where I have requested the creation of Support documentation (with appropriate application and Solaris metadata) to describe this issue and how to work around it. The version of OBDX we are using is well beyond “current – 1”, so I don’t expect any fixes in the installer, but documentation of the workaround would be helpful.
For those interested, my package manifest (obdx-install-prereq.p5m) looks like:
set name=pkg.fmri value="obdx-install-prereq@11.4-11.4.21.1" set name=pkg.summary value="OBDX Installer prerequisites" set name=pkg.description value="Copies of the files in packages removed in sru21 that the OBDX installer requires, incorporating pkg:library/python/cx_python-27@6.3-11.4.19, pkg:developer/oracle/odpi-230@2.3.0-11.4.19 and their dependencies" set name=variant.arch value=sparc depend fmri=pkg:/library/python/cx_oracle@7.1.1 type=require depend fmri=database/oracle/instantclient-122 fmri=database/oracle/instantclient-121 type=require-any file path=usr/lib/libodpic.so.1 mode=555 owner=root group=bin file path=usr/lib/sparcv9/libodpic.so.1 mode=555 owner=root group=bin file path=usr/include/odpi-230/dpi.h mode=444 owner=root group=bin file path=usr/share/man/man3lib/libodpic-230.3lib mode=444 owner=root group=bin file path=usr/lib/python2.7/vendor-packages/64/cx_Oracle.so mode=555 owner=root group=bin file path=usr/lib/python2.7/vendor-packages/cx_Oracle.so mode=555 owner=root group=bin file path=usr/lib/python2.7/vendor-packages/cx_Oracle-6.3-py2.7.egg-info/PKG-INFO mode=555 owner=root group=bin file path=usr/lib/python2.7/vendor-packages/cx_Oracle-6.3-py2.7.egg-info/SOURCES.txt mode=555 owner=root group=bin file path=usr/lib/python2.7/vendor-packages/cx_Oracle-6.3-py2.7.egg-info/dependency_links.txt mode=555 owner=root group=bin file path=usr/lib/python2.7/vendor-packages/cx_Oracle-6.3-py2.7.egg-info/top_level.txt mode=555 owner=root group=bin
