I believe the original concern about making system/filesystem/local part
of single-user was that it changes the definition of single-user. The
zones team was involved in that discussion, and I've just tried to
re-involve them in the resolution discussions. (And have cc'ed them
here. Apologies for the crosspost.) It may just be morning, but I'm
currently having a hard time finding that argument too compelling on its
own -- it seems there must have been something more specific.

Creating a patch milestone has previously been my preferred direction
(as it requires actual definition and design of what the patching
frameworks need, rather than just hoping that single user is
appropriate), but I'm no longer certain this is a sensible investment,
as my understanding of IPS is that they're sticking to their guns and
there will be no live patching of the OS that leads us to these types of
scenarios. Thankfully.

Given that, I wonder if a different option is to confirm that
system/filesystem/local is idempotent (and not already running), and
then have patchadd just run its method directly. It leaves a bad taste
in my mouth, but then again so does the fact that we've got two
different patching systems which require the system to be in different
states when they run.

But, again, I think the zones team explored many of these options in the
first round and would like to make sure they have a chance to weigh in.

liane (in theory, on vacation today.)

Renaud Manus wrote:
> A solution could be to move system/filesystem/local in the single-user
> milestone.
>
> -- Renaud
>
> Jordan Brown wrote:
>> Could some SMFers please weigh in on 6725004 and give some opinions on
>> how best to address it?
>>
>> Here's a summary of the problem:
>>
>> Sun Update Connection Enterprise attempts to install "single-user"
>> patches using an rcS.d script. This has historically been a problem,
>> since zone roots may not have been mounted yet. Patchadd was recently
>> changed to partially work around this problem, by enabling
>> system/filesystem/local, but when that mechanism is triggered from an
>> rcS.d script (or, presumably, from a service on which
>> milestone/single-user depends), it yields a deadlock -
>> system/filesystem/local can't be run, because milestone/single-user
>> hasn't been reached. (This is, I believe, in addition to the technical
>> issues surrounding performing SMF operations from an rcS.d script.)
>>
>> I believe that the most truly correct way to address this problem is to
>> have a deferred patching service that depends on
>> system/filesystem/local, and on which all other later services depend.
>> However, I think the only way to implement such a service would be to
>> modify the dependencies of those later services, and that seems awkward.
>> (One could also rename system/filesystem/local, retain the original
>> name as something of a milestone, and insert the deferred patching
>> service as a new service between the renamed s/f/l and the original
>> s/f/l, but that seems quite ugly.)
>> _______________________________________________

Liane Praza wrote:
> I believe the original concern about making system/filesystem/local part
> of single-user was that it changes the definition of single-user. The
> zones team was involved in that discussion, and I've just tried to
> re-involve them in the resolution discussions. (And have cc'ed them
> here. Apologies for the crosspost.) It may just be morning, but I'm
> currently having a hard time finding that argument too compelling on its
> own -- it seems there must have been something more specific.
>
> Creating a patch milestone has previously been my preferred direction
> (as it requires actual definition and design of what the patching
> frameworks need, rather than just hoping that single user is
> appropriate), but I'm no longer certain this is a sensible investment,
> as my understanding of IPS is that they're sticking to their guns and
> there will be no live patching of the OS that leads us to these types of
> scenarios. Thankfully.
not clear when this will be included, today IPS allows you to update SUNWcsr for instance
in the running system ( ie pkg update as opposed to pkg image-update which updates an
alternate BE ), I have seen issues with compilers failing due to SUNWcsr and SUNWtoo
getting out of sync, because user updated the live system.
Also IPS is not relevant to s10 users, who will be patching their systems for some time to
come. ( i.e. many years )

I'd like to hear why fs/local being incorporated into single user milestone was not
considered a good idea.
The patch milestone does sound interesting, as it would involve an agreed consensus

cheers
Enda

>
> Given that, I wonder if a different option is to confirm that
> system/filesystem/local is idempotent (and not already running), and
> then have patchadd just run its method directly. It leaves a bad taste
> in my mouth, but then again so does the fact that we've got two
> different patching systems which require the system to be in different
> states when they run.
>
> But, again, I think the zones team explored many of these options in the
> first round and would like to make sure they have a chance to weigh in.
>
> liane (in theory, on vacation today.)
>
> Renaud Manus wrote:
>> A solution could be to move system/filesystem/local in the single-user
>> milestone.
>>
>> -- Renaud
>>
>> Jordan Brown wrote:
>>> Could some SMFers please weigh in on 6725004 and give some opinions on
>>> how best to address it?
>>>
>>> Here's a summary of the problem:
>>>
>>> Sun Update Connection Enterprise attempts to install "single-user"
>>> patches using an rcS.d script. This has historically been a problem,
>>> since zone roots may not have been mounted yet. Patchadd was recently
>>> changed to partially work around this problem, by enabling
>>> system/filesystem/local, but when that mechanism is triggered from an
>>> rcS.d script (or, presumably, from a service on which
>>> milestone/single-user depends), it yields a deadlock -
>>> system/filesystem/local can't be run, because milestone/single-user
>>> hasn't been reached. (This is, I believe, in addition to the technical
>>> issues surrounding performing SMF operations from an rcS.d script.)
>>>
>>> I believe that the most truly correct way to address this problem is to
>>> have a deferred patching service that depends on
>>> system/filesystem/local, and on which all other later services depend.
>>> However, I think the only way to implement such a service would be to
>>> modify the dependencies of those later services, and that seems awkward.
>>> (One could also rename system/filesystem/local, retain the original
>>> name as something of a milestone, and insert the deferred patching
>>> service as a new service between the renamed s/f/l and the original
>>> s/f/l, but that seems quite ugly.)
>>> _______________________________________________
> _______________________________________________
> zones-discuss mailing list
> zones-discuss@opensolaris.org

Enda O'Connor ( Sun Micro Systems Ireland) writes:
> alternate BE ), I have seen issues with compilers failing due to SUNWcsr and SUNWtoo
> getting out of sync, because user updated the live system.

I think I understand that problem, and I don't think it has anything
to do with a live update.

The underlying problem there is that there's a hidden dependency built
into the way /usr/bin/ld (in SUNWtoo) is constructed. It depends on a
private version of libld.so.4 (in SUNWcsr) -- one that gets
incremented occasionally.

Thus, if you upgrade SUNWcsr, you must also upgrade SUNWtoo, even if
you didn't want to, and even though the package dependency runs the
other direction (SUNWtoo depends on SUNWcsr, not vice-versa).

Since SUNWcsr is one of the common packages on which everyone depends,
it's at high risk of being upgraded frequently. It'll be upgraded
when someone installs a new package, or when someone upgrades an
existing one, because it's a dependency.

But the system doesn't seem to know that SUNWtoo is tightly bound to a
special library version and must be upgraded as well. It's really a
mutual dependency.

I don't think we know how many of these sorts of special dependencies
exist in ON. Except for the controlled environment of patches, we've
always assumed synchronous delivery of everything built in ON as part
of our design.

--
James Carlson, Solaris Networking <james.d.carlson@sun.com>
Sun Microsystems / 35 Network Drive 71.232W Vox +1 781 442 2084
MS UBUR02-212 / Burlington MA 01803-2757 42.496N Fax +1 781 442 1677

Liane Praza wrote:
> It leaves a bad taste
> in my mouth, but then again so does the fact that we've got two
> different patching systems which require the system to be in different
> states when they run.

Three :-)

Well, sort of.

All of them agree that the system should be "in single user mode". The
difference is how you get there, and *exactly* what it means.

The legacy is that it means interactive, shell-prompt single-user mode.

We're trying to implement (or, rather keep supporting) automatic
installation of these patches, and interactive shell prompt single-user
mode isn't reasonable for automatic installation. (There are one or two
really gross ways to do it - boot the system to single user mode,
including a service that runs before single user mode, puts itself in
the background, and waits for SMF to reach milestone/single-user, then
*while the interactive single-user login is available* do the patch
installs in the background and reboot the system.)

The two automatic schemes attempt to install patches at a system state
that is epsilon different from interactive single-user mode. (Either
epsilon earlier than interactive or epsilon later than single-user would
be OK, with epsilon earlier being easier to implement and epsilon later
being slightly more desirable.) One, UCE, does this by running its
automatic mechanism from a high-numbered rcS.d script. The other,
SunUC-S (a.k.a. smpatch or Update Manager) does it by running its
automatic mechanism from a SMF service that runs during system shutdown
at a point intended to be equivalent to single-user mode.

(The reason that SunUC-S does its work during shutdown rather than
startup is that most of these patches require a subsequent reboot, and
doing the patching during shutdown means that there's only one reboot.
This is better in theory than in practice, because system shutdown is
not as well controlled as system startup and many services are left
running until the bitter end.)

Anyhow, the goal here is to find at least one strategy for automatic
installation of these patches that everybody can agree to support. Of
course, my slight preference is that it be installation during shutdown
(because that reduces the number of reboots), but my expectation is that
it will involve installation epsilon from interactive single-user mode,
with a subsequent reboot required.

Add a new service "do-single-user-patch", make it depend on filesystem-local.
This service is typically disabled. This service will add the patch(es)
and reboot.

In rcS.d/Swhatever, do:

if (we want to do-single-user-patchs)
assert(we are currently booting to single-user milestone)
svcadm enable -tr do-single-user-patch
fi

-Steve L.


On Tue, Aug 19, 2008 at 08:42:58AM -0700, Liane Praza wrote:
> I believe the original concern about making system/filesystem/local part
> of single-user was that it changes the definition of single-user. The
> zones team was involved in that discussion, and I've just tried to
> re-involve them in the resolution discussions. (And have cc'ed them
> here. Apologies for the crosspost.) It may just be morning, but I'm
> currently having a hard time finding that argument too compelling on its
> own -- it seems there must have been something more specific.
>
> Creating a patch milestone has previously been my preferred direction
> (as it requires actual definition and design of what the patching
> frameworks need, rather than just hoping that single user is
> appropriate), but I'm no longer certain this is a sensible investment,
> as my understanding of IPS is that they're sticking to their guns and
> there will be no live patching of the OS that leads us to these types of
> scenarios. Thankfully.
>
> Given that, I wonder if a different option is to confirm that
> system/filesystem/local is idempotent (and not already running), and
> then have patchadd just run its method directly. It leaves a bad taste
> in my mouth, but then again so does the fact that we've got two
> different patching systems which require the system to be in different
> states when they run.
>
> But, again, I think the zones team explored many of these options in the
> first round and would like to make sure they have a chance to weigh in.
>
> liane (in theory, on vacation today.)
>
> Renaud Manus wrote:
> > A solution could be to move system/filesystem/local in the single-user
> > milestone.
> >
> > -- Renaud
> >
> > Jordan Brown wrote:
> >> Could some SMFers please weigh in on 6725004 and give some opinions on
> >> how best to address it?
> >>
> >> Here's a summary of the problem:
> >>
> >> Sun Update Connection Enterprise attempts to install "single-user"
> >> patches using an rcS.d script. This has historically been a problem,
> >> since zone roots may not have been mounted yet. Patchadd was recently
> >> changed to partially work around this problem, by enabling
> >> system/filesystem/local, but when that mechanism is triggered from an
> >> rcS.d script (or, presumably, from a service on which
> >> milestone/single-user depends), it yields a deadlock -
> >> system/filesystem/local can't be run, because milestone/single-user
> >> hasn't been reached. (This is, I believe, in addition to the technical
> >> issues surrounding performing SMF operations from an rcS.d script.)
> >>
> >> I believe that the most truly correct way to address this problem is to
> >> have a deferred patching service that depends on
> >> system/filesystem/local, and on which all other later services depend.
> >> However, I think the only way to implement such a service would be to
> >> modify the dependencies of those later services, and that seems awkward.
> >> (One could also rename system/filesystem/local, retain the original
> >> name as something of a milestone, and insert the deferred patching
> >> service as a new service between the renamed s/f/l and the original
> >> s/f/l, but that seems quite ugly.)
> >> _______________________________________________
> _______________________________________________
> zones-discuss mailing list
> zones-discuss@opensolaris.org

On Tue, 2008-08-19 at 11:36 -0700, Steve Lawrence wrote:
> Add a new service "do-single-user-patch", make it depend on filesystem-local.
> This service is typically disabled. This service will add the patch(es)
> and reboot.

The same could be done with a custom milestone which might be less
confusing. I like the elegance of that solution as it relies less on
what works now in favor of a defined environment which presumably would
get better coverage for zones and zfs [zone]root. And further
refinement would only impact patching rather than the booting process
as a whole.

rc scripts doing things with SMF seem a permanent solution to a
temporary problem. In my virtual universe there are no rc scripts
And then the alarm clock goes off and I return to reality. But it does
promote rc hackery rather than fixing the problem in SMF where it
belongs.

reboot -- -m milestone=patchinstall seems elegantly simple.



Bob

Bob Netherton wrote:
> And further
> refinement would only impact patching rather than the booting process
> as a whole.

Hmm. I don't know how to have a service that runs when a particular
milestone is selected, that *doesn't* run when "all" is selected.
(Other than by dynamically enabling and disabling it.)

> rc scripts doing things with SMF seem a permanent solution to a
> temporary problem. In my virtual universe there are no rc scripts
> And then the alarm clock goes off and I return to reality. But it does
> promote rc hackery rather than fixing the problem in SMF where it
> belongs.

Agreed. Besides, I believe that SMF is locked while rc scripts are
running, and that any attempt to manipulate it deadlocks.

There are related schemes that could work, but the problem is getting
them properly sequenced into system startup.

> reboot -- -m milestone=patchinstall seems elegantly simple.

Plausible, though it doesn't exactly fit the current application usage
model. At the moment, the reboot might or might not be triggered by the
patching application. The patching application leaves the system set up
to do the patching at the next shutdown/reboot, whenever that might be.
(For SunUC-S's shutdown-time processing, it's require that that reboot
be via the "clean" mechanisms - init, shutdown - so that the processing
gets done.)

This scheme would require either

1) having the patch application set the default milestone, and then
having the startup-time processing set it back, or
2) having the patch application do the reboot.

There's still the issue with how to keep this service from running when
you boot to "all".

Hmm. How does single-user-mode login work? What stops it from running
on a normal boot? Is it a special case?

---

BTW: I'm not in a position to commit the patch applications. I'm in
the middle here because I'm relatively familiar with all of the players
and the issues, but in my day job I'm not responsible for *any* of them.

So you want to be able to interrupt any boot to any milestone, and instead do
the patch processing if a patch is pending. You basically want to interrupt
the current milestone, and instead just boot to filesystem-local and do the
patching.

The question is, can the smf milestone be changed mid-milestone?

My test shows that it can. How about:

1. Create patch-test-service, on which single-user depends. This will
"svcadm milestone patch-install-milestone" if a patch needs to be
installed. This service is always enabled.

2. Create patch-install-milestone, which depends on patch-install-service
below.

3. Create patch-install service, which depends on:
single-user
filesystem-local

This service is always enabled. It will install a patch if it is pending,
otherwise, do nothing. If the service fails, it might need to:

# svcadm milestone single-user

So that a maintenance prompt will be appear on the console. This might
not be necessary. you might get this anyway, as console-login is not
reached.

It should be ok to issue smf commands from an smf service, as long as they
do not try to do any synchronous operations (-s).

This approach is also good because an explicit boot to single user WILL NOT
attempt to install pending patches.

Disabling the patch-test and patch-install services will disable the
automatic installation of pending patches on reboot.

Thoughts?

-Steve L.

On Tue, Aug 19, 2008 at 01:03:55PM -0700, Jordan Brown wrote:
> Bob Netherton wrote:
> > And further
> > refinement would only impact patching rather than the booting process
> > as a whole.
>
> Hmm. I don't know how to have a service that runs when a particular
> milestone is selected, that *doesn't* run when "all" is selected.
> (Other than by dynamically enabling and disabling it.)
>
> > rc scripts doing things with SMF seem a permanent solution to a
> > temporary problem. In my virtual universe there are no rc scripts
> > And then the alarm clock goes off and I return to reality. But it does
> > promote rc hackery rather than fixing the problem in SMF where it
> > belongs.
>
> Agreed. Besides, I believe that SMF is locked while rc scripts are
> running, and that any attempt to manipulate it deadlocks.
>
> There are related schemes that could work, but the problem is getting
> them properly sequenced into system startup.
>
> > reboot -- -m milestone=patchinstall seems elegantly simple.
>
> Plausible, though it doesn't exactly fit the current application usage
> model. At the moment, the reboot might or might not be triggered by the
> patching application. The patching application leaves the system set up
> to do the patching at the next shutdown/reboot, whenever that might be.
> (For SunUC-S's shutdown-time processing, it's require that that reboot
> be via the "clean" mechanisms - init, shutdown - so that the processing
> gets done.)
>
> This scheme would require either
>
> 1) having the patch application set the default milestone, and then
> having the startup-time processing set it back, or
> 2) having the patch application do the reboot.
>
> There's still the issue with how to keep this service from running when
> you boot to "all".
>
> Hmm. How does single-user-mode login work? What stops it from running
> on a normal boot? Is it a special case?
>
> ---
>
> BTW: I'm not in a position to commit the patch applications. I'm in
> the middle here because I'm relatively familiar with all of the players
> and the issues, but in my day job I'm not responsible for *any* of them.
> _______________________________________________
> zones-discuss mailing list
> zones-discuss@opensolaris.org

> 2. Create patch-install-milestone, which depends on patch-install-service
> below.

The patch-install-milestone could also depend on single-user and
filesystem-local so that it is generally useful for admins manually
installing patches as well, even if they don't have the patch-test
and patch-install services, but want to safely install patches
manually when they have zones on local filesystems.

-Steve L.

>
> On Tue, Aug 19, 2008 at 01:03:55PM -0700, Jordan Brown wrote:
> > Bob Netherton wrote:
> > > And further
> > > refinement would only impact patching rather than the booting process
> > > as a whole.
> >
> > Hmm. I don't know how to have a service that runs when a particular
> > milestone is selected, that *doesn't* run when "all" is selected.
> > (Other than by dynamically enabling and disabling it.)
> >
> > > rc scripts doing things with SMF seem a permanent solution to a
> > > temporary problem. In my virtual universe there are no rc scripts
> > > And then the alarm clock goes off and I return to reality. But it does
> > > promote rc hackery rather than fixing the problem in SMF where it
> > > belongs.
> >
> > Agreed. Besides, I believe that SMF is locked while rc scripts are
> > running, and that any attempt to manipulate it deadlocks.
> >
> > There are related schemes that could work, but the problem is getting
> > them properly sequenced into system startup.
> >
> > > reboot -- -m milestone=patchinstall seems elegantly simple.
> >
> > Plausible, though it doesn't exactly fit the current application usage
> > model. At the moment, the reboot might or might not be triggered by the
> > patching application. The patching application leaves the system set up
> > to do the patching at the next shutdown/reboot, whenever that might be.
> > (For SunUC-S's shutdown-time processing, it's require that that reboot
> > be via the "clean" mechanisms - init, shutdown - so that the processing
> > gets done.)
> >
> > This scheme would require either
> >
> > 1) having the patch application set the default milestone, and then
> > having the startup-time processing set it back, or
> > 2) having the patch application do the reboot.
> >
> > There's still the issue with how to keep this service from running when
> > you boot to "all".
> >
> > Hmm. How does single-user-mode login work? What stops it from running
> > on a normal boot? Is it a special case?
> >
> > ---
> >
> > BTW: I'm not in a position to commit the patch applications. I'm in
> > the middle here because I'm relatively familiar with all of the players
> > and the issues, but in my day job I'm not responsible for *any* of them.
> > _______________________________________________
> > zones-discuss mailing list
> > zones-discuss@opensolaris.org
> _______________________________________________
> zones-discuss mailing list
> zones-discuss@opensolaris.org

Steve Lawrence wrote:
> So you want to be able to interrupt any boot to any milestone, and instead do
> the patch processing if a patch is pending. You basically want to interrupt
> the current milestone, and instead just boot to filesystem-local and do the
> patching.

That would be my initial plan, at least roughly. I wouldn't think of it
in terms of going to a different milestone, per se, but rather that
there is a point in system startup where we look to see if there are any
patches pending. If not, we continue; if so, we apply them. If they
require reboot, we reboot, else we let the system continue to come up.

That point needs to be early enough so that the system is quiescent, but
late enough so that the services needed by the patches tools (e.g. local
file systems) and the patches themselves are present.

> The question is, can the smf milestone be changed mid-milestone?
>
> My test shows that it can. How about:
>
> 1. Create patch-test-service, on which single-user depends. This will
> "svcadm milestone patch-install-milestone" if a patch needs to be
> installed. This service is always enabled.
>
> 2. Create patch-install-milestone, which depends on patch-install-service
> below.
>
> 3. Create patch-install service, which depends on:
> single-user
> filesystem-local
>
> This service is always enabled. It will install a patch if it is pending,
> otherwise, do nothing. If the service fails, it might need to:
>
> # svcadm milestone single-user
>
> So that a maintenance prompt will be appear on the console. This might
> not be necessary. you might get this anyway, as console-login is not
> reached.
>
> It should be ok to issue smf commands from an smf service, as long as they
> do not try to do any synchronous operations (-s).

Seems a little convoluted, but might be workable.

patch-install-service might need to

ms=`svcprop -p options/milestone svc:/system/svc/restarter:default`
svcadm milestone "$ms"

if the patches installed don't need a reboot.

> This approach is also good because an explicit boot to single user WILL NOT
> attempt to install pending patches.

That would be very nice, but are you sure? It seems like
patch-test-service would override the milestone specified at boot time,
and the system would continue up to the patch installation milestone.

> Disabling the patch-test and patch-install services will disable the
> automatic installation of pending patches on reboot.