Added Ossama

added blocks

Also the PortableInterceptor::register_orb_initializer has to be checked whether the global one is retrieved. We don't have an ORBInitializer that is ORB specific

when taking the locks out of the ACE_DLL_Handle as test (see bug 2994) then I do get two times the TAO::ORBInitializer_Registry::ORBInitializer_Registry called meaning I have two instances which is really bad

From an e-mail discussion I add with Johnny:

Hi Johnny,

I agree with your assessment.  We've been bitten by this problem.  I've actually had to disable the new ORB-local configuration code, and go back to the original global configuration behavior to allow our TAO users to make progress.

There's also this code in tao/ORBInitializer_Registry.cpp that is questionable:

namespace PortableInterceptor
{
  void
  register_orb_initializer (ORBInitializer_ptr init)
  {
    {
...
    // If not, lookup it up.
    TAO::ORBInitializer_Registry_Adapter *orbinitializer_registry_ =
      ACE_Dynamic_Service<TAO::ORBInitializer_Registry_Adapter>::instance
      ("ORBInitializer_Registry", true); // only look in the local repo

#if !defined (TAO_AS_STATIC_LIBS)
...

Shouldn't we be looking in the global service repository?

-Ossama

increased priority and severity

makde this a blocker, reassign to Iliyan

(http://deuce.doc.wustl.edu/bugzilla/﷒0﷓ - In reply
to comment ##4 and 5)

> when taking the locks out of the ACE_DLL_Handle as test (see bug 2994) then I
> do get two times the TAO::ORBInitializer_Registry::ORBInitializer_Registry
> called meaning I have two instances which is really bad

I am sorry for missing to point this out earlier ... 

The argument about the two instances of the registry, based on the behavior of Two_DLL_ORB test is incorrect since the test is *designed* to deliberately cause precisely that situation! Please, bear with me while I try to explain why I think this is still fully compliant with the CORBA spec.

The test exercises TAO's proprietary feature allowing for multiple
independently configured ORBs to coexist in a single process (i.e. when
loaded as services). To ensure that the two ORBs loaded from two DLLs are completely independent, the service configuration *must* create the two instances.

Note that while the two DLLs merely happen to be loaded in the same process space, they are not the same "application".  Logically and architecturally they represent two distinct "applications".  At the very least because neither is written with the expectation that it can be somehow affected by a configuration used in some other application.

Note that the CORBA specification (http://www.omg.org/docs/formal/04-03-12.pdf) provides no definition of the term "application".  Yet it defines the behavior of the ORBInitializer_Registry based on it. For example, see 21.7.3:

...
.. that implements Interceptors will provide an instance of
ORBInitializer. To use a service, an application would first call
register_orb_initializer, passing in the service's ORBInitializer.
After this is complete, the application would make an instantiating
ORB_init call.
...

The specification does not provide definition of the term "process" either. In the popular vernacular, an "application" is not equivalent to a "process".  In fact, an application is often a composition of >1 process. Therefore, it seems that CORBA implementations (and any other) are "free" to decide whether the application:process mapping is 1:1, 1:N, or M:N.  The approach we (OCI) have taken in our changes to the service config is based on the M:N mapping, where multiple applications can map to a single process (address space). 

Finally, as a counter-example in support of my thesis - a question. 

Consider a long-running server "shell" process, which can dynamically load and execute DLL-based services.  One of these DLLs/services is using IONA and the other - TAO. Given that both are (considered) CORBA compliant, would you expect that just because they are in the same process space, the same set of ORB initializers must be executed in each of them?

Is a context in which the two independent services are using the same CORBA implementation significantly different?


--Iliyan

I think the new ORB-local configuration feature in TAO made TAO overly aggressive with respect to which resources it considers ORB-specific.

My understanding was that the ORB-local configuration was originally intended to allow ORBs to have ORB-specific resource/strategy factory configurations, instead of sharing the first ORB's resource/strategy factory configuration.  Had the ORBInitializer_Registry not been split off into a separate this issue would have been moot since it would most likely not have been an ACE_Service_Object, and remained a singleton.  Just because an object, like the ORBInitializer_Registry is dynamically loadable doesn't mean it has to be ORB-specific.  The fact they are now ORB-specific is a change in behavior that is affecting legacy applications.  The fact that ORBInitializers are often registered by static initializers in TAO compounds the problem since it's not always clear which ORB is going to get one of those ORBInitializers.  This appears to be the problem we're running into.

The argument about IONA and TAO being in the same process space is not fair since they were never meant to share ORB resources to begin with.  Each is independent of the other so one would reasonably not expect them to share the same set of ORBInitializers.

Was there a real need to make the ORBInitializer_Registry ORB-specific instead of a singleton as it originally was?

At this point in time, we'd like to see ORB-local configuration disabled by default, but still have the means to enable the feature at run-time since we have plans to use it.  An alternative would be to leave it enabled by default, but allow the user to disable it at compile-time (e.g. through a preprocessor constant) and still enable it at run-time.

(In reply to comment #9)
> My understanding was that the ORB-local configuration was originally intended
> to allow ORBs to have ORB-specific resource/strategy factory configurations,
> instead of sharing the first ORB's resource/strategy factory configuration. 
> Had the ORBInitializer_Registry not been split off into a separate this issue
> would have been moot since it would most likely not have been an
> ACE_Service_Object, and remained a singleton.  Just because an object, like 
> the ORBInitializer_Registry is dynamically loadable doesn't mean it has to be
> ORB-specific.

Certainly, it doesn't. The reason why it became necessary to introduce a separate registry is that initializers are used to initialize/setup a vast array of different objects. For example, registering interceptors and initial references, (policy) factories, modifying ORB parameters that determine which additional services will be loaded, etc. 

Specifically, one cannot have an RT ORB and a regular one loaded in two separate DLL's. Or one with a a bidirectional policies. Or, one with security manager ...
Very soon one is in a situation where configuration from one ORB "spills over" into the others.

The ORB factories, being no longer the only gateway to loading resources dynamically, turned out to be just part of the problem of separating ORBs.

> I think the new ORB-local configuration feature in TAO made TAO overly
> aggressive with respect to which resources it considers ORB-specific.
> ...
> The fact they are now ORB-specific is a change in behavior that
> is affecting legacy applications. 

The only instance that I'm aware of, where a questionable behavior occurs is the test Johnny refers to and that one is specifically designed to with this behavior in mind.  Of course I'm not saying problems are out of the question - I just don't know the problem you are dealing with. Could you please clarify, or perhaps describe a scenario?

> The fact that ORBInitializers are often
> registered by static initializers in TAO compounds the problem since it's not
> always clear which ORB is going to get one of those ORBInitializers.  This
> appears to be the problem we're running into.

Yes, currently the solution appears to rely too much on TSS and therefore I can imagine a case where running the same code in one or two threads may lead to different results. Again, I'd be only speculating ...

> The argument about IONA and TAO being in the same process space is not fair
> since they were never meant to share ORB resources to begin with.  Each is
> independent of the other so one would reasonably not expect them to share the
> same set of ORBInitializers.

My intent was to take the argument through a series of logical corollaries and show a contradiction. reductio ad absurdum ...

Fairness, aside however would you agree (or disagree) with the point about compliance?

> Was there a real need to make the ORBInitializer_Registry ORB-specific instead
> of a singleton as it originally was?

Not sure. By the time we began the research it was already a service.

> At this point in time, we'd like to see ORB-local configuration disabled by
> default, but still have the means to enable the feature at run-time since we
> have plans to use it.  An alternative would be to leave it enabled by default,
> but allow the user to disable it at compile-time (e.g. through a preprocessor
> constant) and still enable it at run-time.

I'm working at a client site at the moment, but I'm helping to work out a solution for this specific configuration issue. I think we should perhaps move it to another ticket.

(In reply to comment #10)

Hi Iliyan,

> (In reply to comment #9)
> > My understanding was that the ORB-local configuration was originally intended
> > to allow ORBs to have ORB-specific resource/strategy factory configurations,
> > instead of sharing the first ORB's resource/strategy factory configuration. 
> > Had the ORBInitializer_Registry not been split off into a separate this issue
> > would have been moot since it would most likely not have been an
> > ACE_Service_Object, and remained a singleton.  Just because an object, like 
> > the ORBInitializer_Registry is dynamically loadable doesn't mean it has to be
> > ORB-specific.
> 
> Certainly, it doesn't. The reason why it became necessary to introduce a
> separate registry is that initializers are used to initialize/setup a vast
> array of different objects. For example, registering interceptors and initial
> references, (policy) factories, modifying ORB parameters that determine which
> additional services will be loaded, etc. 
> 
> Specifically, one cannot have an RT ORB and a regular one loaded in two
> separate DLL's. Or one with a a bidirectional policies. Or, one with security
> manager ...
> Very soon one is in a situation where configuration from one ORB "spills over"
> into the others.

I don't follow.

> The ORB factories, being no longer the only gateway to loading resources
> dynamically, turned out to be just part of the problem of separating ORBs.
> 
> > I think the new ORB-local configuration feature in TAO made TAO overly
> > aggressive with respect to which resources it considers ORB-specific.
> > ...
> > The fact they are now ORB-specific is a change in behavior that
> > is affecting legacy applications. 
> 
> The only instance that I'm aware of, where a questionable behavior occurs is
> the test Johnny refers to and that one is specifically designed to with this
> behavior in mind.  Of course I'm not saying problems are out of the question -
> I just don't know the problem you are dealing with. Could you please clarify,
> or perhaps describe a scenario?

I haven't had time to develop a simple test case yet, but we're seeing a problem where the TAO_Messaging policy factory is not registered with all ORBs.  Recall that the TAO_Messaging ORBInitializer is registered by a static initializer.  Two ORBs are created, both of which attempt to create a connection timeout policy through the standard ORB interface.  Since the corresponding policy factory does not exist, a CORBA::PolicyError exception is thrown.  Once I forced the ORB to always use the global repository, the code worked as expected.

Unfortunately, doing so introduced another side effect where the second ORB would end up creating double the amount of acceptors, apparently because the "-ORBProtocolFactory FooFactory" option was processed twice for the same Resource_Factory.  In this case, it sounds like each ORB_Core is processing the Resource_Factory arguments twice.  I don't think is a problem with the ORB-specific configuration code, however.  The hack I used didn't prevent ACE_Service_Objects from being reinitialized, which isn't a problem with DOC HEAD, as far as I know.

> > The fact that ORBInitializers are often
> > registered by static initializers in TAO compounds the problem since it's not
> > always clear which ORB is going to get one of those ORBInitializers.  This
> > appears to be the problem we're running into.
> 
> Yes, currently the solution appears to rely too much on TSS and therefore I can
> imagine a case where running the same code in one or two threads may lead to
> different results. Again, I'd be only speculating ...

Same here.  I'm not yet sure how to reproduce the problem our early adopters are  running into.

> > The argument about IONA and TAO being in the same process space is not fair
> > since they were never meant to share ORB resources to begin with.  Each is
> > independent of the other so one would reasonably not expect them to share the
> > same set of ORBInitializers.
> 
> My intent was to take the argument through a series of logical corollaries and
> show a contradiction. reductio ad absurdum ...

After looking that Latin up ... okay, I see what you were trying to do but my head now hurts. ;)

> Fairness, aside however would you agree (or disagree) with the point about
> compliance?

I'm still leaning toward the disagreeing side.  Quoting a portion of the paragraph following the one you quoted:

...
An ORBInitializer registered at a given point in time will be called by all instantiating ORB_init calls that occur after that point in time.
...

Note that it says "all instantiating ORB_init calls".  With the current behavior, it is possible that an ORBInitializer will not be registered with all TAO_ORBInitializer_Registrys, meaning it may not be called by "*all* instantiating ORB_init calls".  Right?

> > Was there a real need to make the ORBInitializer_Registry ORB-specific instead
> > of a singleton as it originally was?
> 
> Not sure. By the time we began the research it was already a service.

But it was still a singleton, no?

> > At this point in time, we'd like to see ORB-local configuration disabled by
> > default, but still have the means to enable the feature at run-time since we
> > have plans to use it.  An alternative would be to leave it enabled by default,
> > but allow the user to disable it at compile-time (e.g. through a preprocessor
> > constant) and still enable it at run-time.
> 
> I'm working at a client site at the moment, but I'm helping to work out a
> solution for this specific configuration issue. I think we should perhaps move
> it to another ticket.

Sure.  That's fine.

Thanks alot for your help Iliyan!

(In reply to comment #11)

I just noticed I didn't complete what i wanted to say with respect to my "I don't follow" comment.

> > Certainly, it doesn't. The reason why it became necessary to introduce a
> > separate registry is that initializers are used to initialize/setup a vast
> > array of different objects. For example, registering interceptors and initial
> > references, (policy) factories,

One always had the ability to selectively decide with which ORB a given ORBInitializer related resource is registered based on the ORB ID.  What does an ORB-specific ORBInitializer_Registry buy you in this case other than a different way to achieve a similar, albeit not necessarily the same, affect?  Admittedly, I don't understand all of the ORB-specific Service_Config nuances you had to deal with.

> > modifying ORB parameters that determine which
> > additional services will be loaded, etc. 

Can you please elaborate?

> > Specifically, one cannot have an RT ORB and a regular one loaded in two
> > separate DLL's. Or one with a a bidirectional policies. Or, one with security
> > manager ...
> > Very soon one is in a situation where configuration from one ORB "spills over"
> > into the others.

Perhaps the right thing to do is drop the static initializers in TAO's subset libraries, and instead have users explicitly specify which "feature" they want.  They already have to include the appropriate static initializer header.  Why not ask them place the appropriate feature enabling code/macro before the ORB_init() calls that matter to them.  We would then not have to worry about order of initialization problems, etc.  I suppose this is over simplifying things in the case of ORBInitializers that are brought in through the ACE_Service_Configurator, for example, at run time.  Anyway, just a thought.

Thanks!

> > (In reply to comment ##9,10,11)
> > > My understanding was that the ORB-local configuration was originally intended
> > > to allow ORBs to have ORB-specific resource/strategy factory configurations,
> > > instead of sharing the first ORB's resource/strategy factory configuration. 
> > > Had the ORBInitializer_Registry not been split off into a separate this issue
> > > would have been moot since it would most likely not have been an
> > > ACE_Service_Object, and remained a singleton.  Just because an object, like 
> > > the ORBInitializer_Registry is dynamically loadable doesn't mean it has to be
> > > ORB-specific.
> > 
> > Certainly, it doesn't. The reason why it became necessary to introduce a
> > separate registry is that initializers are used to initialize/setup a vast
> > array of different objects. For example, registering interceptors and initial
> > references, (policy) factories, modifying ORB parameters that determine which
> > additional services will be loaded, etc. 
> > 
> > Specifically, one cannot have an RT ORB and a regular one loaded in two
> > separate DLL's. Or one with a a bidirectional policies. Or, one with security
> > manager ...
> > Very soon one is in a situation where configuration from one ORB "spills over"
> > into the others.

> I don't follow.

It's my fault, not yours :) ... I dug a bit more and here's the main reason why, in some especially interesting cases we need >1 ORB initializer registry.  

Consider the particular use case, illustrated by the Two_ORB_DLL test where we have two DLL's - D1 and D2. Suppose we have a single registry, and the D1 loads first. When D2 loads and creates its ORB, all initializers from D1 will register with D2. 

Now, the most common initializer usage is to create objects and associate them with the ORB (policy factories, initial references, policies). Thus D2 becomes owner of objects, whose code is mapped in D1's code segment! The stage is set for the tragedy, which inevitably unfolds when D1 is unloaded *before* D2.

Note that the service objects themselves are not the problem (well _now_, after our changes they aren't). Our changes enforced the correct order of finalization and associated the services and tied their lifespan with the corresponding DLL handle (which is reference counted). Thus service objects are correctly finalized, but any policies they have created are not. This causes very weird and unpredictable SEGV's while D2 is performing trivial operations. The most elegant way to avoid this is to separate the initializer registries.

I'll address the potential failure scenario you described separately - I need to research it a bit more.

--Iliyan

Set target

Created attachment 846 [details]
presumed fix for this bug

This patch makes the creation of a local configuration context explicit. By default all ORBs use the global configuration context.

lowering priority

Is this patch applied to the repo?

(In reply to comment #15)
> Created an attachment (id=846) [details]
> presumed fix for this bug
> 
> This patch makes the creation of a local configuration context explicit. By
> default all ORBs use the global configuration context.
> 

(In reply to comment #17)
> Is this patch applied to the repo?

It would appear the it has been. Note however, that not all aspects of the functionality are there yet.  The test suite should be ran particularly in the case of forced global configuration, which is still in the works.

added blocks