TAO VERSION: 2.2.4
ACE VERSION: 6.2.4

HOST MACHINE and OPERATING SYSTEM:

Sparc M3000 - Solaris 11.1


COMPILER NAME AND VERSION (AND PATCHLEVEL):

Solaris Studio 12.3

CC -V
CC: Sun C++ 5.12 SunOS_sparc Patch 148506-16 2013/12/03

THE $ACE_ROOT/ace/config.h FILE

#define ACE_HAS_WCHAR
#define ACE_USES_WCHAR
#include "ace/config-sunos5.11.h"

THE $ACE_ROOT/include/makeinclude/platform_macros.GNU FILE

buildbits=64
include $(ACE_ROOT)/include/makeinclude/platform_sunos5_sunc++.GNU
CCFLAGS+=-features=zla -z aslr=disable

The features=zla enables zero length arrays, the -z aslr=disable directs the linker to disable address layout randomization.

No other changes are made to the code.

AREA/CLASS/EXAMPLE AFFECTED:

TAO_Dynamic_Hash_OpTable constructor in TAO_PortableServer

SYNOPSIS:

Application linking libTAO_PortableServer.so.2.2.4 receives SIGBUS when the library's init section (static constructors) is run.

DESCRIPTION:

After building ACE and TAO a simple program is compiled and linked against libTAO_PortableServer.so.

pst.c

#include "stdio.h"
int main() {
        printf("Portable Server!\n");
        return 0;
}

CC -m64 -c -o pst.o pst.c
CC -m64 -o pst pst.o -L$ACE_ROOT/lib -lTAO_PortableServer
./pst
Bus Error (core dumped)

Examining this with dbx (Solaris Studio debugger)

dbx pst
Reading pst
Reading ld.so.1
Reading libTAO_PortableServer.so.2.2.4
Reading libCstd.so.1
Reading libCrun.so.1
Reading libm.so.2
Reading libc.so.1
Reading libTAO_AnyTypeCode.so.2.2.4
Reading libTAO.so.2.2.4
Reading libACE.so.6.2.4
Reading libsendfile.so.1
Reading libkstat.so.1
Reading libsocket.so.1
Reading librt.so.1
Reading libaio.so.1
Reading libdl.so.1
Reading libnsl.so.1
Reading libgen.so.1
Reading libadm.so.1
Reading libsctp.so.1
Reading libmd.so.1
Reading libsoftcrypto.so.1
Reading libcryptoutil.so.1
Reading libelf.so.1
Reading libmp.so.2
(dbx) run
Running: pst
(process id 12598)
signal BUS (invalid address alignment) in TAO::Operation_Skeletons::Operation_Skeletons at line 17 in file "Operation_Table.cpp"
   17     , direct_skel_ptr (0)
(dbx) where
=>[1] TAO::Operation_Skeletons::Operation_Skeletons(this = 0xffffffff47832aa4), line 17 in "Operation_Table.cpp"
  [2] ACE_Hash_Map_Entry<const char*,TAO::Operation_Skeletons>::ACE_Hash_Map_Entry(this = 0xffffffff47832a9c, next = 0xffffffff47832a9c, prev = 0xffffffff47832a9c), line 34 in "Hash_Map_Manager_T.cpp"
  [3] ACE_Hash_Map_Manager_Ex<const char*,TAO::Operation_Skeletons,ACE_Hash<const char*>,ACE_Equal_To<const char*>,ACE_Null_Mutex>::create_buckets(this = 0xffffffff4782fdf0, size = 16U), line 122 in "Hash_Map_Manager_T.cpp"
  [4] ACE_Hash_Map_Manager_Ex<const char*,TAO::Operation_Skeletons,ACE_Hash<const char*>,ACE_Equal_To<const char*>,ACE_Null_Mutex>::open(this = 0xffffffff4782fdf0, size = 16U, table_alloc = 0xffffffff4782fdc8, entry_alloc = 0xffffffff4782fdc8), line 155 in "Hash_Map_Manager_T.cpp"
  [5] ACE_Hash_Map_Manager_Ex<const char*,TAO::Operation_Skeletons,ACE_Hash<const char*>,ACE_Equal_To<const char*>,ACE_Null_Mutex>::ACE_Hash_Map_Manager_Ex(this = 0xffffffff4782fdf0, size = 16U, table_alloc = 0xffffffff4782fdc8, entry_alloc = (nil)), line 19 in "Hash_Map_Manager_T.inl"
  [6] TAO_Dynamic_Hash_OpTable::TAO_Dynamic_Hash_OpTable(this = 0xffffffff4782fde8, db = 0xffffffff47832888, dbsize = 8U, hashtblsize = 16U, alloc = 0xffffffff4782fdc8), line 36 in "Operation_Table_Dynamic_Hash.cpp"
  [7] __SLIP.INIT_C(), line 73 in "PolicyS.cpp"
  [8] __STATIC_CONSTRUCTOR(), line 73 in "PolicyS.cpp"
  [9] 0xffffffff476c20f8(0x0, 0x0, 0x0, 0xffffffff7f747250, 0x3, 0xffffffff7d102a40), at 0xffffffff476c20f8
  [10] call_init(0xffffffff7f7426c0, 0xffffffff476c1e60, 0x80, 0xffdfffff, 0x400000, 0xffffffff7d201e20), at 0xffffffff7f61da3c
  [11] setup(0xffffffff7d201e20, 0xffffffff7f400c98, 0xffffffff7f74226c, 0x0, 0x1000, 0x1000), at 0xffffffff7f61ce70
  [12] _setup(0x6ffffff9, 0xffffffffffffffff, 0xffffffff7ffffb88, 0x100000040, 0xffffffff7f742d80, 0xb00), at 0xffffffff7f62f208
  [13] _rt_boot(0x0, 0x0, 0x0, 0x0, 0x0, 0x0), at 0xffffffff7f60ec6c
(dbx) up 6
Current function is __SLIP.INIT_C
   73   static TAO_Dynamic_Hash_OpTable tao_CORBA_Policy_optable (
(dbx) print tao_CORBA_Policy_optable
tao_CORBA_Policy_optable = {
    hash_ = {
        table_allocator_ = 0xffffffff4782fdc8
        entry_allocator_ = 0xffffffff4782fdc8
        lock_            = {
            lock_ = 0
        }
        hash_key_        = {
        }
        compare_keys_    = {
        }
        table_           = 0xffffffff47832a9c
        total_size_      = 16U
        cur_size_        = 0
    }
}

We can see the value of table_ is the same the this, next, and prev arguments passed to the ACE_Hash_Map_Entry constructor in frame 2

It appears we are using placement new to construct objects in the table. The problem is that table is not 8 byte aligned and so we pass a misaligned pointer as the this pointer for the object being constructed.

If we look at the generated source for PolicyS.cpp we find:

static const ::CORBA::Long _tao_CORBA_Policy_optable_size = sizeof (ACE_Hash_Map_Entry<const char *, TAO::Operation_Skeletons>) * (24);
static char _tao_CORBA_Policy_optable_pool [_tao_CORBA_Policy_optable_size];
static ACE_Static_Allocator_Base _tao_CORBA_Policy_allocator (_tao_CORBA_Policy_optable_pool, _tao_CORBA_Policy_optable_size);
static TAO_Dynamic_Hash_OpTable tao_CORBA_Policy_optable (
    CORBA_Policy_operations,
    8,
    16,
    &_tao_CORBA_Policy_allocator
  );

So a static char[] is used to allocate the table.

The problem is that static char[] is allocated on a 4 byte alignment, even when compiled 64bit.

It is very likely that this problem is specific to Sparc/Solaris 64bit code.

SAMPLE FIX/WORKAROUND:

There are 3 simple fixes for this problem.

The first is to force the alignment of _tao_CORBA_Policy_optable_pool to 8 bytes by preceding it with an align pragma

#pragma align 8 (_tao_CORBA_Policy_optable_pool)

Rebuilding the library and running the test application in dbx gives the following.

This time we need to set a break-point otherwise the process will run to completion.

(dbx) file Operation_Table.cpp
(dbx) stop at 17
(2) stop at "Operation_Table.cpp":17
(dbx) run
Running: pst
(process id 19540)
stopped in TAO::Operation_Skeletons::Operation_Skeletons at line 17 in file "Operation_Table.cpp"
   17     , direct_skel_ptr (0)
(dbx) where
=>[1] TAO::Operation_Skeletons::Operation_Skeletons(this = 0xffffffff44a32aa8), line 17 in "Operation_Table.cpp"
  [2] ACE_Hash_Map_Entry<const char*,TAO::Operation_Skeletons>::ACE_Hash_Map_Entry(this = 0xffffffff44a32aa0, next = 0xffffffff44a32aa0, prev = 0xffffffff44a32aa0), line 34 in "Hash_Map_Manager_T.cpp"
  [3] ACE_Hash_Map_Manager_Ex<const char*,TAO::Operation_Skeletons,ACE_Hash<const char*>,ACE_Equal_To<const char*>,ACE_Null_Mutex>::create_buckets(this = 0xffffffff44a2fdf0, size = 16U), line 122 in "Hash_Map_Manager_T.cpp"
  [4] ACE_Hash_Map_Manager_Ex<const char*,TAO::Operation_Skeletons,ACE_Hash<const char*>,ACE_Equal_To<const char*>,ACE_Null_Mutex>::open(this = 0xffffffff44a2fdf0, size = 16U, table_alloc = 0xffffffff44a2fdc8, entry_alloc = 0xffffffff44a2fdc8), line 155 in "Hash_Map_Manager_T.cpp"
  [5] ACE_Hash_Map_Manager_Ex<const char*,TAO::Operation_Skeletons,ACE_Hash<const char*>,ACE_Equal_To<const char*>,ACE_Null_Mutex>::ACE_Hash_Map_Manager_Ex(this = 0xffffffff44a2fdf0, size = 16U, table_alloc = 0xffffffff44a2fdc8, entry_alloc = (nil)), line 19 in "Hash_Map_Manager_T.inl"
  [6] TAO_Dynamic_Hash_OpTable::TAO_Dynamic_Hash_OpTable(this = 0xffffffff44a2fde8, db = 0xffffffff44a32888, dbsize = 8U, hashtblsize = 16U, alloc = 0xffffffff44a2fdc8), line 36 in "Operation_Table_Dynamic_Hash.cpp"
  [7] __SLIP.INIT_C(), line 74 in "PolicyS.cpp"
  [8] __STATIC_CONSTRUCTOR(), line 74 in "PolicyS.cpp"
  [9] 0xffffffff448c20f8(0x0, 0x0, 0x0, 0xffffffff7f747250, 0x3, 0xffffffff7d102a40), at 0xffffffff448c20f8
  [10] call_init(0xffffffff7f7426c0, 0xffffffff448c1e60, 0x80, 0xffdfffff, 0x400000, 0xffffffff7d201e20), at 0xffffffff7f61da3c
  [11] setup(0xffffffff7d201e20, 0xffffffff7f400c98, 0xffffffff7f74226c, 0x0, 0x1000, 0x1000), at 0xffffffff7f61ce70
  [12] _setup(0x6ffffff9, 0xffffffffffffffff, 0xffffffff7ffffb88, 0x100000040, 0xffffffff7f742d80, 0xb00), at 0xffffffff7f62f208
  [13] _rt_boot(0x0, 0x0, 0x0, 0x0, 0x0, 0x0), at 0xffffffff7f60ec6c
(dbx) up 6
Current function is __SLIP.INIT_C
   74   static TAO_Dynamic_Hash_OpTable tao_CORBA_Policy_optable (
(dbx) print tao_CORBA_Policy_optable
tao_CORBA_Policy_optable = {
    hash_ = {
        table_allocator_ = 0xffffffff44a2fdc8
        entry_allocator_ = 0xffffffff44a2fdc8
        lock_            = {
            lock_ = 0
        }
        hash_key_        = {
        }
        compare_keys_    = {
        }
        table_           = 0xffffffff44a32aa0
        total_size_      = 16U
        cur_size_        = 0
    }
}
(dbx) quit

and we see that the table is now 8 byte aligned.

The second fix requires the latest Studio compiler patches and Solaris linker patches and have the compiler generate/include code to handle misaligned memory access.

$ CC -m64 -c -o pst.o pst.c
$ CC -m64 -o pst pst.o -L$ACE_ROOT/lib -lTAO_PortableServer
$ ./pst
Bus Error (core dumped)
$ CC -m64 -c -o pst.o pst.c -xmemalign=8i
$ CC -m64 -o pst pst.o -L$ACE_ROOT/lib -lTAO_PortableServer -xmemalign=8i
$ ./pst
Portable Server!
$

Without the latest compiler and linker patches the -xmemalign option will not work - because the libraries init sections are run before the code to handle the SIGBUS has been registered. The compiler bug report indicates patches 148906-09, 148921-09, 148925-09 are required for Solaris 10. For Solaris 11 updating the OS and compilers from the support repository is sufficient.

There is a downside to the use of -xmemalign, misaligned access comes at performance cost - which is cpu/system dependent.

The third fix is to change the type of _tao_CORBA_Policy_optable_pool from char[] to one that has 8 byte alignment, such as void*[], adjust the size and cast it as necessary when passing it to the ACE_Static_Allocator_Base constructor.