Solaris 8 threads attributes
Solaris 8 adds several new features to its already powerful set of thread APIs. Key features include:
- Alternate one-level libthread library
- Priority inheritance for user threads
- Priority ceilings for mutex locks
- Robust mutex locks
Let's have a look.
Alternate one-level threads library
Solaris implements a two-level threads architecture (see August 1998's column as well as subsequent columns covering the process model).
The two-level model is implemented by abstracting the user thread as something separate and distinct from the kernel thread and lightweight process (LWP). Hopefully, our previous discussion on bound and unbound threads clarified the distinction between the two types of threads. Experience has shown that a number of threaded applications can benefit from using bound over unbound threads. However, changing the source or recompiling the application may not be possible or practical. For such situations, the alternate libthread library was shipped with Solaris 8.
By linking a multithreaded application to the alternate library, all threads are created as bound threads. That is, every thread is created with an LWP and linked to the LWP for the thread's lifetime. The alternate thread's library is maintained in the
/usr/lib/lwp directory (32-bit) and
/usr/lib/lwp/64 for 64-bit programs. You can link to the alternate libthread either through a compilation flag or by setting the runtime linker's
Here's a simple compile of a thread's program using POSIX threads, followed by a run of the
ldd(1) command on the resulting binary (
ldd(1) lists the dependencies of dynamically-linked objects).
sunsys> cc -o ptdemo ptdemo.c -lpthread sunsys> ldd ptdemo libpthread.so.1 => /usr/lib/libpthread.so.1 libthread.so.1 => /usr/lib/libthread.so.1 libc.so.1 => /usr/lib/libc.so.1 libdl.so.1 => /usr/lib/libdl.so.1 /usr/platform/SUNW,Ultra-60/lib/libc_psr.so.1
As you can see from the
libthread.so.1 are resolved from the
/usr/lib directory. Note that the sample program, ptdemo, uses POSIX threads and doesn't have a direct dependency on
libthread.so.1. The dependency on
libthread.so.1 is in
libpthread.so.1, the POSIX threads library. That is an important point in understanding the procedure for linking to the alternate thread's library for programs that use POSIX threads.
Now we recompile, using the
-R flag, which specifies a path for the runtime linker to search.
sunsys> cc -o ptdemo ptdemo.c -lpthread -lthread -R/usr/lib/lwp sunsys> ldd ptdemo libpthread.so.1 => /usr/lib/libpthread.so.1 libthread.so.1 => /usr/lib/lwp/libthread.so.1 libc.so.1 => /usr/lib/libc.so.1 libdl.so.1 => /usr/lib/libdl.so.1 /usr/platform/SUNW,Ultra-60/lib/libc_psr.so.1 sunsys>
Looking at the
ldd(1) output, we see that
libthread.so.1 is now resolved from
/usr/lib/lwp, which is the alternate thread's library. Note the addition of
-lthread preceding the
-R flag in the compile line. That is necessary for POSIX thread programs because of the dependency we mentioned earlier. An alternate
libpthread.so.1 is not necessary, as the internal thread create function used by both POSIX and Solaris threads is part of the
libthread.so.1. The POSIX library,
libpthread.so.1, exists to set up POSIX-related features and attributes and support POSIX semantics.
If you're using Solaris threads, simply specify the runtime linker path following the libthread inclusion. You would use:
sunsys> cc -o soltdemo soltdemo.c -lthread -R/usr/lib/lwp
If source code is not available, the runtime linker's
LD_LIBRARY_PATH variable can be set prior to running the threaded program.
sunsys> LD_LIBRARY_PATH=/usr/lib/lwp;export LD_LIBRARY_PATH sunsys> ptdemo
/usr/lib/lwp, the runtime linker will search the specified directory to resolve dependencies before looking in the standard location (
For 64-bit applications, use the
/usr/lib/lwp/64 directory path in place of
/usr/lib/lwp in the above examples.
We discussed the problem of priority inversion and the solution, priority inheritance, as it's implemented in the kernel in August 1999's column. Priority inversion describes a scenario in which a higher-priority thread is prevented from running because a lower-priority thread is holding a resource the higher-priority thread requires to execute. For example, thread T1 is a priority 10 thread and grabs mutex lock M1. T1 is now the owner of M1. T2, a realtime thread at priority 60, is running, and it attempts to acquire M1. M1 is not available because it's currently owned by T1. Here we have a priority inversion scenario: higher-priority thread T2 is prevented from executing because of a lower-priority thread, T1.
The solution implemented in the kernel for priority inversion is priority inheritance. The lower-priority thread that's holding a resource needed by a higher-priority thread inherits the waiting thread's priority for a short period. In our example, T1 would inherit T2's priority and begin running as a priority 60 thread. With that increased priority, T1 will execute sooner, allowing T1 to complete what it's doing and release the lock (M1) that T2 needs. Once the lock is released, T1 will go back to its previous priority.
That works fine in the kernel, but multithreaded applications can run into a priority inversion scenario. They're difficult to detect, and up until Solaris 8, an elegant solution was lacking. With Solaris 8, mutex locks have two new attributes that address the priority inversion problem.
Just as POSIX threads have attributes associated with them, user-created mutex locks can also have specific attributes. The initializing and setting of the attributes, from a programming point of view, is very similar to what's involved in setting POSIX thread attributes. In POSIX, mutex locks are created (initialized) using the
pthread_mutex_init(3T) interface. That interface takes two arguments: a pointer to a mutex lock (which must be declared in the code) and a pointer to a mutex attributes structure. That second argument is optional. A
NULL value will result in default attributes for the mutex.
Mutex lock attributes require initializing a mutex attribute object using
pthread_mutexattr_init(3T), which takes a pointer to a mutex attribute structure (
pthread_mutexattr_t *) as an argument. The possible attributes a mutex lock can have are shown below. For completeness, we're showing all attributes, not all of which are applicable to priority inheritance.
- Scope. The scope is the visibility of the mutex, intraprocess, or interprocess.
PTHREAD_PROCESS_PRIVATEmeans the mutex is visible only to threads in the same process.
PTHREAD_PROCESS_SHAREDsets the mutex to systemwide visibility. Reference the
- Type. POSIX specifies several types of mutex behavior related to error detection and support for recursive mutexes. The default mutex type,
PTHREAD_MUTEX_DEFAULT, does not provide for error detection or defined behavior if it's recursively locked.
PTHREAD_MUTEX_ERRORCHECKtypes return errors when a recursive lock attempt is made.
PTHREAD_MUTEX_RECURSIVEtypes provide recursive mutes behavior. That is, a thread can relock a mutex that's already owned (the lock is held). Only
PTHREAD_PROCESS_PRIVATEmutexes can be
PTHREAD_MUTEX_RECURSIVE. The final mutex type is
PTHREAD_MUTEX_NORMAL, which provides essentially the same behavior as
PTHREAD_MUTEX_DEFAULT. Reference the
pthread_mutexattr_settype(3T)man page for more details.
- Protocol. Possible protocol attributes are
PHTHREAD_PRIO_INHERITprovides the priority inheritance behavior described in the previous paragraphs. With the
PTHREAD_PRIO_NONE(default) attribute, a thread's priority is not affected based on mutex ownership.
PTHREAD_PRIO_PROTECTprovides an alternate means of protecting against the priority inversion problem by allowing a priority ceiling to be specified as a mutex attribute, such that a thread acquiring the mutex will have its priority set to the ceiling value if it's greater than the thread's priority. Reference the
- Priority ceiling. A priority ceiling is the minimum priority a thread will run at while holding the mutex. For threads that have the
PTHREAD_PRIO_PROTECTprotocol attribute, a thread will have its priority set to the mutex's ceiling value when it acquires the lock if the thread's current priority is less than the ceiling value. That is an alternative method. Reference
- Robust. Solaris 8 includes support for robust mutex locks. Robust mutexes allow the programmer to have some level of protection against the death or premature termination of a thread that's holding a mutex lock. If a thread terminates while holding a lock, a subsequent
pthread_mutex_lock()call by another thread to get the mutex will get an
EOWNERDEADreturn value if the mutex has the
PTHREAD_MUTEX_STALLED_NProbust attribute set. Reference
Priority inheritance is achieved by setting a thread's protocol attribute to
PTHREAD_PRIO_INHERIT. That needs to be done using the appropriate API prior to initializing the mutex with
pthread_mutex_init(). As with all attributes, once it's set, it cannot be changed without reinitializing the mutex.
If a thread issues a
pthread_mutex_lock() call to acquire a
PRIO_INHERIT mutex that is being held by another thread, the mutex owner will inherit the waiting thread's priority if the waiting thread's priority is better (higher) than the lock holder's priority. The goal is to get the lock holder running so that it can complete what it's doing in the critical code section (the critical section is the code that lives between the
pthread_mutex_unlock() calls) and release the lock.
PRIO_INHERIT locks are implemented in the kernel, which is required to facilitate the actual priority inheritance for the holder. The kernel's turnstile/priority inheritance infrastructure is leveraged.
The other solution to priority inversion,
PRIO_PROTECT locks, implements a priority ceiling attribute for the lock, so that the lock owner has its priority elevated to the ceiling value of the lock if the thread's current priority is less than the ceiling. That is, if a thread has a priority of 50 and the lock's ceiling value is 60, the thread will have its priority promoted to 60 for as long as it owns the lock, after which its priority is reset to 50.
Note that priority inheritance and priority ceilings are available for POSIX mutexes, not for Solaris mutexes. Also note that priority inheritance should only be used with bound threads. If an unbound thread attempts to acquire a
PRIO_INHERIT lock, the library code will temporarily bind it to the LWP anyway.
Robust mutex locks
A mutex lock can set a robust attribute soh that specific behavior can be obtained if a thread holding a mutex lock terminates (exits or dies). It requires all lock acquisition attempts (calls to
pthread_mutex_lock(3T)) to check the return value and errno (error number) set by the lock call to determine if the previous lock owner died while holding the lock and if the lock is in a recoverable state.
The possible attributes are
PTHREAD_MUTEX_ROBUST_NP. The default attribute (
STALLED) means that if the lock holder terminates while holding the lock, subsequent get lock calls will block, ultimately causing the application to hang.
If a mutex has the
ROBOST attribute set and a thread terminates while holding the lock, the next thread that issues a
pthread_mutex_lock() call on the lock will get the lock but also a return value of
EOWNERDEAD. It's then up to the application code to maintain the consistency of the data protected by the lock. If the application code can, it must call
pthread_mutex_consistent_np(3T) for the mutex, then release the lock with
If the data is not made consistent or if another thread attempts to get the lock before the
pthread_mutex_consistent_np(3T) call has completed, the lock acquire call will return with
ENOTRECOVERABLE, which tells the application that the caller cannot get the lock and should not make further attempts.
ENOTRECOVERABLE scenario, the application can do some level of recovery without exiting, provided the data can be brought to a consistent state. The lock needs to be cleared with
pthread_mutex_destroy(3T) and reinitialized with
The issue of a thread terminating while holding a lock is not trivial. Without robust locks, such situations typically result in application hangs as threads pile up, waiting to get a lock that will never be freed because the holder (owner) no longer exists. With robust locks, we can now detect such a condition and implement recovery code in the application. It comes down to the application's ability to restore the state of the data that was being protected by the lock and use the proper interfaces.
Robust locks are implemented in Solaris locks (
mutex_init(3T)) as well as POSIX mutex locks. Using POSIX, you can have robust locks that also have the priority inheritance attribute.
That's a wrap for this month. I wanted to get into the low-level implementation of robust locks and priority inheritance locks. Due to the current length of the column, I'll defer that discussion for another time (assuming there's general interest based on reader feedback). Suffice it to say that, like PI locks, robust locks are handled by the kernel, not by the thread's library.