aboutsummaryrefslogtreecommitdiff
path: root/nptl/pthread_cond_timedwait.c
AgeCommit message (Collapse)AuthorFilesLines
2016-12-31New condvar implementation that provides stronger ordering guarantees.Torvald Riegel1-268/+0
This is a new implementation for condition variables, required after http://austingroupbugs.net/view.php?id=609 to fix bug 13165. In essence, we need to be stricter in which waiters a signal or broadcast is required to wake up; this couldn't be solved using the old algorithm. ISO C++ made a similar clarification, so this also fixes a bug in current libstdc++, for example. We can't use the old algorithm anymore because futexes do not guarantee to wake in FIFO order. Thus, when we wake, we can't simply let any waiter grab a signal, but we need to ensure that one of the waiters happening before the signal is woken up. This is something the previous algorithm violated (see bug 13165). There's another issue specific to condvars: ABA issues on the underlying futexes. Unlike mutexes that have just three states, or semaphores that have no tokens or a limited number of them, the state of a condvar is the *order* of the waiters. A waiter on a semaphore can grab a token whenever one is available; a condvar waiter must only consume a signal if it is eligible to do so as determined by the relative order of the waiter and the signal. Therefore, this new algorithm maintains two groups of waiters: Those eligible to consume signals (G1), and those that have to wait until previous waiters have consumed signals (G2). Once G1 is empty, G2 becomes the new G1. 64b counters are used to avoid ABA issues. This condvar doesn't yet use a requeue optimization (ie, on a broadcast, waking just one thread and requeueing all others on the futex of the mutex supplied by the program). I don't think doing the requeue is necessarily the right approach (but I haven't done real measurements yet): * If a program expects to wake many threads at the same time and make that scalable, a condvar isn't great anyway because of how it requires waiters to operate mutually exclusive (due to the mutex usage). Thus, a thundering herd problem is a scalability problem with or without the optimization. Using something like a semaphore might be more appropriate in such a case. * The scalability problem is actually at the mutex side; the condvar could help (and it tries to with the requeue optimization), but it should be the mutex who decides how that is done, and whether it is done at all. * Forcing all but one waiter into the kernel-side wait queue of the mutex prevents/avoids the use of lock elision on the mutex. Thus, it prevents the only cure against the underlying scalability problem inherent to condvars. * If condvars use short critical sections (ie, hold the mutex just to check a binary flag or such), which they should do ideally, then forcing all those waiter to proceed serially with kernel-based hand-off (ie, futex ops in the mutex' contended state, via the futex wait queues) will be less efficient than just letting a scalable mutex implementation take care of it. Our current mutex impl doesn't employ spinning at all, but if critical sections are short, spinning can be much better. * Doing the requeue stuff requires all waiters to always drive the mutex into the contended state. This leads to each waiter having to call futex_wake after lock release, even if this wouldn't be necessary. [BZ #13165] * nptl/pthread_cond_broadcast.c (__pthread_cond_broadcast): Rewrite to use new algorithm. * nptl/pthread_cond_destroy.c (__pthread_cond_destroy): Likewise. * nptl/pthread_cond_init.c (__pthread_cond_init): Likewise. * nptl/pthread_cond_signal.c (__pthread_cond_signal): Likewise. * nptl/pthread_cond_wait.c (__pthread_cond_wait): Likewise. (__pthread_cond_timedwait): Move here from pthread_cond_timedwait.c. (__condvar_confirm_wakeup, __condvar_cancel_waiting, __condvar_cleanup_waiting, __condvar_dec_grefs, __pthread_cond_wait_common): New. (__condvar_cleanup): Remove. * npt/pthread_condattr_getclock.c (pthread_condattr_getclock): Adapt. * npt/pthread_condattr_setclock.c (pthread_condattr_setclock): Likewise. * npt/pthread_condattr_getpshared.c (pthread_condattr_getpshared): Likewise. * npt/pthread_condattr_init.c (pthread_condattr_init): Likewise. * nptl/tst-cond1.c: Add comment. * nptl/tst-cond20.c (do_test): Adapt. * nptl/tst-cond22.c (do_test): Likewise. * sysdeps/aarch64/nptl/bits/pthreadtypes.h (pthread_cond_t): Adapt structure. * sysdeps/arm/nptl/bits/pthreadtypes.h (pthread_cond_t): Likewise. * sysdeps/ia64/nptl/bits/pthreadtypes.h (pthread_cond_t): Likewise. * sysdeps/m68k/nptl/bits/pthreadtypes.h (pthread_cond_t): Likewise. * sysdeps/microblaze/nptl/bits/pthreadtypes.h (pthread_cond_t): Likewise. * sysdeps/mips/nptl/bits/pthreadtypes.h (pthread_cond_t): Likewise. * sysdeps/nios2/nptl/bits/pthreadtypes.h (pthread_cond_t): Likewise. * sysdeps/s390/nptl/bits/pthreadtypes.h (pthread_cond_t): Likewise. * sysdeps/sh/nptl/bits/pthreadtypes.h (pthread_cond_t): Likewise. * sysdeps/tile/nptl/bits/pthreadtypes.h (pthread_cond_t): Likewise. * sysdeps/unix/sysv/linux/alpha/bits/pthreadtypes.h (pthread_cond_t): Likewise. * sysdeps/unix/sysv/linux/powerpc/bits/pthreadtypes.h (pthread_cond_t): Likewise. * sysdeps/x86/bits/pthreadtypes.h (pthread_cond_t): Likewise. * sysdeps/nptl/internaltypes.h (COND_NWAITERS_SHIFT): Remove. (COND_CLOCK_BITS): Adapt. * sysdeps/nptl/pthread.h (PTHREAD_COND_INITIALIZER): Adapt. * nptl/pthreadP.h (__PTHREAD_COND_CLOCK_MONOTONIC_MASK, __PTHREAD_COND_SHARED_MASK): New. * nptl/nptl-printers.py (CLOCK_IDS): Remove. (ConditionVariablePrinter, ConditionVariableAttributesPrinter): Adapt. * nptl/nptl_lock_constants.pysym: Adapt. * nptl/test-cond-printers.py: Adapt. * sysdeps/unix/sysv/linux/hppa/internaltypes.h (cond_compat_clear, cond_compat_check_and_clear): Adapt. * sysdeps/unix/sysv/linux/hppa/pthread_cond_timedwait.c: Remove file ... * sysdeps/unix/sysv/linux/hppa/pthread_cond_wait.c (__pthread_cond_timedwait): ... and move here. * nptl/DESIGN-condvar.txt: Remove file. * nptl/lowlevelcond.sym: Likewise. * nptl/pthread_cond_timedwait.c: Likewise. * sysdeps/unix/sysv/linux/i386/i486/pthread_cond_broadcast.S: Likewise. * sysdeps/unix/sysv/linux/i386/i486/pthread_cond_signal.S: Likewise. * sysdeps/unix/sysv/linux/i386/i486/pthread_cond_timedwait.S: Likewise. * sysdeps/unix/sysv/linux/i386/i486/pthread_cond_wait.S: Likewise. * sysdeps/unix/sysv/linux/i386/i586/pthread_cond_broadcast.S: Likewise. * sysdeps/unix/sysv/linux/i386/i586/pthread_cond_signal.S: Likewise. * sysdeps/unix/sysv/linux/i386/i586/pthread_cond_timedwait.S: Likewise. * sysdeps/unix/sysv/linux/i386/i586/pthread_cond_wait.S: Likewise. * sysdeps/unix/sysv/linux/i386/i686/pthread_cond_broadcast.S: Likewise. * sysdeps/unix/sysv/linux/i386/i686/pthread_cond_signal.S: Likewise. * sysdeps/unix/sysv/linux/i386/i686/pthread_cond_timedwait.S: Likewise. * sysdeps/unix/sysv/linux/i386/i686/pthread_cond_wait.S: Likewise. * sysdeps/unix/sysv/linux/x86_64/pthread_cond_broadcast.S: Likewise. * sysdeps/unix/sysv/linux/x86_64/pthread_cond_signal.S: Likewise. * sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S: Likewise. * sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S: Likewise.
2016-01-04Update copyright dates with scripts/update-copyrights.Joseph Myers1-1/+1
2015-04-20libc-vdso.h place consolidationAdhemerval Zanella1-1/+1
This patch moves the libc-vdso.h internal header from bits folder to default architecture one and also corrects the remaning includes in the files.
2015-01-06Function declaration cleanupAdhemerval Zanella1-4/+2
This patch changes the some function declaration from K&R style to default ANSI C. * nptl/pthread_cancel.c (pthread_cancel): Use ANSI prototype. * nptl/pthread_cond_timedwait.c (__pthread_cond_timedwait): Likewise. * nptl/pthread_cond_wait.c (__pthread_cond_wait): Likewise. * nptl/pthread_exit.c (__pthread_exit): Likewise. * nptl/pthread_join.c (pthread_join): Likewise. * nptl/pthread_timedjoin.c (pthread_timedjoin_np): Likewise. * sysdeps/posix/waitid.c (__waitid): Likewise. * sysdeps/unix/sysv/linux/arm/pread.c (__libc_pread): Likewise. * sysdeps/unix/sysv/linux/arm/pread64.c (__libc_pread64): Likewise. * sysdeps/unix/sysv/linux/arm/pwrite.c (__libc_pwrite): Likewise. * sysdeps/unix/sysv/linux/arm/pwrite64.c (__libc_pwrite64): Likewise. * sysdeps/unix/sysv/linux/generic/wordsize-32/pread.c (__libc_pread): Likewise. * sysdeps/unix/sysv/linux/generic/wordsize-32/pread64.c (__libc_pread64): Likewise. * sysdeps/unix/sysv/linux/generic/wordsize-32/pwrite.c (__libc_pwrite): Likewise. * sysdeps/unix/sysv/linux/generic/wordsize-32/pwrite64.c (__libc_pwrite64): Likewsie. * sysdeps/unix/sysv/linux/mips/pread.c (__libc_pread): Likewise. * sysdeps/unix/sysv/linux/mips/pread64.c (__libc_pread64): Likewise. * sysdeps/unix/sysv/linux/mips/pwrite.c (__libc_pwrite): Likewise. * sysdeps/unix/sysv/linux/mips/pwrite64.c (__libc_pwrite64): Likewise. * sysdeps/unix/sysv/linux/msgrcv.c (__libc_msgrcv): Likewise. * sysdeps/unix/sysv/linux/msgsnd.c (__libc_msgsnd): Likewise. * sysdeps/unix/sysv/linux/openat.c (OPENAT_NOT_CANCEL): Likewise. * sysdeps/unix/sysv/linux/powerpc/powerpc32/pread.c (__libc_pread): Likewise. * sysdeps/unix/sysv/linux/powerpc/powerpc32/pread64.c (__libc_pread64): Likewise. * sysdeps/unix/sysv/linux/powerpc/powerpc32/pwrite.c (__libc_pwrite): Likewise. * sysdeps/unix/sysv/linux/powerpc/powerpc32/pwrite64.c (__libc_pwrite64): Likewise. * sysdeps/unix/sysv/linux/pread.c (__libc_pread): Likewise. * sysdeps/unix/sysv/linux/pread64.c (__libc_pread64): Likewise. * sysdeps/unix/sysv/linux/preadv.c (PREADV): Likewise. * sysdeps/unix/sysv/linux/pthread_kill.c (__pthread_kill): Likewise. * sysdeps/unix/sysv/linux/pwrite.c (__libc_pwrite): Likewise. * sysdeps/unix/sysv/linux/pwrite64.c (__libc_pwrite64): Likewise. * sysdeps/unix/sysv/linux/pwritev.c (__libc_pwritev): Likewise. * sysdeps/unix/sysv/linux/sh/pread.c (__libc_pread): Likewise. * sysdeps/unix/sysv/linux/sh/pread64.c (__libc_pread64): Likewise. * sysdeps/unix/sysv/linux/sh/pwrite.c (__libc_pwrite): Likewise. * sysdeps/unix/sysv/linux/sh/pwrite64.c (__libc_write64): Likewise. * sysdeps/unix/sysv/linux/sigsuspend.c (__sigsuspend): Likewise. * sysdeps/unix/sysv/linux/sigtimedwait.c (__sigtimedwait): Likewise. * sysdeps/unix/sysv/linux/sigwait.c (__sigwait): Likewise. * sysdeps/unix/sysv/linux/sigwaitinfo.c (__sigwaitinfo): Likewise. * sysdeps/unix/sysv/linux/sparc/sparc64/msgrcv.c (__libc_msgrcv): Likewise.
2015-01-02Update copyright dates with scripts/update-copyrights.Joseph Myers1-1/+1
2014-12-06Fix pthreads getrlimit, gettimeofday namespace (bug 17682).Joseph Myers1-1/+1
Some pthreads functions use getrlimit and gettimeofday, but these functions are XSI, not base POSIX; this is a namespace issue for dynamic linking as well as static linking. This patch makes them use __getrlimit and __gettimeofday instead - the former needed to be newly exported from libc.so at GLIBC_PRIVATE (and so now needs libc_hidden_proto / libc_hidden_def), the latter was already exported. Tested for x86_64 (testsuite, and that disassembly of installed shared libraries is unchanged by the patch). [BZ #17682] * resource/Versions (libc): Add __getrlimit at GLIBC_PRIVATE. * resource/getrlimit.c (__getrlimit): Use libc_hidden_def. * sysdeps/mach/hurd/getrlimit.c (__getrlimit): Likewise. * include/sys/resource.h (__getrlimit): Use libc_hidden_proto. * nptl/nptl-init.c (__pthread_initialize_minimal_internal): Use __getrlimit instead of getrlimit. * nptl/pthread_cond_timedwait.c (__pthread_cond_timedwait): Use __gettimeofday instead of gettimeofday. * nptl/pthread_rwlock_timedrdlock.c (pthread_rwlock_timedrdlock): Likewise. * nptl/pthread_rwlock_timedwrlock.c (pthread_rwlock_timedwrlock): Likewise. * sysdeps/pthread/aio_misc.c (handle_fildes_io): Likewise. * conform/Makefile (test-xfail-POSIX2008/aio.h/linknamespace): Remove variable. (test-xfail-POSIX2008/pthread.h/linknamespace): Likewise. (test-xfail-POSIX2008/time.h/linknamespace): Likewise.
2014-10-20NPTL: Add some missing #include'sRoland McGrath1-0/+1
2014-02-10Use glibc_likely instead __builtin_expect.Ondřej Bílka1-3/+3
2014-01-01Update copyright notices with scripts/update-copyrightsAllan McRae1-1/+1
2013-02-18FUTEX_*_REQUEUE_PI support for non-x86 codeSiddhesh Poyarekar1-8/+51
Add FUTEX_*_REQUEUE_PI support for the default C code and also add implementations for s-390 and ppc.
2013-01-02Update copyright notices with scripts/update-copyrights.Joseph Myers1-1/+1
2012-11-21Remove unused variable from pthread_cond_timedwait.c.Joseph Myers1-5/+4
2012-11-05[S390,PPC] Implement FUTEX_WAIT_BITSET for timedwait functionsSiddhesh Poyarekar1-3/+19
Since the FUTEX_WAIT operation takes a relative timeout, the pthread_cond_timedwait and other timed function implementations have to get a relative timeout from the absolute timeout parameter it gets before it makes the futex syscall. This value is then converted back into an absolute timeout within the kernel. This is a waste and has hence been improved upon by a FUTEX_WAIT_BITSET operation (OR'd with FUTEX_CLOCK_REALTIME to make the kernel use the realtime clock instead of the default monotonic clock). This was implemented only in the x86 and sh assembly code and not in the C code. This patch implements support for FUTEX_WAIT_BITSET whenever available (since linux-2.6.29) for s390 and powerpc.
2012-08-16Remove __ASSUME_POSIX_TIMERS.Joseph Myers1-18/+4
2012-02-09Replace FSF snail mail address with URLs.Paul Eggert1-3/+2
2011-01-19Fix comment typos.Roland McGrath1-2/+2
2010-03-24Enable common version of pthread_cond_timedwait to use clock_gettime in VDSOLuis Machado1-2/+10
2007-08-13* nscd/servicescache.c: Include kernel-features.h.Ulrich Drepper1-0/+1
* nscd/gai.c: Likewise. * sysdeps/unix/sysv/linux/statfs64.c: Likewise. * sysdeps/unix/sysv/linux/fstatfs64.c: Likewise. * sysdeps/unix/sysv/linux/fxstatat.c: Likewise. * sysdeps/unix/sysv/linux/s390/s390-32/mmap.S: Likewise. * sysdeps/unix/sysv/linux/s390/s390-32/mmap64.S: Likewise. * sysdeps/unix/sysv/linux/xstatconv.c: Likewise. * sysdeps/unix/sysv/linux/if_index.c: Likewise. * sysdeps/unix/sysv/linux/sparc/sparc64/xstat.c: Likewise. * sysdeps/unix/sysv/linux/ifaddrs.c: Likewise.
2007-08-11* pthreadP.h (PTHREAD_ROBUST_MUTEX_PSHARED): Define.Ulrich Drepper1-11/+10
* pthread_mutex_lock.c: Use it instead of PTHREAD_MUTEX_PSHARED when dealing with robust mutexes. * pthread_mutex_timedlock.c: Likewise. * pthread_mutex_trylock.c: Likewise. * pthread_mutex_unlock.c: Likewise. * sysdeps/unix/sysv/linux/pthread_mutex_cond_lock.c: Likewise. 2007-08-06 Jakub Jelinek <jakub@redhat.com> * pthreadP.h (PTHREAD_MUTEX_PSHARED_BIT): Define. (PTHREAD_MUTEX_TYPE): Mask __kind with 127. (PTHREAD_MUTEX_PSHARED): Define. * pthread_mutex_init.c (__pthread_mutex_init): Set PTHREAD_MUTEX_PSHARED_BIT for pshared or robust mutexes. * pthread_mutex_lock.c (LLL_MUTEX_LOCK): Take mutex as argument instead of its __data.__lock field, pass PTHREAD_MUTEX_PSHARED as second argument to lll_lock. (LLL_MUTEX_TRYLOCK): Take mutex as argument instead of its __data.__lock field. (LLL_ROBUST_MUTEX_LOCK): Take mutex as argument instead of its __data.__lock field, pass PTHREAD_MUTEX_PSHARED as second argument to lll_robust_lock. (__pthread_mutex_lock): Update LLL_MUTEX_LOCK, LLL_MUTEX_TRYLOCK, LLL_ROBUST_MUTEX_LOCK users, use PTHREAD_MUTEX_TYPE (mutex) instead of mutex->__data.__kind directly, pass PTHREAD_MUTEX_PSHARED (mutex) to lll_unlock and lll_futex_wait. * pthread_mutex_trylock.c (__pthread_mutex_trylock): Use PTHREAD_MUTEX_TYPE (mutex) instead of mutex->__data.__kind directly, pass PTHREAD_MUTEX_PSHARED (mutex) to lll_unlock. (pthread_mutex_timedlock): Pass PTHREAD_MUTEX_PSHARED (mutex) to lll_timedlock, lll_robust_timedlock, lll_unlock and lll_futex_timed_wait. Use PTHREAD_MUTEX_TYPE (mutex) instead of mutex->__data.__kind directly. * pthread_mutex_timedlock.c (pthread_mutex_timedlock): Pass PTHREAD_MUTEX_PSHARED (mutex) to lll_timedlock, lll_robust_timedlock, lll_unlock and lll_futex_timed_wait. Use PTHREAD_MUTEX_TYPE (mutex) instead of mutex->__data.__kind directly. * pthread_mutex_unlock.c (__pthread_mutex_unlock_usercnt): Pass PTHREAD_MUTEX_PSHARED (mutex) to lll_unlock, lll_robust_unlock and lll_futex_wake. * pthread_mutex_setprioceiling.c (pthread_mutex_setprioceiling): Pass PTHREAD_MUTEX_PSHARED (mutex) to lll_futex_wait and lll_futex_wake. Use PTHREAD_MUTEX_TYPE (mutex) instead of mutex->__data.__kind directly. * sysdeps/unix/sysv/linux/pthread_mutex_cond_lock.c (LLL_MUTEX_LOCK): Take mutex as argument instead of its __data.__lock field, pass PTHREAD_MUTEX_PSHARED as second argument to lll_cond_lock. (LLL_MUTEX_TRYLOCK): Take mutex as argument instead of its __data.__lock field. (LLL_ROBUST_MUTEX_LOCK): Take mutex as argument instead of its __data.__lock field, pass PTHREAD_MUTEX_PSHARED as second argument to lll_robust_cond_lock. * pthread_cond_broadcast.c (__pthread_cond_broadcast): Add pshared variable, pass it to lll_lock, lll_unlock, lll_futex_requeue and lll_futex_wake. Don't use lll_futex_requeue if dependent mutex has PTHREAD_MUTEX_PSHARED_BIT bit set in its __data.__kind. * pthread_cond_destroy.c (__pthread_cond_destroy): Add pshared variable, pass it to lll_lock, lll_unlock, lll_futex_wake and lll_futex_wait. * pthread_cond_signal.c (__pthread_cond_signal): Add pshared variable, pass it to lll_lock, lll_unlock, lll_futex_wake_unlock and lll_futex_wake. * pthread_cond_timedwait.c (__pthread_cond_wait): Add pshared variable, pass it to lll_lock, lll_unlock, lll_futex_timedwait and lll_futex_wake. * pthread_cond_wait.c (__condvar_cleanup, __pthread_cond_wait): Add pshared variable, pass it to lll_lock, lll_unlock, lll_futex_wait and lll_futex_wake. * sysdeps/unix/sysv/linux/alpha/lowlevellock.h (lll_futex_requeue, lll_futex_wake_unlock): Add private argument, use __lll_private_flag macro. * sysdeps/unix/sysv/linux/ia64/lowlevellock.h (lll_futex_requeue, lll_futex_wake_unlock): Likewise. * sysdeps/unix/sysv/linux/powerpc/lowlevellock.h (lll_futex_requeue): Likewise. * sysdeps/unix/sysv/linux/sparc/lowlevellock.h (lll_futex_requeue, lll_futex_wake_unlock): Likewise. * sysdeps/unix/sysv/linux/x86_64/lowlevellock.h (lll_futex_requeue): Likewise. * sysdeps/unix/sysv/linux/s390/lowlevellock.h (lll_futex_requeue, lll_futex_wake_unlock): Likewise. (lll_futex_wake): Fix a typo. * sysdeps/unix/sysv/linux/pthread-pi-defines.sym (PS_BIT): Add. * sysdeps/unix/sysv/linux/x86_64/pthread_cond_broadcast.S (__pthread_cond_broadcast): Pass LLL_PRIVATE to lll_* and or FUTEX_PRIVATE_FLAG into SYS_futex op if cv is process private. Don't use FUTEX_CMP_REQUEUE if dep_mutex is not process private. * sysdeps/unix/sysv/linux/x86_64/pthread_cond_signal.S (__pthread_cond_signal): Pass LLL_PRIVATE to lll_* and or FUTEX_PRIVATE_FLAG into SYS_futex op if cv is process private. * sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S (__pthread_cond_timedwait): Likewise. * sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S: (__condvar_cleanup, __pthread_cond_wait): Likewise.
2007-08-01* sysdeps/unix/sysv/linux/powerpc/sem_post.c (__new_sem_post):Ulrich Drepper1-5/+5
Use __asm __volatile (__lll_acq_instr ::: "memory") instead of atomic_full_barrier. 2007-07-31 Jakub Jelinek <jakub@redhat.com> * allocatestack.c (stack_cache_lock): Change type to int. (get_cached_stack, allocate_stack, __deallocate_stack, __make_stacks_executable, __find_thread_by_id, __nptl_setxid, __pthread_init_static_tls, __wait_lookup_done): Add LLL_PRIVATE as second argument to lll_lock and lll_unlock macros on stack_cache_lock. * pthread_create.c (__find_in_stack_list): Likewise. (start_thread): Similarly with pd->lock. Use lll_robust_dead macro instead of lll_robust_mutex_dead, pass LLL_SHARED to it as second argument. * descr.h (struct pthread): Change lock and setxid_futex field type to int. * old_pthread_cond_broadcast.c (__pthread_cond_broadcast_2_0): Use LLL_LOCK_INITIALIZER instead of LLL_MUTEX_LOCK_INITIALIZER. * old_pthread_cond_signal.c (__pthread_cond_signal_2_0): Likewise. * old_pthread_cond_timedwait.c (__pthread_cond_timedwait_2_0): Likewise. * old_pthread_cond_wait.c (__pthread_cond_wait_2_0): Likewise. * pthread_cond_init.c (__pthread_cond_init): Likewise. * pthreadP.h (__attr_list_lock): Change type to int. * pthread_attr_init.c (__attr_list_lock): Likewise. * pthread_barrier_destroy.c (pthread_barrier_destroy): Pass ibarrier->private ^ FUTEX_PRIVATE_FLAG as second argument to lll_{,un}lock. * pthread_barrier_wait.c (pthread_barrier_wait): Likewise and also for lll_futex_{wake,wait}. * pthread_barrier_init.c (pthread_barrier_init): Make iattr a pointer to const. * pthread_cond_broadcast.c (__pthread_cond_broadcast): Pass LLL_SHARED as second argument to lll_{,un}lock. * pthread_cond_destroy.c (__pthread_cond_destroy): Likewise. * pthread_cond_signal.c (__pthread_cond_singal): Likewise. * pthread_cond_timedwait.c (__pthread_cond_timedwait): Likewise. * pthread_cond_wait.c (__condvar_cleanup, __pthread_cond_wait): Likewise. * pthread_getattr_np.c (pthread_getattr_np): Add LLL_PRIVATE as second argument to lll_{,un}lock macros on pd->lock. * pthread_getschedparam.c (__pthread_getschedparam): Likewise. * pthread_setschedparam.c (__pthread_setschedparam): Likewise. * pthread_setschedprio.c (pthread_setschedprio): Likewise. * tpp.c (__pthread_tpp_change_priority, __pthread_current_priority): Likewise. * sysdeps/pthread/createthread.c (do_clone, create_thread): Likewise. * pthread_once.c (once_lock): Change type to int. (__pthread_once): Pass LLL_PRIVATE as second argument to lll_{,un}lock macros on once_lock. * pthread_rwlock_rdlock.c (__pthread_rwlock_rdlock): Use lll_{,un}lock macros instead of lll_mutex_{,un}lock, pass rwlock->__data.__shared as second argument to them and similarly for lll_futex_w*. * pthread_rwlock_timedrdlock.c (pthread_rwlock_timedrdlock): Likewise. * pthread_rwlock_timedwrlock.c (pthread_rwlock_timedwrlock): Likewise. * pthread_rwlock_tryrdlock.c (__pthread_rwlock_tryrdlock): Likewise. * pthread_rwlock_trywrlock.c (__pthread_rwlock_trywrlock): Likewise. * pthread_rwlock_unlock.c (__pthread_rwlock_unlock): Likewise. * pthread_rwlock_wrlock.c (__pthread_rwlock_wrlock): Likewise. * sem_close.c (sem_close): Pass LLL_PRIVATE as second argument to lll_{,un}lock macros on __sem_mappings_lock. * sem_open.c (check_add_mapping): Likewise. (__sem_mappings_lock): Change type to int. * semaphoreP.h (__sem_mappings_lock): Likewise. * pthread_mutex_lock.c (LLL_MUTEX_LOCK, LLL_MUTEX_TRYLOCK, LLL_ROBUST_MUTEX_LOCK): Use lll_{,try,robust_}lock macros instead of lll_*mutex_*, pass LLL_SHARED as last argument. (__pthread_mutex_lock): Use lll_unlock instead of lll_mutex_unlock, pass LLL_SHARED as last argument. * sysdeps/unix/sysv/linux/pthread_mutex_cond_lock.c (LLL_MUTEX_LOCK, LLL_MUTEX_TRYLOCK, LLL_ROBUST_MUTEX_LOCK): Use lll_{cond_,cond_try,robust_cond}lock macros instead of lll_*mutex_*, pass LLL_SHARED as last argument. * pthread_mutex_timedlock.c (pthread_mutex_timedlock): Use lll_{timed,try,robust_timed,un}lock instead of lll_*mutex*, pass LLL_SHARED as last argument. * pthread_mutex_trylock.c (__pthread_mutex_trylock): Similarly. * pthread_mutex_unlock.c (__pthread_mutex_unlock_usercnt): Similarly. * sysdeps/pthread/bits/libc-lock.h (__libc_lock_lock, __libc_lock_lock_recursive, __libc_lock_unlock, __libc_lock_unlock_recursive): Pass LLL_PRIVATE as second argument to lll_{,un}lock. * sysdeps/pthread/bits/stdio-lock.h (_IO_lock_lock, _IO_lock_unlock): Likewise. * sysdeps/unix/sysv/linux/fork.c (__libc_fork): Don't use compound literal. * sysdeps/unix/sysv/linux/unregister-atfork.c (__unregister_atfork): Pass LLL_PRIVATE as second argument to lll_{,un}lock macros on __fork_lock. * sysdeps/unix/sysv/linux/register-atfork.c (__register_atfork, free_mem): Likewise. (__fork_lock): Change type to int. * sysdeps/unix/sysv/linux/fork.h (__fork_lock): Likewise. * sysdeps/unix/sysv/linux/sem_post.c (__new_sem_post): Pass isem->private ^ FUTEX_PRIVATE_FLAG as second argument to lll_futex_wake. * sysdeps/unix/sysv/linux/sem_timedwait.c (sem_timedwait): Likewise. * sysdeps/unix/sysv/linux/sem_wait.c (__new_sem_wait): Likewise. * sysdeps/unix/sysv/linux/lowlevellock.c (__lll_lock_wait_private): New function. (__lll_lock_wait, __lll_timedlock_wait): Add private argument and pass it through to lll_futex_*wait, only compile in when IS_IN_libpthread. * sysdeps/unix/sysv/linux/lowlevelrobustlock.c (__lll_robust_lock_wait, __lll_robust_timedlock_wait): Add private argument and pass it through to lll_futex_*wait. * sysdeps/unix/sysv/linux/alpha/lowlevellock.h: Renamed all lll_mutex_* resp. lll_robust_mutex_* macros to lll_* resp. lll_robust_*. Renamed all __lll_mutex_* resp. __lll_robust_mutex_* inline functions to __lll_* resp. __lll_robust_*. (LLL_MUTEX_LOCK_INITIALIZER): Remove. (lll_mutex_dead): Add private argument. (__lll_lock_wait_private): New prototype. (__lll_lock_wait, __lll_robust_lock_wait, __lll_lock_timedwait, __lll_robust_lock_timedwait): Add private argument to prototypes. (__lll_lock): Add private argument, if it is constant LLL_PRIVATE, call __lll_lock_wait_private, otherwise pass private to __lll_lock_wait. (__lll_robust_lock, __lll_cond_lock, __lll_timedlock, __lll_robust_timedlock): Add private argument, pass it to __lll_*wait functions. (__lll_unlock): Add private argument, if it is constant LLL_PRIVATE, call __lll_unlock_wake_private, otherwise pass private to __lll_unlock_wake. (__lll_robust_unlock): Add private argument, pass it to __lll_robust_unlock_wake. (lll_lock, lll_robust_lock, lll_cond_lock, lll_timedlock, lll_robust_timedlock, lll_unlock, lll_robust_unlock): Add private argument, pass it through to __lll_* inline function. (__lll_mutex_unlock_force, lll_mutex_unlock_force): Remove. (lll_lock_t): Remove. (__lll_cond_wait, __lll_cond_timedwait, __lll_cond_wake, __lll_cond_broadcast, lll_cond_wait, lll_cond_timedwait, lll_cond_wake, lll_cond_broadcast): Remove. * sysdeps/unix/sysv/linux/ia64/lowlevellock.h: Likewise. * sysdeps/unix/sysv/linux/powerpc/lowlevellock.h: Likewise. * sysdeps/unix/sysv/linux/s390/lowlevellock.h: Likewise. * sysdeps/unix/sysv/linux/sparc/lowlevellock.h: Likewise. * sysdeps/unix/sysv/linux/i386/lowlevellock.h: Allow including the header from assembler. Renamed all lll_mutex_* resp. lll_robust_mutex_* macros to lll_* resp. lll_robust_*. (LOCK, FUTEX_CMP_REQUEUE, FUTEX_WAKE_OP, FUTEX_OP_CLEAR_WAKE_IF_GT_ONE): Define. (LLL_MUTEX_LOCK_INITIALIZER, LLL_MUTEX_LOCK_INITIALIZER_LOCKED, LLL_MUTEX_LOCK_INITIALIZER_WAITERS): Remove. (__lll_mutex_lock_wait, __lll_mutex_timedlock_wait, __lll_mutex_unlock_wake, __lll_lock_wait, __lll_unlock_wake): Remove prototype. (__lll_trylock_asm, __lll_lock_asm_start, __lll_unlock_asm): Define. (lll_robust_trylock, lll_cond_trylock): Use LLL_LOCK_INITIALIZER* rather than LLL_MUTEX_LOCK_INITIALIZER* macros. (lll_trylock): Likewise, use __lll_trylock_asm, pass MULTIPLE_THREADS_OFFSET as another asm operand. (lll_lock): Add private argument, use __lll_lock_asm_start, pass MULTIPLE_THREADS_OFFSET as last asm operand, call __lll_lock_wait_private if private is constant LLL_PRIVATE, otherwise pass private as another argument to __lll_lock_wait. (lll_robust_lock, lll_cond_lock, lll_robust_cond_lock, lll_timedlock, lll_robust_timedlock): Add private argument, pass private as another argument to __lll_*lock_wait call. (lll_unlock): Add private argument, use __lll_unlock_asm, pass MULTIPLE_THREADS_OFFSET as another asm operand, call __lll_unlock_wake_private if private is constant LLL_PRIVATE, otherwise pass private as another argument to __lll_unlock_wake. (lll_robust_unlock): Add private argument, pass private as another argument to __lll_unlock_wake. (lll_robust_dead): Add private argument, use __lll_private_flag macro. (lll_islocked): Use LLL_LOCK_INITIALIZER instead of LLL_MUTEX_LOCK_INITIALIZER. (lll_lock_t): Remove. (LLL_LOCK_INITIALIZER_WAITERS): Define. (__lll_cond_wait, __lll_cond_timedwait, __lll_cond_wake, __lll_cond_broadcast, lll_cond_wait, lll_cond_timedwait, lll_cond_wake, lll_cond_broadcast): Remove. * sysdeps/unix/sysv/linux/x86_64/lowlevellock.h: Likewise. * sysdeps/unix/sysv/linux/i386/i486/libc-lowlevellock.S: Revert 2007-05-2{3,9} changes. * sysdeps/unix/sysv/linux/i386/i486/lowlevellock.S: Include kernel-features.h and lowlevellock.h. (LOAD_PRIVATE_FUTEX_WAIT): Define. (LOAD_FUTEX_WAIT): Rewritten. (LOCK, SYS_gettimeofday, SYS_futex, FUTEX_WAIT, FUTEX_WAKE): Don't define. (__lll_lock_wait_private, __lll_unlock_wake_private): New functions. (__lll_mutex_lock_wait): Rename to ... (__lll_lock_wait): ... this. Take futex addr from %edx instead of %ecx, %ecx is now private argument. Don't compile in for libc.so. (__lll_mutex_timedlock_wait): Rename to ... (__lll_timedlock_wait): ... this. Use __NR_gettimeofday. %esi contains private argument. Don't compile in for libc.so. (__lll_mutex_unlock_wake): Rename to ... (__lll_unlock_wake): ... this. %ecx contains private argument. Don't compile in for libc.so. (__lll_timedwait_tid): Use __NR_gettimeofday. * sysdeps/unix/sysv/linux/i386/i486/lowlevelrobustlock.S: Include kernel-features.h and lowlevellock.h. (LOAD_FUTEX_WAIT): Define. (LOCK, SYS_gettimeofday, SYS_futex, FUTEX_WAIT, FUTEX_WAKE): Don't define. (__lll_robust_mutex_lock_wait): Rename to ... (__lll_robust_lock_wait): ... this. Futex addr is now in %edx argument, %ecx argument contains private. Use LOAD_FUTEX_WAIT macro. (__lll_robust_mutex_timedlock_wait): Rename to ... (__lll_robust_timedlock_wait): ... this. Use __NR_gettimeofday. %esi argument contains private, use LOAD_FUTEX_WAIT macro. * sysdeps/unix/sysv/linux/i386/i486/pthread_barrier_wait.S: Include lowlevellock.h. (SYS_futex, FUTEX_WAIT, FUTEX_WAKE, LOCK): Don't define. (pthread_barrier_wait): Rename __lll_mutex_* to __lll_*, pass PRIVATE(%ebx) ^ LLL_SHARED as private argument in %ecx to __lll_lock_wait and __lll_unlock_wake, pass MUTEX(%ebx) address to __lll_lock_wait in %edx. * sysdeps/unix/sysv/linux/i386/i486/pthread_cond_broadcast.S: Include lowlevellock.h and pthread-errnos.h. (SYS_futex, FUTEX_WAIT, FUTEX_WAKE, FUTEX_REQUEUE, FUTEX_CMP_REQUEUE, EINVAL, LOCK): Don't define. (__pthread_cond_broadcast): Rename __lll_mutex_* to __lll_*, pass cond_lock address in %edx rather than %ecx to __lll_lock_wait, pass LLL_SHARED in %ecx to both __lll_lock_wait and __lll_unlock_wake. * sysdeps/unix/sysv/linux/i386/i486/pthread_cond_signal.S: Include lowlevellock.h and pthread-errnos.h. (SYS_futex, FUTEX_WAIT, FUTEX_WAKE, FUTEX_WAKE_OP, FUTEX_OP_CLEAR_WAKE_IF_GT_ONE, EINVAL, LOCK): Don't define. (__pthread_cond_signal): Rename __lll_mutex_* to __lll_*, pass cond_lock address in %edx rather than %ecx to __lll_lock_wait, pass LLL_SHARED in %ecx to both __lll_lock_wait and __lll_unlock_wake. * sysdeps/unix/sysv/linux/i386/i486/pthread_cond_timedwait.S: Include lowlevellock.h. (SYS_futex, SYS_gettimeofday, FUTEX_WAIT, FUTEX_WAKE, LOCK): Don't define. (__pthread_cond_timedwait): Rename __lll_mutex_* to __lll_*, pass cond_lock address in %edx rather than %ecx to __lll_lock_wait, pass LLL_SHARED in %ecx to both __lll_lock_wait and __lll_unlock_wake. Use __NR_gettimeofday. * sysdeps/unix/sysv/linux/i386/i486/pthread_cond_wait.S: Include lowlevellock.h. (SYS_futex, FUTEX_WAIT, FUTEX_WAKE, LOCK): Don't define. (__pthread_cond_wait, __condvar_w_cleanup): Rename __lll_mutex_* to __lll_*, pass cond_lock address in %edx rather than %ecx to __lll_lock_wait, pass LLL_SHARED in %ecx to both __lll_lock_wait and __lll_unlock_wake. * sysdeps/unix/sysv/linux/i386/i486/pthread_rwlock_rdlock.S: Include lowlevellock.h. (SYS_futex, FUTEX_WAIT, FUTEX_WAKE, LOCK): Don't define. (__pthread_rwlock_rdlock): Rename __lll_mutex_* to __lll_*, pass MUTEX(%ebx) address in %edx rather than %ecx to __lll_lock_wait, pass PSHARED(%ebx) in %ecx to both __lll_lock_wait and __lll_unlock_wake. Move return value from %ecx to %edx register. * sysdeps/unix/sysv/linux/i386/i486/pthread_rwlock_timedrdlock.S: Include lowlevellock.h. (SYS_futex, SYS_gettimeofday, FUTEX_WAIT, FUTEX_WAKE, LOCK): Don't define. (__pthread_rwlock_wrlock): Rename __lll_mutex_* to __lll_*, pass MUTEX(%ebp) address in %edx rather than %ecx to __lll_lock_wait, pass PSHARED(%ebp) in %ecx to both __lll_lock_wait and __lll_unlock_wake. Move return value from %ecx to %edx register. Use __NR_gettimeofday. * sysdeps/unix/sysv/linux/i386/i486/pthread_rwlock_timedwrlock.S: Include lowlevellock.h. (SYS_futex, SYS_gettimeofday, FUTEX_WAIT, FUTEX_WAKE, LOCK): Don't define. (__pthread_rwlock_wrlock): Rename __lll_mutex_* to __lll_*, pass MUTEX(%ebp) address in %edx rather than %ecx to __lll_lock_wait, pass PSHARED(%ebp) in %ecx to both __lll_lock_wait and __lll_unlock_wake. Move return value from %ecx to %edx register. Use __NR_gettimeofday. * sysdeps/unix/sysv/linux/i386/i486/pthread_rwlock_unlock.S: Include lowlevellock.h. (SYS_futex, FUTEX_WAIT, FUTEX_WAKE, LOCK): Don't define. (__pthread_rwlock_unlock): Rename __lll_mutex_* to __lll_*, pass MUTEX(%edi) address in %edx rather than %ecx to __lll_lock_wait, pass PSHARED(%edi) in %ecx to both __lll_lock_wait and __lll_unlock_wake. * sysdeps/unix/sysv/linux/i386/i486/pthread_rwlock_wrlock.S: Include lowlevellock.h. (SYS_futex, FUTEX_WAIT, FUTEX_WAKE, LOCK): Don't define. (__pthread_rwlock_wrlock): Rename __lll_mutex_* to __lll_*, pass MUTEX(%ebx) address in %edx rather than %ecx to __lll_lock_wait, pass PSHARED(%ebx) in %ecx to both __lll_lock_wait and __lll_unlock_wake. Move return value from %ecx to %edx register. * sysdeps/unix/sysv/linux/i386/pthread_once.S: Include lowlevellock.h. (LOCK, SYS_futex, FUTEX_WAIT, FUTEX_WAKE, FUTEX_PRIVATE_FLAG): Don't define. * sysdeps/unix/sysv/linux/i386/i486/sem_post.S: Include lowlevellock.h. (LOCK, SYS_futex, FUTEX_WAKE): Don't define. * sysdeps/unix/sysv/linux/i386/i486/sem_timedwait.S: Include lowlevellock.h. (LOCK, SYS_futex, SYS_gettimeofday, FUTEX_WAIT): Don't define. (sem_timedwait): Use __NR_gettimeofday. * sysdeps/unix/sysv/linux/i386/i486/sem_trywait.S: Include lowlevellock.h. (LOCK): Don't define. * sysdeps/unix/sysv/linux/i386/i486/sem_wait.S: Include lowlevellock.h. (LOCK, SYS_futex, FUTEX_WAIT): Don't define. * sysdeps/unix/sysv/linux/powerpc/sem_post.c: Wake only when there are waiters. * sysdeps/unix/sysv/linux/x86_64/libc-lowlevellock.S: Revert 2007-05-2{3,9} changes. * sysdeps/unix/sysv/linux/x86_64/lowlevellock