One can execute an exec() system call while holding a lock, and also from within an RCU read-side critical section. The exact semantics depends on the type of primitive.
In the case of non-persistent primitives (including pthread_mutex_lock(), pthread_rwlock_rdlock(), and RCU), if the exec() succeeds, the whole address space vanishes, along with any locks being held. Of course, if the exec() fails, the address space still lives, so any associated locks would also still live. A bit strange perhaps, but reasonably well defined.
On the other hand, persistent primitives (including the flock family, lockf(), System V semaphores, and the O_CREAT flag to open()) would survive regardless of whether the exec() succeeded or failed, so that the exec()ed program might well release them.
Quick Quiz 17.1: What about non-persistent primitives represented by data structures in mmap() regions of memory? What happens when their is an exec() within a critical section of such a primitive? End Quick Quiz
What happens when you attempt to execute an exec() system call from within a transaction?
The exec() system call is perhaps the strangest example of an obstacle to universal TM applicability, as it is not completely clear what approach makes sense, and some might argue that this is merely a reflection of the perils of interacting with execs in real life. That said, the two options prohibiting exec() within transactions are perhaps the most logical of the group.
Paul E. McKenney 2011-12-16