Figure
shows rcu_start_gp(), which starts a new grace period,
also releasing the root rcu_node structure's lock, which
must be acquired by the caller.
Line 4 is annotation for the sparse utility, indicating that rcu_start_gp() releases the root rcu_node structure's lock. Local variable rdp references the running CPU's rcu_data structure, rnp references the root rcu_node structure, and rnp_cur and rnp_end are used as cursors in traversing the rcu_node hierarchy.
Line 10 invokes cpu_needs_another_gp() to see if this CPU really needs another grace period to be started, and if not, line 11 releases the root rcu_node structure's lock and line 12 returns. This code path can be executed due to multiple CPUs concurrently attempting to start a grace period. In this case, the winner will start the grace period, and the losers will exit out via this code path.
Otherwise, line 14 increments the specified rcu_state structure's ->gpnum field, officially marking the start of a new grace period.
Quick Quiz D.43:
But there has been no initialization yet at line 15 of
Figure !
What happens if a CPU notices the new grace period and
immediately attempts to report a quiescent state?
Won't it get confused?
End Quick Quiz
Line 15 sets the ->signaled field to RCU_GP_INIT in order to prevent any other CPU from attempting to force an end to the new grace period before its initialization completes. Lines 16-18 schedule the next attempt to force an end to the new grace period, first in terms of jiffies and second in terms of the number of calls to rcu_pending. Of course, if the grace period ends naturally before that time, there will be no need to attempt to force it. Line 20 invokes record_gp_stall_check_time() to schedule a longer-term progress check--if the grace period extends beyond this time, it should be considered to be an error. Line 22 invokes note_new_gpnum() in order to initialize this CPU's rcu_data structure to account for the new grace period.
Lines 23-26 advance all of this CPU's callbacks so that they will be eligible to be invoked at the end of this new grace period. This represents an acceleration of callbacks, as other CPUs would only be able to move the RCU_NEXT_READY_TAIL batch to be serviced by the current grace period; the RCU_NEXT_TAIL would instead need to be advanced to the RCU_NEXT_READY_TAIL batch. The reason that this CPU can accelerate the RCU_NEXT_TAIL batch is that it knows exactly when this new grace period started. In contrast, other CPUs would be unable to correctly resolve the race between the start of a new grace period and the arrival of a new RCU callback.
Line 27 checks to see if there is but one rcu_node structure in the hierarchy, and if so, line 28 sets the ->qsmask bits corresponding to all online CPUs, in other words, corresponding to those CPUs that must pass through a quiescent state for the new grace period to end. Line 29 releases the root rcu_node structure's lock and line 30 returns. In this case, gcc's dead-code elimination is expected to dispense with lines 32-46.
Otherwise, the rcu_node hierarchy has multiple structures, requiring a more involved initialization scheme. Line 32 releases the root rcu_node structure's lock, but keeps interrupts disabled, and then line 33 acquires the specified rcu_state structure's ->onofflock, preventing any concurrent CPU-hotplug operations from manipulating RCU-specific state.
Line 34 sets the rnp_end local variable to reference the first leaf rcu_node structure, which also happens to be the rcu_node structure immediately following the last non-leaf rcu_node structure in the ->node array. Line 35 sets the rnp_cur local variable to reference the root rcu_node structure, which also happens to be first such structure in the ->node array. Lines 36 and 37 then traverse all of the non-leaf rcu_node structures, setting the bits corresponding to lower-level rcu_node structures that have CPUs that must pass through quiescent states in order for the new grace period to end.
Quick Quiz D.44:
Hey!
Shouldn't we hold the non-leaf rcu_node structures'
locks when munging their state in line 37 of
Figure ???
End Quick Quiz
Line 38 sets local variable rnp_end to one past the last leaf rcu_node structure, and line 39 sets local variable rnp_cur to the first leaf rcu_node structure, so that the loop spanning lines 40-44 traverses all leaves of the rcu_node hierarchy. During each pass through this loop, line 41 acquires the current leaf rcu_node structure's lock, line 42 sets the bits corresponding to online CPUs (each of which must pass through a quiescent state before the new grace period can end), and line 43 releases the lock.
Quick Quiz D.45:
Why can't we merge the loop spanning lines 36-37 with
the loop spanning lines 40-44 in
Figure ?
End Quick Quiz
Line 45 then sets the specified rcu_state structure's ->signaled field to permit forcing of quiescent states, and line 46 releases the ->onofflock to permit CPU-hotplug operations to manipulate RCU state.
Paul E. McKenney 2011-12-16