Merge pull request #2360 from RosettaCommons/aleaverfay/threadsafe_rts
This PR lays down some thread safety for an eventual JD3 MultiThreadedJobDistributor. Here, I am working to ensure thread safety for the just-in-time ResidueType loading system as implemented by @rhiju and maintained by @roccomoretti and @everyday847. The idea is this:
The ResidueTypeSet (RTS) will create read locks on its ResidueTypeSetCache (RTSC) before each time it is read from. If it determines that an operation it requries will modify the RTSC, then it releases the read lock and then tries to obtain a write lock.
(Briefly, the "ReadWriteMutex" allows several threads to say "I'm reading" by incrementing a mutex-controlled counter when they start reading, and decrementing that counter when they are done. If a thread decides it needs write permission, then it prevents any more readers from coming along and gaining read permission, while waiting for all the existing threads to complete their work. Once they've completed, then the write lock is granted and the thread may write. One complexity of working with read/write locks is that a thread may deadlock itself if it does something such as obtain a read lock, and then later try to obtain a write lock. Since the read lock is never released, the write lock will never be obtained and the thread will sit there and do nothing. Indeed, if the locks are obtained in the opposite order (first the write lock, then the read lock), the system will also deadlock. Systems involving read/write locks need to build into themselves rules about which functions are off limits at which times.)
The RTS's public interface will handle all the locking required -- the complexity of who is allowed to do what is managed internally by the RTS and its derived classes. The RTS base class provides several template methods that derived classes will supply implementations for (i.e. the GlobalResidueTypeSet and the PoseResidueTypeSet) where these methods will come with lock-based promises and requirements (e.g. generate_residue_type_write_locked is only invoked by the base class after a write lock has been obtained therefor all interactions inside this function with the RTSC are safe -- but the function should not call any of the methods of the RTS that themselves try and obtain read or write locks).
This system is significantly less brittle than the one I had begun working on, and which @roccomoretti helped talk me out of, where the ResidueTypeFinder would be obtaining read and write locks on the RTSC -- that system might have slight performance advantages, however, as it allows the thread to obtain a single read lock or a single write lock and then to complete all of its operations. The PR where I began working on that, PR #2353, has been shelved. Perhaps if this current system proves too slow, we can reconsider its approach