Merge pull request #4089 from RosettaCommons/vmullig/multithreaded_packer_setup2
Multi-thread packer setup using the RosettaThreadManager work vector interface
This pull request replaces pull request #3964.
A current challenge is figuring out how to get all of the existing Rosetta protocols to take advantage of parallel processing, with as little burden placed on the user as possible. Much of the time in a typical Rosetta run is spent on:
- Computing interaction graphs (easy to parallelize).
- Running packing trajectories (hard to parallelize).
- Computing gradient vectors and inverse Hessian approximations for minimization (easy to parallelize).
- Computing the scoring function (somewhat hard to parallelize).
- Computing kinematics (may be parallelizable?)
This pull request aims to address the first point: it will allow interaction graphs to be calculated using the `RosettaThreadManager`'s thread pool, implemented in pull request #3946. The `RosettaThreadManager` is intended to allow us to experiment with parallelizing things on many different levels in Rosetta (from apps to job distributors to movers, down to low-level components like the packer, minimizer, and scoring machinery), without having things that call things that call things create thread explosions. (So, for example, Rosetta could launch 16 threads, the job distributor could request parallel job execution in 4 threads, and each parallel job thread could request up to 16 threads for packing, receiving 1 to 13 depending on availability so that the global pool never exceeds 16 running threads. Note that switching the `MultithreadedJobDistributor` to use the `RosettaThreadManager` will be for a future pull request.) It also allows very finely-grained threading, without having to encapsulate atomistic tasks in jobs that might have more overhead than the tasks themselves (as would be the case if we wanted to consider individual entries in the interaction graph calculation as "jobs").
Tasks:
- [ ] **FIX PYROSETTA DISTRIBUTED ISSUE**
- [x] Pull request #4310 must be merged before this one.
- [x] Pull request #3946 must be merged before this one.
- [x] Pull request #3957 must be merged before this one.
- [x] Add option (flag) for default number of threads that the packer will request.
- [x] Ensure that `fixbb` respects the option.
- [x] Test `fixbb`.
- [x] Ensure that the `PackRotamersMover` respects the option by way of TaskOperations.
- [x] Test `PackRotamersMover`. (Update test).
- [x] Ensure that the test checks that the single- and multi-threaded PDB output is identical.
- [x] Ensure that `FastRelax` respects the option by way of TaskOperations.
- [x] Ensure that `FastDesign` respects the option by way of TaskOperations.
- [x] Test `FastDesign`. (Update test).
- [x] Update the no-threads case in the `FastDesign` and `PackRotamersMover` tests.
- [x] ~~Modify `PackRotamersMover`:~~
- [x] ~~Add `MultithreadedMover` class that derives from `Mover` class.~~
- [x] ~~Have `MultithreadedMover` take an XML-configurable option for number of threads to request.~~
- [x] ~~Make the `PackRotamersMover` a `MultithreadedMover`, with its "number of threads" option overriding the global default if and only if provided by the user.~~
- [x] ~~Have the `PackRotamersMover` call the multithreaded `core::pack::pack_rotamers()` if number of threads is greater than 1.~~
- [x] Parallelize interaction graph setup.
- [x] Pre-identify (in a single thread) the one- and two-body interactions to be computed.
- [x] Pre-create the nodes and edges needed (using a single thread) in the interaction graph.
- [x] Handle the symmetric case, too -- need virtual function overrides in `SymmetricRotamerSets`.
- [x] Modify the interaction graph to allow parallel, threadsafe computation of elements.
- [x] Make a work vector of all onebody and twobody energies to compute, and pass it to the `RosettaThreadManager`.
- [x] Handle the lazy IG case -- probably not much that multithreading can do, but take a look anyway. --> Can only multithread the onebody energy precalculation. Meh. Better than nothing.
- [x] Integration/unit test the 1-thread case.
- [x] Integration/unit test the multi-threaded case.
- In the future, integration test multi-threaded packing in the context of multi-threaded job distribution.
- [x] Looks like I've got the same rare segfault issue. Track it down. --> Got it. Can't call functions that finalize edges from threads.
- [x] Integration test: confirm that interaction graphs computed in parallel are identical to interaction graphs computed in serial.
- [x] Symmetric case, too.
- [x] Move the `-packing:packing_threads` option to `-multi_threading:packing_threads` to keep all of these options together.
- [x] Same for the optH threads option.
- [x] Pilot app for performance testing.
- [x] Switch tracer output to debug output, or remove entirely.
- [x] Special-case the logic for 1 thread requested to use the non-threaded code.
- [x] Do this for `SymmetricRotamerSets`, too.
- [x] Fix the twobody energy calculation for the symmetric case.
- [x] Handle the case of threads requested in a non-threaded build.
- [x] **FIX PYROSETTA DISTRIBUTED ISSUE.**
- [ ] Developer documentation.
- [ ] User documentation.
- [x] Beauty.
TODO:
- [x] Make commandline flag more specific (`-interaction_graph_threads` instead of `packing_threads`.)
- [x] Take out new options from hydrate, FastRelax, FastDesign, PackRotamerMover, etc. for threads. Instead...
- [x] Add a new TaskOperation for setting number or threads.
- [x] PackerTask stores number of threads to request for interaction graph setup.
- [x] Maintain commutativity: taskop can only REDUCE the number of threads.
- [x] Have default behaviour be to use all available threads.
- [x] Switch integration tests to use this.
- [x] See note about Andrew's suggestion for removing code duplication; using work vector even in non-threaded build.
- [x] Allow thread manager to be called in threaded or non-threaded build.
- [x] Ensure that thread manager just executes the work that it is given (single-threaded) in the non-threaded build.
- ~~Add to correctness test at least one score value. (Especially in symmetric case.)~~ --> Already in unit tests.
- [x] Pull construction of temporary rotamer sets in symmetry code out of the precomputation and into the work vector work units (so that they don't persist).
- [x] Scrap the MultithreadedMover base class. For now, TaskOperations are the way to go; later, if we have multithreaded movers, we can add utility functions for their setup rather than having a base class for it.
At some point in the future:
- At some point, remove the default thread counts from `core/util/disulfide_util.hh`, find everything that fails to compile, and switch it all to explicitly request threads.
- At some point, remove the default thread counts from `core/pack/pack_rotamers.hh`, find everything that fails to compile, and switch it all to explicitly request threads.
- At some point, remove the default thread counts from `core/util/SwitchResidueTypeSet.hh`, find everything that fails to compile, and switch it all to explicitly request threads.