Merge pull request #4072 from RosettaCommons/roccomoretti/component_loading_both
Fix CCD component loading such that it is more complete.
We're now providing the CCD for reading of arbitrary wwPDB recognized ligands, but we're doing it incompletely.
Right now we have an odd system where we'll load a CCD version, but only if there isn't a Rosetta residue type which has the same three letter code. This is a bit silly, as there's a large number of Rosetta residue types with three letter codes which are not equivalent to the wwPDB residue with that three letter code, and these will obscure the CCD version.
We already have a mechanism for removing CCD types that are equivalent to Rosetta types. This is the exclude_pdb_component_list.txt in the Rosetta database. (And associated facilities for -extra_res_fa etc.) If a three letter code is listed there, we don't load the CCD version.
If the CCD version isn't excluded, and has the same three letter code as the (chemically non-equivalent) Rosetta version, we can load both and then let the atom name heuristic (or other such facility) pick the best residue type match. (The CCD types should come after the Rosetta types, so all else being equal, Rosetta should prefer the database version.)
If, for some reason, this behavior doesn't work for you, you can always add -check_all_PDB_components false (now defaults true) to the command line, or call rtf.set_no_CCD_on_name3_match(true) in-code for your ResidueTypeFinder, and this should re-enable the old behavior.
With this, I've dug through the centroid and fa_standard residue type sets, and updated their exclude_pdb_component_list.txt to be more complete.
notify author
notify list [rosetta-logs@googlegroups.com]
Merge pull request #2978 from RosettaCommons/roccomoretti/crash_report
(Client-side code for) Crash Reporter System for Rosetta
This PR merges a bunch of code which improves Rosetta's crash/error reporting handling.
Some highlights
* Backtrace printing is moved from stdout to another file (ROSETTA_CRASH.log).
* This makes the tracer-outputted error message cleaner to see (not hidden behind backtrace gobbledygook), and sets us up to potentially collect the crash log information later.
* This can be turned off by the NOCRASHREPORT compiler define, which is enabled in PyRosetta and BOINC builds.
* Normal runs install a signal handler which should give better diagnostics in cases of segfaults and the like.
* the `-run:nosignal` command line option (or the NOCRASHREPORT define) should turn this off, in case your debugger gives you issues with it.
* New `user_fixable_issue_exit()` and `user_fixable_issue_assert()` macros (parallel to the `utility_exit_with_message()` macro) are provided which will cause an exit without printing the backtrace/creating the crash log.
* Intended only for cases where you *know* that the user made a mistake, and can provide an error message which tells them exactly how to fix it. -- Not for cases where you get a residue number zero condition six levels deep and only infer that the user *probably* made a mistake "somewhere" in their setup.
* There are some changes to the exception hierarchy to support this.
Future directions are to build a system which can collect this info and summarize it (hopefully pointing out where common causes of errors in Rosetta are) and to clean up/improve our error handling and error messages so they're less obtuse.
notify author
notify list [rosetta-logs@googlegroups.com]
Merge pull request #4116 from RosettaCommons/roccomoretti/fix_valgrind
Fix Valgrind integration tests
A few minor changes to integration.valgrind to fix the current failures.
notify author
notify list [rosetta-logs@googlegroups.com]
Merge pull request #4113 from RosettaCommons/vmullig/fix_error_msg
Fix an error message.
A weird, cryptic, empty error message is troubling some users. Trying to fix.
notify author
notify list [rosetta-logs@googlegroups.com]
Merge pull request #4111 from RosettaCommons/roccomoretti/fix_features_parallel
Fix features_parallel integration test
The features_parallel integration test wasn't using the -testing:INTEGRATION_TEST flag, so we were getting random number seed tracer changes.
notify author
notify list [rosetta-logs@googlegroups.com]
Merge pull request #4097 from RosettaCommons/roccomoretti/fix_python2_mac
Fix integration test on Mac w/r/t Python2
Recent changes to integration.py assumed python2 could be found. This doesn't hold on the test server Macs.
Change things to fall back to plain python if python2 can't be found (assuming that plain python will be python2).
notify author
notify list [rosetta-logs@googlegroups.com]
Merge pull request #3946 from RosettaCommons/vmullig/rosetta_thread_manager
Implement a Rosetta thread manager
In Rosetta, we have many different levels at which we might want to launch threads. For example, a job distributor could try to carry out jobs in parallel threads. A mover might parallelize its own work. Low-level functions, like interaction graph setup for the packer, gradient vector calculation for the minimizer, or scorefunction evaluation could also be parallelized over threads. So what happens when modules at many different levels all try to spawn threads? You could end up with a nasty thread explosion. Imagine, for example, that my job distributor launches 16 threads, each of which calls a mover that launches 16 threads, each of which calls another mover that launches 16 threads, each of which calls the packer, which launches 16 threads. You'd end up with 65,536 threads all contending for hardware resources.
Another consideration is that the master/slave relationship that tends to work well for MPI communication isn't ideally suited to threads, where you might just have 4 or 8 cores available, all of which should be _doing_ work rather than _managing_ work. So we want something that ensures that many different layers of Rosetta can ask for threads, without stepping on one another's toes, _and_ we want that thing not to hog resources to monitor everything.
In addition, thread-based parallelism, unlike process-based parallelism, works best with small, finely-grained parallel tasks that are all accessing similar regions of memory, which means that a certain amount of synchronization makes sense. Asynchronous job-level parallelism might make more sense with MPI calls than with threads -- I'm not sure. I want us to be able to experiment with parallelism on many different levels, but right now, we risk stepping on one another's toes if I'm parallelizing the packer and someone else is parallelizing a mover and someone else is writing a parallel job distributor.
I'm proposing the `RosettaThreadManager` as a solution to this. Given N total threads that the user wants to run at any given time (assuming he or she has N cores on his or her system), the global `RosettaThreadManager` spins up and maintains a pool of N-1 threads. When a module wants to do something in parallel (either from the master thread or from a child thread), it bundles the function that it wants to run in threads with its arguments using `boost::bind`, then passes that function to the `RosettaThreadManager` with a request for M threads. The `RosettaThreadManager` launches that process _synchronously_ on available idle threads, including the calling thread, with a firm guarantee that 1 <= number of threads assigned to the function <= min(total threads, number requested). The function is responsible for carrying out its work, with its parallel siblings, in a threadsafe manner. At the end, the requesting thread blocks until all sibling functions terminate, the threads are released to idle status, and other work can run on them.
Based on community feedback, I have also added a basic API that accepts a vector of work to be done. The advanced API, which allows an arbitrary thread function to run concurrently, is now access-gated with a key class that has a private constructor, so that only whitelisted friend classes can use the advanced API. This forces developers _either_ to use the basic API, _or_ to justify to the community why their class should be allowed access to the advanced API.
This PR implements the basic infrastructure for this, as well as an application that shows the effects of launching threads at three different levels. In the app, a master thread runs a level 1 function in N threads, and each level 1 function runs M level 2 functions, and each level 2 function runs P level 3 functions. The level 3 functions all work together to calculate a times table (though I could just as easily have only those level 3 functions that were launched by a single level 2 function coorperate on a piece of data). When all levels return, the master thread checks that the times table was calculated correctly, in parallel.
TODO:
- [x] Write `RosettaThreadAssignmentInfo` class.
- [x] Finish `RosettaThreadManager` class.
- ~~Wait for, and spin down, all running threads on `RosettaThreadPool` destruction. (Probably not necessary given the model of ensuring that functions are run in both the calling thread and assigned threads, and the calling thread waits for assigned threads to finish before returning, but could prevent fragility in the future if the model changes.)~~ --> I realized that this might cause the app to hang when an exception is thrown, instead of exiting properly... --> Never mind. I did get this to work sensibly.
- [x] Means of getting a thread's thread ID.
- [x] Test app.
- [x] Talk to Sergey about cxx11thread-mode integration tests.
- [x] Pull request #3957 adds support for cxx11thread-mode integration tests. That pull request must be merged before this one. (This one has that one merged into it.)
- [x] Make this into a unit/integration test.
- [x] Add information to the `RosettaThreadAssignmentInfo` class about the level from which a multithreading request comes, in case we want to make decisions about thread assignments based on that in the future.
- [x] Enums for app-level, job distributor-level, mover-level, filter-level, taskop-level, resselector-level, simplemetric-level, core compenent level.
- [x] Beauty.
- [ ] Add developer docs.
- [ ] Document what `{?}` means.
- [x] Switch from `boost::bind` and `boost::function` to `std::bind` and `std::function`.
- [x] Switch to `condition_variable` instead of using a wait in a loop.
Additional changes:
- [x] Switch enum to enum class (see Sergey's comments).
- [x] Switch to crash with nonzero exit code on thread termination failure.
For a future pull request:
- Parallelize the interaction graph set-up.
- Parallelize `GeneralizedKIC`.
- Switch the `MultithreadedJobDistributor` to use this infrastructure.
- Integration test for the above three things.
notify author
notify list [rosetta-logs@googlegroups.com]
Merge pull request #4108 from RosettaCommons/jadolfbr/sm_schema
Fixes the Schema for PerResidue metrics, where it was incorrectly labeled as a string. The metrics are integration tested, so this will fix the auto-generated docs. We also explicitly say what the default is now.
notify author
notify list [rosetta-logs@googlegroups.com]
Merge pull request #4104 from RosettaCommons/vmullig/rename_scorefunction_tracer
Renaming ScoreFunction tracer to actually reflect the class name and not just the namespace.
This will unfortunately change a bunch of integration tests. It brings the tracer in line with Rosetta conventions, though.
notify author
notify list [rosetta-logs@googlegroups.com]
Merge pull request #4090 from RosettaCommons/sergey/f2
fixing integration.valgrind script so it does not fail when subtest result contain invalid unicode sequence
notify author
notify list [rosetta-logs@googlegroups.com]
Merge pull request #4086 from RosettaCommons/sergey/scientific
fixing `mhc_epitope_energy.debug` so it does not report sub-tests failures when test is passed
notify author
notify list [rosetta-logs@googlegroups.com]