Split T900_distributed.py module into separate sub-tests (#648)
This PR accomplishes several things:
1. Splits the `PyRosettaCluster` unit tests specified in the
`T900_distributed` module into separate modules (`T901`–`T921`). Due to
parallelism from `self-test.py`, tests seem to run faster overall.
2. Splits the `PyRosettaCluster` unit test modules into separate modules
(as discussed in PR #630)
3. Provides a streaming and non-streaming version of a
`pyrosetta.distributed` framework unit test runner; all `T900`-series
tests use the legacy non-streaming `subprocess.getstatusoutput` function
by default.
4. Micro-improvements to `PyRosettaCluster` such as better error
messages, and temporary directory cleanup during exceptions.
5. Stabilizes the `TestReproducibilityRemodelTaskUpdates` test case by:
_(i)_ serializing double precision Python `float` values in Rosetta
command-line options for precise round-trips through Rosetta C++ double
precision `core::Real` values, and _(ii)_ serializing `PackedPose`
objects via PDB strings for updating task dictionaries between PyRosetta
protocols.
6. Adds Python-3.8 compatibility in the `TestReproducibilityTaskUpdates`
test case: `random.randbytes(14)` -> `random.getrandbits(14 *
8).to_bytes(14, "little")`
7. Stabilizes the `WorkerPreemptionTest` test case by setting
`_sleep_time=8`, thereby allowing the Dask nanny more time to recover
after Dask worker preemption. More time is needed now that heavy
`T900`-series tests are parallelized across forked children in
`self-test.py`, without which cluster shutdown can sporadically hang.
I've also added `cluster.close(timeout=3)` so that test teardown cannot
hang even if 8 seconds is not enough time.