Adding dask security to PyRosettaCluster (#531)
A primary feature of `PyRosettaCluster` is that arbitrary user-provided
PyRosetta protocols are pickled, sent over a network, and unpickled,
which allows the user to run customized macromolecular design and
modeling workflows. If the user is operating `PyRosettaCluster` behind a
trusted private network segment (i.e., a firewall), the current
implementation is already secure from external threats (such as
eavesdropping, tampering or impersonation). However, in cases of running
`PyRosettaCluster` without a truly isolated and trusted environment, the
`dask` library can be configured to use TLS/SSL communication between
network endpoints for authenticated and encrypted transmission of data.
This PR aims to integrate Dask's TLS/SSL communication into
`PyRosettaCluster`, as well as implement a few additional security
measures:
1. Adds a `security` keyword argument to `PyRosettaCluster`, which can
accept a `dask.distributed.Security()` object. Alternatively, it accepts
a `bool` object, where if `True` we use the `cryptography` package
through the `dask` and `dask-jobqueue` APIs to generate a temporary
`dask.distributed.Security()` object for the simulation. Because
`PyRosettaCluster` supports remote dask worker instantiation via the
`dask-jobqueue` module, security is now enabled by default for the use
of remote clusters (such as `SLURMCluster`), and thus this PR adds
[cryptography](https://pypi.org/project/cryptography/) as a required
package for the `pyrosetta.distributed` framework (note that there are
very few `cryptography` dependencies, only including `cffi`, and
`openssl` which already ships with standard Python installations).
2. Adds a `pyrosetta.distributed.cluster.generate_dask_tls_security()`
function, which uses the OpenSSL executable that ships with standard
Python installations (due to the native python `ssl` library) to
generate a pre-configured `dask.distributed.Security()` object with the
necessary key/certificate pairs.
3. Enables Hash-based Message Authentication Code (HMAC)-SHA256
verification of `cloudpickle`d data (including the arbitrary
user-provided PyRosetta protocols and task `kwargs`) between network
endpoints (including the host node process, each dask worker process,
and the `billiard` subprocesses; i.e., client ↔ worker, client ↔
subprocess), where the cryptographic pseudo-random key is sent to dask
workers out-of-band using a dask worker plugin.
4. Adds nonce caching on the host node process and all worker processes
if security is disabled, with a `max_nonce` keyword argument that allows
setting the maximum nonce cache size in each process. Nonces are unique
keys added to each distributed message over the network (see the
`cryptography` package
[Glossary](https://cryptography.io/en/latest/glossary/) for more
information), where if the same nonce is encountered twice in the nonce
cache, it may indicate a replay attack and the simulation is
intentionally terminated for security reasons. Note that nonce caching
is disabled if dask security is already enabled, since the nonce caches
may add several additional MB of memory per process (which is not much).