Merge pull request #3880 from RosettaCommons/vmullig/tensorflow_link
Add ability to link Tensorflow's C libraries, to allow evaluation of feedforward neural networks on CPU or GPU
Note that this is _not_ related to (or intended to compete with) the work being done to evaluate the Rosetta scoring function on the GPU using Torch. This is intended to allow evaluation within Rosetta of Tensorflow neural networks that were previously trained outside of Rosetta, for machine learning-related applications.
Tasks:
- [x] Add headers for C api,
- [x] Add linker command for linking the prebuilt CPU or GPU Tensorflow binaries.
- [x] Ensure that `extras=tensorflow` defines `USE_TENSORFLOW`.
- ~~It might be good to _also_ have `extras=tensorflow` define `TENSORFLOW_CPU`, and `extras=tensorflow_gpu` define `TENSORFLOW_GPU`. I can conceive of situations in which a Rosetta developer might want to bundle work differently if it's going to be evaluated on a CPU _vs._ on a GPU.~~ --> There's not actually a good way to know which TensorFlow library is being linked. It just depends on the `LD_LIBRARY_PATH` environment variable. The API is the same for both versions.
- [x] Add basic pilot app to test successful linking.
- [x] Add pilot apps to test functionality.
- [x] Integration test.
- [x] Add tensorflow integration test flavour.
- [x] Add build test for `extras=tensorflow`. (Note that `extras=tensorflow_gpu` will not be tested until we get GPU nodes on the test server.)
- [x] refactoring `integration.py` and deprecate `--mpi-tests` in favor of more general `--suffix` option
- [x] Switch pilot app output to use tracer.
Down the road we'll add a good Rosetta-side interface for Tensorflow calculations, but first we want to do some experiments and see what works and what doesn't. So for now, our pilot apps will be making Tensorflow function calls directly.
@lyskov @JackMaguire
Note: I confirmed with Dennis Hanson at UW Comotion that this is kosher, given the Rosetta and Tensorflow licences.