Branch: rosetta:commits 「revision: №20787」
Test: ubuntu.gcc.cxx11thread.serialization.python310.PyRosetta.unit
SubTest: T900_distributed
SubTest files: 「file-system-view」
Daemon: takeshi    
State: T900_distributed

sys.version_info: sys.version_info(major=3, minor=10, micro=10, releaselevel='final', serial=0) Executing: /home/benchmark/prefix/takeshi/ubuntu/python_virtual_environments/python-3.10/5a2e1c043c063f7fd7b81cc951011c5f/bin/python -m unittest pyrosetta.tests.bindings.core.test_pose Output: ./home/benchmark/rosetta/source/build/PyRosetta/Linux-5.4.0-84-generic-x86_64-with-glibc2.27/gcc-7.5.0/python-3.10/minsizerel.serialization.thread/build/pyrosetta/tests/bindings/core/test_pose.py:30: DeprecationWarning: Pose.__len__ is deprecated, prefer 'pose.residues.__len__'. self.assertEqual(8, len(pose1)) # Deprecated /home/benchmark/rosetta/source/build/PyRosetta/Linux-5.4.0-84-generic-x86_64-with-glibc2.27/gcc-7.5.0/python-3.10/minsizerel.serialization.thread/build/pyrosetta/tests/bindings/core/test_pose.py:35: DeprecationWarning: Pose.__iter__ is deprecated, prefer 'pose.residues.__iter__'. self.assertEqual(8, len(list(pose1))) # Deprecated /home/benchmark/rosetta/source/build/PyRosetta/Linux-5.4.0-84-generic-x86_64-with-glibc2.27/gcc-7.5.0/python-3.10/minsizerel.serialization.thread/build/pyrosetta/tests/bindings/core/test_pose.py:35: DeprecationWarning: Pose.__len__ is deprecated, prefer 'pose.residues.__len__'. self.assertEqual(8, len(list(pose1))) # Deprecated /home/benchmark/rosetta/source/build/PyRosetta/Linux-5.4.0-84-generic-x86_64-with-glibc2.27/gcc-7.5.0/python-3.10/minsizerel.serialization.thread/build/pyrosetta/tests/bindings/core/test_pose.py:67: DeprecationWarning: Pose.__getitem__ is deprecated, prefer 'pose.residues.__getitem__'. ''.join([res.annotated_name() for res in pose1[1:9]]), 'A[ALA:NtermProteinFull]CDEFGHI[ILE:CtermProteinFull]') # Deprecated /home/benchmark/rosetta/source/build/PyRosetta/Linux-5.4.0-84-generic-x86_64-with-glibc2.27/gcc-7.5.0/python-3.10/minsizerel.serialization.thread/build/pyrosetta/tests/bindings/core/test_pose.py:69: DeprecationWarning: Pose.__getitem__ is deprecated, prefer 'pose.residues.__getitem__'. ''.join([res.annotated_name() for res in pose1[-6:8]]), 'DEFGH') # Deprecated ../home/benchmark/rosetta/source/build/PyRosetta/Linux-5.4.0-84-generic-x86_64-with-glibc2.27/gcc-7.5.0/python-3.10/minsizerel.serialization.thread/build/pyrosetta/bindings/energies.py:166: DeprecationWarning: Conversion of an array with ndim > 0 to a scalar is deprecated, and will error in future. Ensure you extract a single element from your array before performing this operation. (Deprecated NumPy 1.25.) return { n : float(total_array[n]) for n in total_array.dtype.names } .. ---------------------------------------------------------------------- Ran 5 tests in 4.902s OK ┌──────────────────────────────────────────────────────────────────────────────┐ │ PyRosetta-4 │ │ Created in JHU by Sergey Lyskov and PyRosetta Team │ │ (C) Copyright Rosetta Commons Member Institutions │ │ │ │ NOTE: USE OF PyRosetta FOR COMMERCIAL PURPOSES REQUIRE PURCHASE OF A LICENSE │ │ See LICENSE.PyRosetta.md or email license@uw.edu for details │ └──────────────────────────────────────────────────────────────────────────────┘ PyRosetta-4 2025 [Rosetta devel 2025.23.post.dev+4.HEAD.8f929e5d110b1396a0d36e8bf9238a8c0dbc9118 2025-06-03T12:24:00] retrieved from: git@github.com:RosettaCommons/rosetta.git Executing: /home/benchmark/prefix/takeshi/ubuntu/python_virtual_environments/python-3.10/5a2e1c043c063f7fd7b81cc951011c5f/bin/python -m unittest pyrosetta.tests.distributed.cluster.test_logging Output: WARNING:root:`conda env export --prefix /home/benchmark/prefix/takeshi/ubuntu/python_virtual_environments/python-3.10/5a2e1c043c063f7fd7b81cc951011c5f` did not run successfully, so the active conda environment YML file string was not saved! It is recommended to run: `conda env export --prefix /home/benchmark/prefix/takeshi/ubuntu/python_virtual_environments/python-3.10/5a2e1c043c063f7fd7b81cc951011c5f > environment.yml` to reproduce this simulation later. Warning: Use of `pyrosetta.distributed.cluster` namespace requires Anaconda (or Miniconda) to be properly installed for reproducibility of PyRosetta simulations. Please install Anaconda (or Miniconda) onto your system to enable running `which conda`. For installation instructions, visit: https://docs.anaconda.com/anaconda/install Warning: Use of `pyrosetta.distributed.cluster` namespace requires Anaconda (or Miniconda) to be properly installed for reproducibility of PyRosetta simulations. Please install Anaconda (or Miniconda) onto your system to enable running `which conda`. For installation instructions, visit: https://docs.anaconda.com/anaconda/install . ---------------------------------------------------------------------- Ran 1 test in 24.010s OK Executing: /home/benchmark/prefix/takeshi/ubuntu/python_virtual_environments/python-3.10/5a2e1c043c063f7fd7b81cc951011c5f/bin/python -m unittest pyrosetta.tests.distributed.cluster.test_reproducibility Output: WARNING:root:`conda env export --prefix /home/benchmark/prefix/takeshi/ubuntu/python_virtual_environments/python-3.10/5a2e1c043c063f7fd7b81cc951011c5f` did not run successfully, so the active conda environment YML file string was not saved! It is recommended to run: `conda env export --prefix /home/benchmark/prefix/takeshi/ubuntu/python_virtual_environments/python-3.10/5a2e1c043c063f7fd7b81cc951011c5f > environment.yml` to reproduce this simulation later. Warning: Use of `pyrosetta.distributed.cluster` namespace requires Anaconda (or Miniconda) to be properly installed for reproducibility of PyRosetta simulations. Please install Anaconda (or Miniconda) onto your system to enable running `which conda`. For installation instructions, visit: https://docs.anaconda.com/anaconda/install Warning: Use of `pyrosetta.distributed.cluster` namespace requires Anaconda (or Miniconda) to be properly installed for reproducibility of PyRosetta simulations. Please install Anaconda (or Miniconda) onto your system to enable running `which conda`. For installation instructions, visit: https://docs.anaconda.com/anaconda/install /home/benchmark/rosetta/source/build/PyRosetta/Linux-5.4.0-84-generic-x86_64-with-glibc2.27/gcc-7.5.0/python-3.10/minsizerel.serialization.thread/build/pyrosetta/bindings/energies.py:166: DeprecationWarning: Conversion of an array with ndim > 0 to a scalar is deprecated, and will error in future. Ensure you extract a single element from your array before performing this operation. (Deprecated NumPy 1.25.) return { n : float(total_array[n]) for n in total_array.dtype.names } .WARNING:root:`conda env export --prefix /home/benchmark/prefix/takeshi/ubuntu/python_virtual_environments/python-3.10/5a2e1c043c063f7fd7b81cc951011c5f` did not run successfully, so the active conda environment YML file string was not saved! It is recommended to run: `conda env export --prefix /home/benchmark/prefix/takeshi/ubuntu/python_virtual_environments/python-3.10/5a2e1c043c063f7fd7b81cc951011c5f > environment.yml` to reproduce this simulation later. Warning: Use of `pyrosetta.distributed.cluster` namespace requires Anaconda (or Miniconda) to be properly installed for reproducibility of PyRosetta simulations. Please install Anaconda (or Miniconda) onto your system to enable running `which conda`. For installation instructions, visit: https://docs.anaconda.com/anaconda/install /home/benchmark/rosetta/source/build/PyRosetta/Linux-5.4.0-84-generic-x86_64-with-glibc2.27/gcc-7.5.0/python-3.10/minsizerel.serialization.thread/build/pyrosetta/bindings/energies.py:166: DeprecationWarning: Conversion of an array with ndim > 0 to a scalar is deprecated, and will error in future. Ensure you extract a single element from your array before performing this operation. (Deprecated NumPy 1.25.) return { n : float(total_array[n]) for n in total_array.dtype.names } .WARNING:root:`conda env export --prefix /home/benchmark/prefix/takeshi/ubuntu/python_virtual_environments/python-3.10/5a2e1c043c063f7fd7b81cc951011c5f` did not run successfully, so the active conda environment YML file string was not saved! It is recommended to run: `conda env export --prefix /home/benchmark/prefix/takeshi/ubuntu/python_virtual_environments/python-3.10/5a2e1c043c063f7fd7b81cc951011c5f > environment.yml` to reproduce this simulation later. Warning: Use of `pyrosetta.distributed.cluster` namespace requires Anaconda (or Miniconda) to be properly installed for reproducibility of PyRosetta simulations. Please install Anaconda (or Miniconda) onto your system to enable running `which conda`. For installation instructions, visit: https://docs.anaconda.com/anaconda/install /home/benchmark/rosetta/source/build/PyRosetta/Linux-5.4.0-84-generic-x86_64-with-glibc2.27/gcc-7.5.0/python-3.10/minsizerel.serialization.thread/build/pyrosetta/bindings/energies.py:166: DeprecationWarning: Conversion of an array with ndim > 0 to a scalar is deprecated, and will error in future. Ensure you extract a single element from your array before performing this operation. (Deprecated NumPy 1.25.) return { n : float(total_array[n]) for n in total_array.dtype.names } WARNING:root:`conda env export --prefix /home/benchmark/prefix/takeshi/ubuntu/python_virtual_environments/python-3.10/5a2e1c043c063f7fd7b81cc951011c5f` did not run successfully, so the active conda environment YML file string was not saved! It is recommended to run: `conda env export --prefix /home/benchmark/prefix/takeshi/ubuntu/python_virtual_environments/python-3.10/5a2e1c043c063f7fd7b81cc951011c5f > environment.yml` to reproduce this simulation later. Warning: Use of `pyrosetta.distributed.cluster` namespace requires Anaconda (or Miniconda) to be properly installed for reproducibility of PyRosetta simulations. Please install Anaconda (or Miniconda) onto your system to enable running `which conda`. For installation instructions, visit: https://docs.anaconda.com/anaconda/install /home/benchmark/rosetta/source/build/PyRosetta/Linux-5.4.0-84-generic-x86_64-with-glibc2.27/gcc-7.5.0/python-3.10/minsizerel.serialization.thread/build/pyrosetta/bindings/energies.py:166: DeprecationWarning: Conversion of an array with ndim > 0 to a scalar is deprecated, and will error in future. Ensure you extract a single element from your array before performing this operation. (Deprecated NumPy 1.25.) return { n : float(total_array[n]) for n in total_array.dtype.names } .WARNING:root:`conda env export --prefix /home/benchmark/prefix/takeshi/ubuntu/python_virtual_environments/python-3.10/5a2e1c043c063f7fd7b81cc951011c5f` did not run successfully, so the active conda environment YML file string was not saved! It is recommended to run: `conda env export --prefix /home/benchmark/prefix/takeshi/ubuntu/python_virtual_environments/python-3.10/5a2e1c043c063f7fd7b81cc951011c5f > environment.yml` to reproduce this simulation later. Warning: Use of `pyrosetta.distributed.cluster` namespace requires Anaconda (or Miniconda) to be properly installed for reproducibility of PyRosetta simulations. Please install Anaconda (or Miniconda) onto your system to enable running `which conda`. For installation instructions, visit: https://docs.anaconda.com/anaconda/install /home/benchmark/rosetta/source/build/PyRosetta/Linux-5.4.0-84-generic-x86_64-with-glibc2.27/gcc-7.5.0/python-3.10/minsizerel.serialization.thread/build/pyrosetta/bindings/energies.py:166: DeprecationWarning: Conversion of an array with ndim > 0 to a scalar is deprecated, and will error in future. Ensure you extract a single element from your array before performing this operation. (Deprecated NumPy 1.25.) return { n : float(total_array[n]) for n in total_array.dtype.names } WARNING:root:The input 'environment' parameter argument is an empty string, which is not a valid YML file string capturing the active conda environment! Reproduction simulations may not necessarily reproduce the original decoy(s)! Please verify that your active conda environment is identical to the original conda environment that generated the decoy(s) you wish to reproduce! Bypassing conda environment validation... Warning: Use of `pyrosetta.distributed.cluster` namespace requires Anaconda (or Miniconda) to be properly installed for reproducibility of PyRosetta simulations. Please install Anaconda (or Miniconda) onto your system to enable running `which conda`. For installation instructions, visit: https://docs.anaconda.com/anaconda/install /home/benchmark/rosetta/source/build/PyRosetta/Linux-5.4.0-84-generic-x86_64-with-glibc2.27/gcc-7.5.0/python-3.10/minsizerel.serialization.thread/build/pyrosetta/bindings/energies.py:166: DeprecationWarning: Conversion of an array with ndim > 0 to a scalar is deprecated, and will error in future. Ensure you extract a single element from your array before performing this operation. (Deprecated NumPy 1.25.) return { n : float(total_array[n]) for n in total_array.dtype.names } WARNING:root:The input 'environment' parameter argument is an empty string, which is not a valid YML file string capturing the active conda environment! Reproduction simulations may not necessarily reproduce the original decoy(s)! Please verify that your active conda environment is identical to the original conda environment that generated the decoy(s) you wish to reproduce! Bypassing conda environment validation... Warning: Use of `pyrosetta.distributed.cluster` namespace requires Anaconda (or Miniconda) to be properly installed for reproducibility of PyRosetta simulations. Please install Anaconda (or Miniconda) onto your system to enable running `which conda`. For installation instructions, visit: https://docs.anaconda.com/anaconda/install /home/benchmark/rosetta/source/build/PyRosetta/Linux-5.4.0-84-generic-x86_64-with-glibc2.27/gcc-7.5.0/python-3.10/minsizerel.serialization.thread/build/pyrosetta/bindings/energies.py:166: DeprecationWarning: Conversion of an array with ndim > 0 to a scalar is deprecated, and will error in future. Ensure you extract a single element from your array before performing this operation. (Deprecated NumPy 1.25.) return { n : float(total_array[n]) for n in total_array.dtype.names } .WARNING:root:`conda env export --prefix /home/benchmark/prefix/takeshi/ubuntu/python_virtual_environments/python-3.10/5a2e1c043c063f7fd7b81cc951011c5f` did not run successfully, so the active conda environment YML file string was not saved! It is recommended to run: `conda env export --prefix /home/benchmark/prefix/takeshi/ubuntu/python_virtual_environments/python-3.10/5a2e1c043c063f7fd7b81cc951011c5f > environment.yml` to reproduce this simulation later. Warning: Use of `pyrosetta.distributed.cluster` namespace requires Anaconda (or Miniconda) to be properly installed for reproducibility of PyRosetta simulations. Please install Anaconda (or Miniconda) onto your system to enable running `which conda`. For installation instructions, visit: https://docs.anaconda.com/anaconda/install /home/benchmark/rosetta/source/build/PyRosetta/Linux-5.4.0-84-generic-x86_64-with-glibc2.27/gcc-7.5.0/python-3.10/minsizerel.serialization.thread/build/pyrosetta/bindings/energies.py:166: DeprecationWarning: Conversion of an array with ndim > 0 to a scalar is deprecated, and will error in future. Ensure you extract a single element from your array before performing this operation. (Deprecated NumPy 1.25.) return { n : float(total_array[n]) for n in total_array.dtype.names } .WARNING:root:`conda env export --prefix /home/benchmark/prefix/takeshi/ubuntu/python_virtual_environments/python-3.10/5a2e1c043c063f7fd7b81cc951011c5f` did not run successfully, so the active conda environment YML file string was not saved! It is recommended to run: `conda env export --prefix /home/benchmark/prefix/takeshi/ubuntu/python_virtual_environments/python-3.10/5a2e1c043c063f7fd7b81cc951011c5f > environment.yml` to reproduce this simulation later. Warning: Use of `pyrosetta.distributed.cluster` namespace requires Anaconda (or Miniconda) to be properly installed for reproducibility of PyRosetta simulations. Please install Anaconda (or Miniconda) onto your system to enable running `which conda`. For installation instructions, visit: https://docs.anaconda.com/anaconda/install /home/benchmark/rosetta/source/build/PyRosetta/Linux-5.4.0-84-generic-x86_64-with-glibc2.27/gcc-7.5.0/python-3.10/minsizerel.serialization.thread/build/pyrosetta/bindings/energies.py:166: DeprecationWarning: Conversion of an array with ndim > 0 to a scalar is deprecated, and will error in future. Ensure you extract a single element from your array before performing this operation. (Deprecated NumPy 1.25.) return { n : float(total_array[n]) for n in total_array.dtype.names } . ---------------------------------------------------------------------- Ran 6 tests in 305.669s OK Executing: /home/benchmark/prefix/takeshi/ubuntu/python_virtual_environments/python-3.10/5a2e1c043c063f7fd7b81cc951011c5f/bin/python -m unittest pyrosetta.tests.distributed.cluster.test_smoke Output: Warning: Use of `pyrosetta.distributed.cluster` namespace requires Anaconda (or Miniconda) to be properly installed for reproducibility of PyRosetta simulations. Please install Anaconda (or Miniconda) onto your system to enable running `which conda`. For installation instructions, visit: https://docs.anaconda.com/anaconda/install WARNING:root:`conda env export --prefix /home/benchmark/prefix/takeshi/ubuntu/python_virtual_environments/python-3.10/5a2e1c043c063f7fd7b81cc951011c5f` did not run successfully, so the active conda environment YML file string was not saved! It is recommended to run: `conda env export --prefix /home/benchmark/prefix/takeshi/ubuntu/python_virtual_environments/python-3.10/5a2e1c043c063f7fd7b81cc951011c5f > environment.yml` to reproduce this simulation later. Warning: Use of `pyrosetta.distributed.cluster` namespace requires Anaconda (or Miniconda) to be properly installed for reproducibility of PyRosetta simulations. Please install Anaconda (or Miniconda) onto your system to enable running `which conda`. For installation instructions, visit: https://docs.anaconda.com/anaconda/install Warning: Use of `pyrosetta.distributed.cluster` namespace requires Anaconda (or Miniconda) to be properly installed for reproducibility of PyRosetta simulations. Please install Anaconda (or Miniconda) onto your system to enable running `which conda`. For installation instructions, visit: https://docs.anaconda.com/anaconda/install .WARNING:root:`conda env export --prefix /home/benchmark/prefix/takeshi/ubuntu/python_virtual_environments/python-3.10/5a2e1c043c063f7fd7b81cc951011c5f` did not run successfully, so the active conda environment YML file string was not saved! It is recommended to run: `conda env export --prefix /home/benchmark/prefix/takeshi/ubuntu/python_virtual_environments/python-3.10/5a2e1c043c063f7fd7b81cc951011c5f > environment.yml` to reproduce this simulation later. Warning: Use of `pyrosetta.distributed.cluster` namespace requires Anaconda (or Miniconda) to be properly installed for reproducibility of PyRosetta simulations. Please install Anaconda (or Miniconda) onto your system to enable running `which conda`. For installation instructions, visit: https://docs.anaconda.com/anaconda/install .WARNING:root:`conda env export --prefix /home/benchmark/prefix/takeshi/ubuntu/python_virtual_environments/python-3.10/5a2e1c043c063f7fd7b81cc951011c5f` did not run successfully, so the active conda environment YML file string was not saved! It is recommended to run: `conda env export --prefix /home/benchmark/prefix/takeshi/ubuntu/python_virtual_environments/python-3.10/5a2e1c043c063f7fd7b81cc951011c5f > environment.yml` to reproduce this simulation later. Warning: Use of `pyrosetta.distributed.cluster` namespace requires Anaconda (or Miniconda) to be properly installed for reproducibility of PyRosetta simulations. Please install Anaconda (or Miniconda) onto your system to enable running `which conda`. For installation instructions, visit: https://docs.anaconda.com/anaconda/install Warning: Use of `pyrosetta.distributed.cluster` namespace requires Anaconda (or Miniconda) to be properly installed for reproducibility of PyRosetta simulations. Please install Anaconda (or Miniconda) onto your system to enable running `which conda`. For installation instructions, visit: https://docs.anaconda.com/anaconda/install .WARNING:root:`conda env export --prefix /home/benchmark/prefix/takeshi/ubuntu/python_virtual_environments/python-3.10/5a2e1c043c063f7fd7b81cc951011c5f` did not run successfully, so the active conda environment YML file string was not saved! It is recommended to run: `conda env export --prefix /home/benchmark/prefix/takeshi/ubuntu/python_virtual_environments/python-3.10/5a2e1c043c063f7fd7b81cc951011c5f > environment.yml` to reproduce this simulation later. /home/benchmark/prefix/takeshi/ubuntu/python_virtual_environments/python-3.10/5a2e1c043c063f7fd7b81cc951011c5f/lib/python3.10/site-packages/distributed/node.py:187: UserWarning: Port 8787 is already in use. Perhaps you already have a cluster running? Hosting the HTTP server on port 35859 instead warnings.warn( Warning: Use of `pyrosetta.distributed.cluster` namespace requires Anaconda (or Miniconda) to be properly installed for reproducibility of PyRosetta simulations. Please install Anaconda (or Miniconda) onto your system to enable running `which conda`. For installation instructions, visit: https://docs.anaconda.com/anaconda/install ./home/benchmark/prefix/takeshi/ubuntu/python_virtual_environments/python-3.10/5a2e1c043c063f7fd7b81cc951011c5f/lib/python3.10/site-packages/distributed/node.py:187: UserWarning: Port 8787 is already in use. Perhaps you already have a cluster running? Hosting the HTTP server on port 46757 instead warnings.warn( Warning: Use of `pyrosetta.distributed.cluster` namespace requires Anaconda (or Miniconda) to be properly installed for reproducibility of PyRosetta simulations. Please install Anaconda (or Miniconda) onto your system to enable running `which conda`. For installation instructions, visit: https://docs.anaconda.com/anaconda/install 2025-06-03 16:44:38,813 - distributed.scheduler - WARNING - Removing worker 'tcp://127.0.0.1:34911' caused the cluster to lose already computed task(s), which will be recomputed elsewhere: {'user_spawn_thread-dcbe594e-032a-4f4d-9f0a-b6f63701b3b9'} (stimulus_id='handle-worker-cleanup-1748983478.8132377') 2025-06-03 16:44:38,813 - distributed.scheduler - ERROR - Removing worker 'tcp://127.0.0.1:34911' caused the cluster to lose scattered data, which can't be recovered: {'NoneType-975258884365472b91d82271ceb3e20f', 'str-44c0b2d1b5c04f6a81bc41e89bcb942e', 'float-6f1eb897defb4352ab8de0fe7bcf4e45', 'bytes-523a6e84a7f54c95a0a092f7dec5988a', 'str-b236e890c40f49d59bda945e868989b5', 'bool-5bda227e5cdd4949bcd0a51adb5fb48f', 'bool-513998b1d399457eaa1735be7e2464d4', 'str-d7f8512fb26647bf9e2d6289d9d9ee2c', 'set-25b70642378243bd8ef836ba0ab6df1c', 'str-e26180c5648e477d9b7470e79f5eec3d', 'float-f3143a35088b4d6cb972f7388ea733af', 'dict-b07c950c4ea94e88ba4b77857bb671b1', 'function-0c059eb5ed174c9aa6969a3c2396e51b', 'bytes-554bb554ad7d44c1878a815fd727b74f'} (stimulus_id='handle-worker-cleanup-1748983478.8132377') ../home/benchmark/prefix/takeshi/ubuntu/python_virtual_environments/python-3.10/5a2e1c043c063f7fd7b81cc951011c5f/lib/python3.10/site-packages/distributed/node.py:187: UserWarning: Port 8787 is already in use. Perhaps you already have a cluster running? Hosting the HTTP server on port 44865 instead warnings.warn( Warning: Use of `pyrosetta.distributed.cluster` namespace requires Anaconda (or Miniconda) to be properly installed for reproducibility of PyRosetta simulations. Please install Anaconda (or Miniconda) onto your system to enable running `which conda`. For installation instructions, visit: https://docs.anaconda.com/anaconda/install WARNING:root:`conda env export --prefix /home/benchmark/prefix/takeshi/ubuntu/python_virtual_environments/python-3.10/5a2e1c043c063f7fd7b81cc951011c5f` did not run successfully, so the active conda environment YML file string was not saved! It is recommended to run: `conda env export --prefix /home/benchmark/prefix/takeshi/ubuntu/python_virtual_environments/python-3.10/5a2e1c043c063f7fd7b81cc951011c5f > environment.yml` to reproduce this simulation later. /home/benchmark/prefix/takeshi/ubuntu/python_virtual_environments/python-3.10/5a2e1c043c063f7fd7b81cc951011c5f/lib/python3.10/site-packages/distributed/node.py:187: UserWarning: Port 8787 is already in use. Perhaps you already have a cluster running? Hosting the HTTP server on port 41149 instead warnings.warn( Warning: Use of `pyrosetta.distributed.cluster` namespace requires Anaconda (or Miniconda) to be properly installed for reproducibility of PyRosetta simulations. Please install Anaconda (or Miniconda) onto your system to enable running `which conda`. For installation instructions, visit: https://docs.anaconda.com/anaconda/install Process Process-1: Traceback (most recent call last): File "/home/benchmark/prefix/takeshi/ubuntu/python_virtual_environments/python-3.10/5a2e1c043c063f7fd7b81cc951011c5f/lib/python3.10/site-packages/billiard/process.py", line 323, in _bootstrap self.run() File "/home/benchmark/prefix/takeshi/ubuntu/python_virtual_environments/python-3.10/5a2e1c043c063f7fd7b81cc951011c5f/lib/python3.10/site-packages/billiard/process.py", line 110, in run self._target(*self._args, **self._kwargs) File "/home/benchmark/rosetta/source/build/PyRosetta/Linux-5.4.0-84-generic-x86_64-with-glibc2.27/gcc-7.5.0/python-3.10/minsizerel.serialization.thread/build/pyrosetta/distributed/cluster/logging_support.py", line 108, in wrapper result = func( File "/home/benchmark/rosetta/source/build/PyRosetta/Linux-5.4.0-84-generic-x86_64-with-glibc2.27/gcc-7.5.0/python-3.10/minsizerel.serialization.thread/build/pyrosetta/distributed/__init__.py", line 119, in fwrap return func(*args, **kwargs) File "/home/benchmark/rosetta/source/build/PyRosetta/Linux-5.4.0-84-generic-x86_64-with-glibc2.27/gcc-7.5.0/python-3.10/minsizerel.serialization.thread/build/pyrosetta/distributed/cluster/multiprocessing.py", line 150, in target results = run_protocol( File "/home/benchmark/rosetta/source/build/PyRosetta/Linux-5.4.0-84-generic-x86_64-with-glibc2.27/gcc-7.5.0/python-3.10/minsizerel.serialization.thread/build/pyrosetta/distributed/cluster/base.py", line 224, in wrapper return func( File "/home/benchmark/rosetta/source/build/PyRosetta/Linux-5.4.0-84-generic-x86_64-with-glibc2.27/gcc-7.5.0/python-3.10/minsizerel.serialization.thread/build/pyrosetta/distributed/cluster/multiprocessing.py", line 100, in run_protocol result = user_protocol(packed_pose, protocol, ignore_errors, **kwargs) File "/home/benchmark/rosetta/source/build/PyRosetta/Linux-5.4.0-84-generic-x86_64-with-glibc2.27/gcc-7.5.0/python-3.10/minsizerel.serialization.thread/build/pyrosetta/distributed/cluster/exceptions.py", line 116, in wrapper result = func(compressed_packed_pose, protocol, ignore_errors, **kwargs) File "/home/benchmark/rosetta/source/build/PyRosetta/Linux-5.4.0-84-generic-x86_64-with-glibc2.27/gcc-7.5.0/python-3.10/minsizerel.serialization.thread/build/pyrosetta/distributed/cluster/multiprocessing.py", line 82, in user_protocol result = protocol(packed_pose, **kwargs) File "/home/benchmark/rosetta/source/build/PyRosetta/Linux-5.4.0-84-generic-x86_64-with-glibc2.27/gcc-7.5.0/python-3.10/minsizerel.serialization.thread/build/pyrosetta/tests/distributed/cluster/test_smoke.py", line 144, in protocol_with_error raise NotImplementedError("Testing an error in a user-provided PyRosetta protocol.") NotImplementedError: Testing an error in a user-provided PyRosetta protocol. 2025-06-03 16:44:55,167 - distributed.worker - ERROR - Compute Failed Key: user_spawn_thread-5619b7ab-2281-46a8-8061-f32c43036e10 State: executing Task: <Task 'user_spawn_thread-5619b7ab-2281-46a8-8061-f32c43036e10' user_spawn_thread(...)> Exception: 'WorkerError("Worker thread killed due to an error or segmentation fault encountered in the user-provided PyRosetta protocol \'protocol_with_error\'.")' Traceback: ' File "/home/benchmark/rosetta/source/build/PyRosetta/Linux-5.4.0-84-generic-x86_64-with-glibc2.27/gcc-7.5.0/python-3.10/minsizerel.serialization.thread/build/pyrosetta/distributed/cluster/multiprocessing.py", line 200, in user_spawn_thread\n results = get_target_results_kwargs(\n File "/home/benchmark/rosetta/source/build/PyRosetta/Linux-5.4.0-84-generic-x86_64-with-glibc2.27/gcc-7.5.0/python-3.10/minsizerel.serialization.thread/build/pyrosetta/distributed/cluster/exceptions.py", line 152, in wrapper\n raise WorkerError(protocol_name)\n' ./home/benchmark/rosetta/source/build/PyRosetta/Linux-5.4.0-84-generic-x86_64-with-glibc2.27/gcc-7.5.0/python-3.10/minsizerel.serialization.thread/build/pyrosetta/distributed/cluster/logging_support.py:41: ResourceWarning: unclosed file <_io.TextIOWrapper name='/tmp/tmpvd91y18i/outputs/logs/PyRosettaCluster.log' mode='a' encoding='UTF-8'> logger.handlers = [] ResourceWarning: Enable tracemalloc to get the object allocation traceback /home/benchmark/prefix/takeshi/ubuntu/python_virtual_environments/python-3.10/5a2e1c043c063f7fd7b81cc951011c5f/lib/python3.10/site-packages/distributed/node.py:187: UserWarning: Port 8787 is already in use. Perhaps you already have a cluster running? Hosting the HTTP server on port 40795 instead warnings.warn( Warning: Use of `pyrosetta.distributed.cluster` namespace requires Anaconda (or Miniconda) to be properly installed for reproducibility of PyRosetta simulations. Please install Anaconda (or Miniconda) onto your system to enable running `which conda`. For installation instructions, visit: https://docs.anaconda.com/anaconda/install WARNING:root:`conda env export --prefix /home/benchmark/prefix/takeshi/ubuntu/python_virtual_environments/python-3.10/5a2e1c043c063f7fd7b81cc951011c5f` did not run successfully, so the active conda environment YML file string was not saved! It is recommended to run: `conda env export --prefix /home/benchmark/prefix/takeshi/ubuntu/python_virtual_environments/python-3.10/5a2e1c043c063f7fd7b81cc951011c5f > environment.yml` to reproduce this simulation later. /home/benchmark/prefix/takeshi/ubuntu/python_virtual_environments/python-3.10/5a2e1c043c063f7fd7b81cc951011c5f/lib/python3.10/site-packages/distributed/node.py:187: UserWarning: Port 8787 is already in use. Perhaps you already have a cluster running? Hosting the HTTP server on port 33439 instead warnings.warn( Warning: Use of `pyrosetta.distributed.cluster` namespace requires Anaconda (or Miniconda) to be properly installed for reproducibility of PyRosetta simulations. Please install Anaconda (or Miniconda) onto your system to enable running `which conda`. For installation instructions, visit: https://docs.anaconda.com/anaconda/install WARNING:root:`conda env export --prefix /home/benchmark/prefix/takeshi/ubuntu/python_virtual_environments/python-3.10/5a2e1c043c063f7fd7b81cc951011c5f` did not run successfully, so the active conda environment YML file string was not saved! It is recommended to run: `conda env export --prefix /home/benchmark/prefix/takeshi/ubuntu/python_virtual_environments/python-3.10/5a2e1c043c063f7fd7b81cc951011c5f > environment.yml` to reproduce this simulation later. /home/benchmark/prefix/takeshi/ubuntu/python_virtual_environments/python-3.10/5a2e1c043c063f7fd7b81cc951011c5f/lib/python3.10/site-packages/distributed/node.py:187: UserWarning: Port 8787 is already in use. Perhaps you already have a cluster running? Hosting the HTTP server on port 36069 instead warnings.warn( Warning: Use of `pyrosetta.distributed.cluster` namespace requires Anaconda (or Miniconda) to be properly installed for reproducibility of PyRosetta simulations. Please install Anaconda (or Miniconda) onto your system to enable running `which conda`. For installation instructions, visit: https://docs.anaconda.com/anaconda/install .WARNING:root:`conda env export --prefix /home/benchmark/prefix/takeshi/ubuntu/python_virtual_environments/python-3.10/5a2e1c043c063f7fd7b81cc951011c5f` did not run successfully, so the active conda environment YML file string was not saved! It is recommended to run: `conda env export --prefix /home/benchmark/prefix/takeshi/ubuntu/python_virtual_environments/python-3.10/5a2e1c043c063f7fd7b81cc951011c5f > environment.yml` to reproduce this simulation later. /home/benchmark/prefix/takeshi/ubuntu/python_virtual_environments/python-3.10/5a2e1c043c063f7fd7b81cc951011c5f/lib/python3.10/site-packages/distributed/node.py:187: UserWarning: Port 8787 is already in use. Perhaps you already have a cluster running? Hosting the HTTP server on port 43851 instead warnings.warn( Warning: Use of `pyrosetta.distributed.cluster` namespace requires Anaconda (or Miniconda) to be properly installed for reproducibility of PyRosetta simulations. Please install Anaconda (or Miniconda) onto your system to enable running `which conda`. For installation instructions, visit: https://docs.anaconda.com/anaconda/install 2025-06-03 16:50:20,102 - distributed.scheduler - WARNING - Removing worker 'tcp://127.0.0.1:39563' caused the cluster to lose already computed task(s), which will be recomputed elsewhere: {'user_spawn_thread-f286b08b-91fb-411e-836e-b6b23a4c005c'} (stimulus_id='handle-worker-cleanup-1748983820.1025147') 2025-06-03 16:50:20,102 - distributed.scheduler - ERROR - Removing worker 'tcp://127.0.0.1:39563' caused the cluster to lose scattered data, which can't be recovered: {'bytes-c83138139ead4d52999c11b3edc5019b', 'str-06eeaca5308f43c896e4151e47fcdb49', 'set-9fb65a981c1c4337bd181c73bd46d3ea', 'str-6d4df74b861349058d71d0a27df905e2', 'bytes-53c1f58b9ce248ebafdc6ac6d6c00856', 'dict-4ca98744db914319b2eaed0305dd9141', 'float-4902d558c4594acb9951972005ec7e2b', 'str-7771a3e357234588b04176dba00f3041', 'float-1ba4f4d966ec4b68b5cca1b91c38564c', 'str-14d2229b4ef44efb8f50c67415630f0d', 'bool-854c755685bb4b81886c8a12745e4fee', 'NoneType-ca67240bd83c4cef9551f7509584171a', 'bool-958b9ce3d8f04b1f9ca3fbd4696f3f74', 'function-90a27882266a4c9eaf5af70101f2bf9e'} (stimulus_id='handle-worker-cleanup-1748983820.1025147') .WARNING:root:`conda env export --prefix /home/benchmark/prefix/takeshi/ubuntu/python_virtual_environments/python-3.10/5a2e1c043c063f7fd7b81cc951011c5f` did not run successfully, so the active conda environment YML file string was not saved! It is recommended to run: `conda env export --prefix /home/benchmark/prefix/takeshi/ubuntu/python_virtual_environments/python-3.10/5a2e1c043c063f7fd7b81cc951011c5f > environment.yml` to reproduce this simulation later. /home/benchmark/prefix/takeshi/ubuntu/python_virtual_environments/python-3.10/5a2e1c043c063f7fd7b81cc951011c5f/lib/python3.10/site-packages/distributed/node.py:187: UserWarning: Port 8787 is already in use. Perhaps you already have a cluster running? Hosting the HTTP server on port 40441 instead warnings.warn( Warning: Use of `pyrosetta.distributed.cluster` namespace requires Anaconda (or Miniconda) to be properly installed for reproducibility of PyRosetta simulations. Please install Anaconda (or Miniconda) onto your system to enable running `which conda`. For installation instructions, visit: https://docs.anaconda.com/anaconda/install Warning: Use of `pyrosetta.distributed.cluster` namespace requires Anaconda (or Miniconda) to be properly installed for reproducibility of PyRosetta simulations. Please install Anaconda (or Miniconda) onto your system to enable running `which conda`. For installation instructions, visit: https://docs.anaconda.com/anaconda/install Warning: Use of `pyrosetta.distributed.cluster` namespace requires Anaconda (or Miniconda) to be properly installed for reproducibility of PyRosetta simulations. Please install Anaconda (or Miniconda) onto your system to enable running `which conda`. For installation instructions, visit: https://docs.anaconda.com/anaconda/install Warning: Use of `pyrosetta.distributed.cluster` namespace requires Anaconda (or Miniconda) to be properly installed for reproducibility of PyRosetta simulations. Please install Anaconda (or Miniconda) onto your system to enable running `which conda`. For installation instructions, visit: https://docs.anaconda.com/anaconda/install Warning: Use of `pyrosetta.distributed.cluster` namespace requires Anaconda (or Miniconda) to be properly installed for reproducibility of PyRosetta simulations. Please install Anaconda (or Miniconda) onto your system to enable running `which conda`. For installation instructions, visit: https://docs.anaconda.com/anaconda/install WARNING:root:`conda env export --prefix /home/benchmark/prefix/takeshi/ubuntu/python_virtual_environments/python-3.10/5a2e1c043c063f7fd7b81cc951011c5f` did not run successfully, so the active conda environment YML file string was not saved! It is recommended to run: `conda env export --prefix /home/benchmark/prefix/takeshi/ubuntu/python_virtual_environments/python-3.10/5a2e1c043c063f7fd7b81cc951011c5f > environment.yml` to reproduce this simulation later. /home/benchmark/prefix/takeshi/ubuntu/python_virtual_environments/python-3.10/5a2e1c043c063f7fd7b81cc951011c5f/lib/python3.10/site-packages/distributed/node.py:187: UserWarning: Port 8787 is already in use. Perhaps you already have a cluster running? Hosting the HTTP server on port 42541 instead warnings.warn( Warning: Use of `pyrosetta.distributed.cluster` namespace requires Anaconda (or Miniconda) to be properly installed for reproducibility of PyRosetta simulations. Please install Anaconda (or Miniconda) onto your system to enable running `which conda`. For installation instructions, visit: https://docs.anaconda.com/anaconda/install . ---------------------------------------------------------------------- Ran 10 tests in 542.643s OK 2025-06-03 16:51:07,934 - distributed.diskutils - ERROR - Failed to remove '/tmp/tmp_i65qrmp/dask-scratch-space/worker-dqb_ev2y' (failed in <built-in function lstat>): [Errno 2] No such file or directory: '/tmp/tmp_i65qrmp/dask-scratch-space/worker-dqb_ev2y' 2025-06-03 16:51:08,815 - distributed.scheduler - INFO - Retire worker addresses (stimulus_id='retire-workers-1748983868.8159406') (0,) 2025-06-03 16:51:08,816 - distributed.nanny - INFO - Closing Nanny at 'tcp://127.0.0.1:34961'. Reason: nanny-close 2025-06-03 16:51:08,816 - distributed.nanny - INFO - Nanny asking worker to close. Reason: nanny-close 2025-06-03 16:51:08,819 - distributed.scheduler - INFO - Remove client Client-worker-58d0209c-40bb-11f0-bb07-2b66feaf7c23 2025-06-03 16:51:08,819 - distributed.core - INFO - Received 'close-stream' from tcp://127.0.0.1:43308; closing. 2025-06-03 16:51:08,820 - distributed.scheduler - INFO - Remove client Client-worker-58d0209c-40bb-11f0-bb07-2b66feaf7c23 2025-06-03 16:51:08,820 - distributed.scheduler - INFO - Close client connection: Client-worker-58d0209c-40bb-11f0-bb07-2b66feaf7c23 2025-06-03 16:51:08,821 - distributed.core - INFO - Received 'close-stream' from tcp://127.0.0.1:43286; closing. 2025-06-03 16:51:08,821 - distributed.scheduler - INFO - Remove worker addr: tcp://127.0.0.1:35605 name: 0 (stimulus_id='handle-worker-cleanup-1748983868.821722') 2025-06-03 16:51:08,822 - distributed.scheduler - INFO - Lost all workers 2025-06-03 16:51:08,823 - distributed.diskutils - ERROR - Failed to remove '/tmp/tmpqr1gu3r8/dask-scratch-space/worker-ra7uezpd' (failed in <built-in function lstat>): [Errno 2] No such file or directory: '/tmp/tmpqr1gu3r8/dask-scratch-space/worker-ra7uezpd' 2025-06-03 16:51:09,719 - distributed.nanny - INFO - Nanny at 'tcp://127.0.0.1:34961' closed. 2025-06-03 16:51:09,721 - distributed.scheduler - INFO - Closing scheduler. Reason: unknown 2025-06-03 16:51:09,722 - distributed.scheduler - INFO - Scheduler closing all comms 2025-06-03 16:51:09,728 - distributed.scheduler - INFO - Retire worker addresses (stimulus_id='retire-workers-1748983869.7283182') (0,) 2025-06-03 16:51:09,729 - distributed.nanny - INFO - Closing Nanny at 'tcp://127.0.0.1:36923'. Reason: nanny-close 2025-06-03 16:51:09,729 - distributed.nanny - INFO - Nanny asking worker to close. Reason: nanny-close 2025-06-03 16:51:09,734 - distributed.scheduler - INFO - Remove client Client-worker-3c45b269-40bb-11f0-ba97-b543afcea154 2025-06-03 16:51:09,735 - distributed.core - INFO - Received 'close-stream' from tcp://127.0.0.1:52902; closing. 2025-06-03 16:51:09,735 - distributed.scheduler - INFO - Remove client Client-worker-3c45b269-40bb-11f0-ba97-b543afcea154 2025-06-03 16:51:09,736 - distributed.scheduler - INFO - Close client connection: Client-worker-3c45b269-40bb-11f0-ba97-b543afcea154 2025-06-03 16:51:09,737 - distributed.core - INFO - Received 'close-stream' from tcp://127.0.0.1:52880; closing. 2025-06-03 16:51:09,738 - distributed.scheduler - INFO - Remove worker addr: tcp://127.0.0.1:40973 name: 0 (stimulus_id='handle-worker-cleanup-1748983869.7384036') 2025-06-03 16:51:09,738 - distributed.scheduler - INFO - Lost all workers 2025-06-03 16:51:09,739 - distributed.diskutils - ERROR - Failed to remove '/tmp/tmp_i65qrmp/dask-scratch-space/worker-_2f35clr' (failed in <built-in function lstat>): [Errno 2] No such file or directory: '/tmp/tmp_i65qrmp/dask-scratch-space/worker-_2f35clr' 2025-06-03 16:51:10,638 - distributed.nanny - INFO - Nanny at 'tcp://127.0.0.1:36923' closed. 2025-06-03 16:51:10,640 - distributed.scheduler - INFO - Closing scheduler. Reason: unknown 2025-06-03 16:51:10,641 - distributed.scheduler - INFO - Scheduler closing all comms 2025-06-03 16:51:10,646 - distributed.scheduler - INFO - Retire worker addresses (stimulus_id='retire-workers-1748983870.6468291') (0,) 2025-06-03 16:51:10,647 - distributed.nanny - INFO - Closing Nanny at 'tcp://127.0.0.1:36641'. Reason: nanny-close 2025-06-03 16:51:10,648 - distributed.nanny - INFO - Nanny asking worker to close. Reason: nanny-close 2025-06-03 16:51:10,652 - distributed.scheduler - INFO - Remove client Client-worker-4af5e7c7-40bb-11f0-badb-09ed2ab821bb 2025-06-03 16:51:10,652 - distributed.core - INFO - Received 'close-stream' from tcp://127.0.0.1:59332; closing. 2025-06-03 16:51:10,653 - distributed.scheduler - INFO - Remove client Client-worker-4af5e7c7-40bb-11f0-badb-09ed2ab821bb 2025-06-03 16:51:10,654 - distributed.scheduler - INFO - Close client connection: Client-worker-4af5e7c7-40bb-11f0-badb-09ed2ab821bb 2025-06-03 16:51:10,655 - distributed.core - INFO - Received 'close-stream' from tcp://127.0.0.1:59324; closing. 2025-06-03 16:51:10,656 - distributed.scheduler - INFO - Remove worker addr: tcp://127.0.0.1:37643 name: 0 (stimulus_id='handle-worker-cleanup-1748983870.6562257') 2025-06-03 16:51:10,656 - distributed.scheduler - INFO - Lost all workers 2025-06-03 16:51:10,656 - distributed.diskutils - ERROR - Failed to remove '/tmp/tmpsw0_sfqq/dask-scratch-space/worker-r9y3ndfc' (failed in <built-in function lstat>): [Errno 2] No such file or directory: '/tmp/tmpsw0_sfqq/dask-scratch-space/worker-r9y3ndfc' 2025-06-03 16:51:11,775 - distributed.nanny - INFO - Nanny at 'tcp://127.0.0.1:36641' closed. 2025-06-03 16:51:11,777 - distributed.scheduler - INFO - Closing scheduler. Reason: unknown 2025-06-03 16:51:11,778 - distributed.scheduler - INFO - Scheduler closing all comms 2025-06-03 16:51:11,783 - distributed.scheduler - INFO - Retire worker addresses (stimulus_id='retire-workers-1748983871.7832155') (0,) 2025-06-03 16:51:11,784 - distributed.nanny - INFO - Closing Nanny at 'tcp://127.0.0.1:33187'. Reason: nanny-close 2025-06-03 16:51:11,784 - distributed.nanny - INFO - Nanny asking worker to close. Reason: nanny-close 2025-06-03 16:51:11,789 - distributed.scheduler - INFO - Remove client Client-worker-55f0c73a-40bb-11f0-bafc-2b8232deb620 2025-06-03 16:51:11,790 - distributed.core - INFO - Received 'close-stream' from tcp://127.0.0.1:33872; closing. 2025-06-03 16:51:11,790 - distributed.scheduler - INFO - Remove client Client-worker-55f0c73a-40bb-11f0-bafc-2b8232deb620 2025-06-03 16:51:11,791 - distributed.scheduler - INFO - Close client connection: Client-worker-55f0c73a-40bb-11f0-bafc-2b8232deb620 2025-06-03 16:51:11,793 - distributed.core - INFO - Received 'close-stream' from tcp://127.0.0.1:33850; closing. 2025-06-03 16:51:11,794 - distributed.scheduler - INFO - Remove worker addr: tcp://127.0.0.1:41771 name: 0 (stimulus_id='handle-worker-cleanup-1748983871.794044') 2025-06-03 16:51:11,794 - distributed.scheduler - INFO - Lost all workers 2025-06-03 16:51:11,796 - distributed.diskutils - ERROR - Failed to remove '/tmp/tmpqr1gu3r8/dask-scratch-space/worker-kyqwc340' (failed in <built-in function lstat>): [Errno 2] No such file or directory: '/tmp/tmpqr1gu3r8/dask-scratch-space/worker-kyqwc340' 2025-06-03 16:51:12,788 - distributed.nanny - INFO - Nanny at 'tcp://127.0.0.1:33187' closed. 2025-06-03 16:51:12,789 - distributed.scheduler - INFO - Closing scheduler. Reason: unknown 2025-06-03 16:51:12,790 - distributed.scheduler - INFO - Scheduler closing all comms 2025-06-03 16:51:12,792 - distributed.scheduler - INFO - Retire worker addresses (stimulus_id='retire-workers-1748983872.7927773') (0,) 2025-06-03 16:51:12,793 - distributed.nanny - INFO - Closing Nanny at 'tcp://127.0.0.1:34109'. Reason: nanny-close 2025-06-03 16:51:12,793 - distributed.nanny - INFO - Nanny asking worker to close. Reason: nanny-close 2025-06-03 16:51:12,796 - distributed.scheduler - INFO - Remove client Client-worker-9acea4d9-40bb-11f0-bba5-5b569317d6ac 2025-06-03 16:51:12,796 - distributed.core - INFO - Received 'close-stream' from tcp://127.0.0.1:35556; closing. 2025-06-03 16:51:12,796 - distributed.scheduler - INFO - Remove client Client-worker-9acea4d9-40bb-11f0-bba5-5b569317d6ac 2025-06-03 16:51:12,796 - distributed.scheduler - INFO - Close client connection: Client-worker-9acea4d9-40bb-11f0-bba5-5b569317d6ac 2025-06-03 16:51:12,797 - distributed.core - INFO - Received 'close-stream' from tcp://127.0.0.1:35548; closing. 2025-06-03 16:51:12,798 - distributed.scheduler - INFO - Remove worker addr: tcp://127.0.0.1:41239 name: 0 (stimulus_id='handle-worker-cleanup-1748983872.7981524') 2025-06-03 16:51:12,798 - distributed.scheduler - INFO - Lost all workers 2025-06-03 16:51:12,799 - distributed.diskutils - ERROR - Failed to remove '/tmp/tmpvd91y18i/dask-scratch-space/worker-do9zwubs' (failed in <built-in function lstat>): [Errno 2] No such file or directory: '/tmp/tmpvd91y18i/dask-scratch-space/worker-do9zwubs' 2025-06-03 16:51:13,498 - distributed.nanny - INFO - Nanny at 'tcp://127.0.0.1:34109' closed. 2025-06-03 16:51:13,499 - distributed.scheduler - INFO - Closing scheduler. Reason: unknown 2025-06-03 16:51:13,500 - distributed.scheduler - INFO - Scheduler closing all comms Executing: /home/benchmark/prefix/takeshi/ubuntu/python_virtual_environments/python-3.10/5a2e1c043c063f7fd7b81cc951011c5f/bin/python -m unittest pyrosetta.tests.distributed.test_concurrency Output: ss/home/benchmark/rosetta/source/build/PyRosetta/Linux-5.4.0-84-generic-x86_64-with-glibc2.27/gcc-7.5.0/python-3.10/minsizerel.serialization.thread/build/pyrosetta/bindings/energies.py:166: DeprecationWarning: Conversion of an array with ndim > 0 to a scalar is deprecated, and will error in future. Ensure you extract a single element from your array before performing this operation. (Deprecated NumPy 1.25.) return { n : float(total_array[n]) for n in total_array.dtype.names } . ---------------------------------------------------------------------- Ran 3 tests in 4.611s OK (skipped=2) Executing: /home/benchmark/prefix/takeshi/ubuntu/python_virtual_environments/python-3.10/5a2e1c043c063f7fd7b81cc951011c5f/bin/python -m unittest pyrosetta.tests.distributed.test_dask Output: 2025-06-03 16:51:27,383 - distributed.nanny - ERROR - Failed to initialize worker Traceback (most recent call last): File "/home/benchmark/prefix/takeshi/ubuntu/python_virtual_environments/python-3.10/5a2e1c043c063f7fd7b81cc951011c5f/lib/python3.10/site-packages/distributed/nanny.py", line 956, in run worker = worker_factory() File "/home/benchmark/prefix/takeshi/ubuntu/python_virtual_environments/python-3.10/5a2e1c043c063f7fd7b81cc951011c5f/lib/python3.10/site-packages/distributed/worker.py", line 732, in __init__ ServerNode.__init__( TypeError: Server.__init__() got an unexpected keyword argument 'local_dir' 2025-06-03 16:51:27,425 - distributed.nanny - ERROR - Failed to start process Traceback (most recent call last): File "/home/benchmark/prefix/takeshi/ubuntu/python_virtual_environments/python-3.10/5a2e1c043c063f7fd7b81cc951011c5f/lib/python3.10/site-packages/distributed/nanny.py", line 452, in instantiate result = await self.process.start() File "/home/benchmark/prefix/takeshi/ubuntu/python_virtual_environments/python-3.10/5a2e1c043c063f7fd7b81cc951011c5f/lib/python3.10/site-packages/distributed/nanny.py", line 759, in start msg = await self._wait_until_connected(uid) File "/home/benchmark/prefix/takeshi/ubuntu/python_virtual_environments/python-3.10/5a2e1c043c063f7fd7b81cc951011c5f/lib/python3.10/site-packages/distributed/nanny.py", line 905, in _wait_until_connected raise msg["exception"] File "/home/benchmark/prefix/takeshi/ubuntu/python_virtual_environments/python-3.10/5a2e1c043c063f7fd7b81cc951011c5f/lib/python3.10/site-packages/distributed/nanny.py", line 956, in run worker = worker_factory() File "/home/benchmark/prefix/takeshi/ubuntu/python_virtual_environments/python-3.10/5a2e1c043c063f7fd7b81cc951011c5f/lib/python3.10/site-packages/distributed/worker.py", line 732, in __init__ ServerNode.__init__( TypeError: Server.__init__() got an unexpected keyword argument 'local_dir' 2025-06-03 16:51:27,444 - distributed.nanny - ERROR - Failed to initialize worker Traceback (most recent call last): File "/home/benchmark/prefix/takeshi/ubuntu/python_virtual_environments/python-3.10/5a2e1c043c063f7fd7b81cc951011c5f/lib/python3.10/site-packages/distributed/nanny.py", line 956, in run worker = worker_factory() File "/home/benchmark/prefix/takeshi/ubuntu/python_virtual_environments/python-3.10/5a2e1c043c063f7fd7b81cc951011c5f/lib/python3.10/site-packages/distributed/worker.py", line 732, in __init__ ServerNode.__init__( TypeError: Server.__init__() got an unexpected keyword argument 'local_dir' 2025-06-03 16:51:27,475 - distributed.nanny - ERROR - Failed to start process Traceback (most recent call last): File "/home/benchmark/prefix/takeshi/ubuntu/python_virtual_environments/python-3.10/5a2e1c043c063f7fd7b81cc951011c5f/lib/python3.10/site-packages/distributed/nanny.py", line 452, in instantiate result = await self.process.start() File "/home/benchmark/prefix/takeshi/ubuntu/python_virtual_environments/python-3.10/5a2e1c043c063f7fd7b81cc951011c5f/lib/python3.10/site-packages/distributed/nanny.py", line 759, in start msg = await self._wait_until_connected(uid) File "/home/benchmark/prefix/takeshi/ubuntu/python_virtual_environments/python-3.10/5a2e1c043c063f7fd7b81cc951011c5f/lib/python3.10/site-packages/distributed/nanny.py", line 905, in _wait_until_connected raise msg["exception"] File "/home/benchmark/prefix/takeshi/ubuntu/python_virtual_environments/python-3.10/5a2e1c043c063f7fd7b81cc951011c5f/lib/python3.10/site-packages/distributed/nanny.py", line 956, in run worker = worker_factory() File "/home/benchmark/prefix/takeshi/ubuntu/python_virtual_environments/python-3.10/5a2e1c043c063f7fd7b81cc951011c5f/lib/python3.10/site-packages/distributed/worker.py", line 732, in __init__ ServerNode.__init__( TypeError: Server.__init__() got an unexpected keyword argument 'local_dir' E2025-06-03 16:51:28,497 - distributed.nanny - ERROR - Failed to initialize worker Traceback (most recent call last): File "/home/benchmark/prefix/takeshi/ubuntu/python_virtual_environments/python-3.10/5a2e1c043c063f7fd7b81cc951011c5f/lib/python3.10/site-packages/distributed/nanny.py", line 956, in run worker = worker_factory() File "/home/benchmark/prefix/takeshi/ubuntu/python_virtual_environments/python-3.10/5a2e1c043c063f7fd7b81cc951011c5f/lib/python3.10/site-packages/distributed/worker.py", line 732, in __init__ ServerNode.__init__( TypeError: Server.__init__() got an unexpected keyword argument 'local_dir' 2025-06-03 16:51:28,508 - distributed.nanny - ERROR - Failed to initialize worker Traceback (most recent call last): File "/home/benchmark/prefix/takeshi/ubuntu/python_virtual_environments/python-3.10/5a2e1c043c063f7fd7b81cc951011c5f/lib/python3.10/site-packages/distributed/nanny.py", line 956, in run worker = worker_factory() File "/home/benchmark/prefix/takeshi/ubuntu/python_virtual_environments/python-3.10/5a2e1c043c063f7fd7b81cc951011c5f/lib/python3.10/site-packages/distributed/worker.py", line 732, in __init__ ServerNode.__init__( TypeError: Server.__init__() got an unexpected keyword argument 'local_dir' 2025-06-03 16:51:28,509 - distributed.nanny - ERROR - Failed to start process Traceback (most recent call last): File "/home/benchmark/prefix/takeshi/ubuntu/python_virtual_environments/python-3.10/5a2e1c043c063f7fd7b81cc951011c5f/lib/python3.10/site-packages/distributed/nanny.py", line 452, in instantiate result = await self.process.start() File "/home/benchmark/prefix/takeshi/ubuntu/python_virtual_environments/python-3.10/5a2e1c043c063f7fd7b81cc951011c5f/lib/python3.10/site-packages/distributed/nanny.py", line 759, in start msg = await self._wait_until_connected(uid) File "/home/benchmark/prefix/takeshi/ubuntu/python_virtual_environments/python-3.10/5a2e1c043c063f7fd7b81cc951011c5f/lib/python3.10/site-packages/distributed/nanny.py", line 905, in _wait_until_connected raise msg["exception"] File "/home/benchmark/prefix/takeshi/ubuntu/python_virtual_environments/python-3.10/5a2e1c043c063f7fd7b81cc951011c5f/lib/python3.10/site-packages/distributed/nanny.py", line 956, in run worker = worker_factory() File "/home/benchmark/prefix/takeshi/ubuntu/python_virtual_environments/python-3.10/5a2e1c043c063f7fd7b81cc951011c5f/lib/python3.10/site-packages/distributed/worker.py", line 732, in __init__ ServerNode.__init__( TypeError: Server.__init__() got an unexpected keyword argument 'local_dir' 2025-06-03 16:51:28,511 - distributed.nanny - ERROR - Failed to start process Traceback (most recent call last): File "/home/benchmark/prefix/takeshi/ubuntu/python_virtual_environments/python-3.10/5a2e1c043c063f7fd7b81cc951011c5f/lib/python3.10/site-packages/distributed/nanny.py", line 452, in instantiate result = await self.process.start() File "/home/benchmark/prefix/takeshi/ubuntu/python_virtual_environments/python-3.10/5a2e1c043c063f7fd7b81cc951011c5f/lib/python3.10/site-packages/distributed/nanny.py", line 759, in start msg = await self._wait_until_connected(uid) File "/home/benchmark/prefix/takeshi/ubuntu/python_virtual_environments/python-3.10/5a2e1c043c063f7fd7b81cc951011c5f/lib/python3.10/site-packages/distributed/nanny.py", line 905, in _wait_until_connected raise msg["exception"] File "/home/benchmark/prefix/takeshi/ubuntu/python_virtual_environments/python-3.10/5a2e1c043c063f7fd7b81cc951011c5f/lib/python3.10/site-packages/distributed/nanny.py", line 956, in run worker = worker_factory() File "/home/benchmark/prefix/takeshi/ubuntu/python_virtual_environments/python-3.10/5a2e1c043c063f7fd7b81cc951011c5f/lib/python3.10/site-packages/distributed/worker.py", line 732, in __init__ ServerNode.__init__( TypeError: Server.__init__() got an unexpected keyword argument 'local_dir' E ====================================================================== ERROR: test_basic (pyrosetta.tests.distributed.test_dask.TestDaskDistribution) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/benchmark/prefix/takeshi/ubuntu/python_virtual_environments/python-3.10/5a2e1c043c063f7fd7b81cc951011c5f/lib/python3.10/site-packages/distributed/core.py", line 528, in start await wait_for(self.start_unsafe(), timeout=timeout) File "/home/benchmark/prefix/takeshi/ubuntu/python_virtual_environments/python-3.10/5a2e1c043c063f7fd7b81cc951011c5f/lib/python3.10/site-packages/distributed/utils.py", line 1928, in wait_for return await asyncio.wait_for(fut, timeout) File "/home/benchmark/prefix/takeshi/ubuntu/python-3.10.gcc/69716d1907a38bef273dea4dd5921890/lib/python3.10/asyncio/tasks.py", line 408, in wait_for return await fut File "/home/benchmark/prefix/takeshi/ubuntu/python_virtual_environments/python-3.10/5a2e1c043c063f7fd7b81cc951011c5f/lib/python3.10/site-packages/distributed/nanny.py", line 369, in start_unsafe response = await self.instantiate() File "/home/benchmark/prefix/takeshi/ubuntu/python_virtual_environments/python-3.10/5a2e1c043c063f7fd7b81cc951011c5f/lib/python3.10/site-packages/distributed/nanny.py", line 452, in instantiate result = await self.process.start() File "/home/benchmark/prefix/takeshi/ubuntu/python_virtual_environments/python-3.10/5a2e1c043c063f7fd7b81cc951011c5f/lib/python3.10/site-packages/distributed/nanny.py", line 759, in start msg = await self._wait_until_connected(uid) File "/home/benchmark/prefix/takeshi/ubuntu/python_virtual_environments/python-3.10/5a2e1c043c063f7fd7b81cc951011c5f/lib/python3.10/site-packages/distributed/nanny.py", line 905, in _wait_until_connected raise msg["exception"] File "/home/benchmark/prefix/takeshi/ubuntu/python_virtual_environments/python-3.10/5a2e1c043c063f7fd7b81cc951011c5f/lib/python3.10/site-packages/distributed/nanny.py", line 956, in run worker = worker_factory() File "/home/benchmark/prefix/takeshi/ubuntu/python_virtual_environments/python-3.10/5a2e1c043c063f7fd7b81cc951011c5f/lib/python3.10/site-packages/distributed/worker.py", line 732, in __init__ ServerNode.__init__( TypeError: Server.__init__() got an unexpected keyword argument 'local_dir' The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/home/benchmark/rosetta/source/build/PyRosetta/Linux-5.4.0-84-generic-x86_64-with-glibc2.27/gcc-7.5.0/python-3.10/minsizerel.serialization.thread/build/pyrosetta/tests/distributed/test_dask.py", line 28, in setUp self.local_cluster = dask.distributed.LocalCluster( File "/home/benchmark/prefix/takeshi/ubuntu/python_virtual_environments/python-3.10/5a2e1c043c063f7fd7b81cc951011c5f/lib/python3.10/site-packages/distributed/deploy/local.py", line 256, in __init__ super().__init__( File "/home/benchmark/prefix/takeshi/ubuntu/python_virtual_environments/python-3.10/5a2e1c043c063f7fd7b81cc951011c5f/lib/python3.10/site-packages/distributed/deploy/spec.py", line 292, in __init__ self.sync(self._correct_state) File "/home/benchmark/prefix/takeshi/ubuntu/python_virtual_environments/python-3.10/5a2e1c043c063f7fd7b81cc951011c5f/lib/python3.10/site-packages/distributed/utils.py", line 376, in sync return sync( File "/home/benchmark/prefix/takeshi/ubuntu/python_virtual_environments/python-3.10/5a2e1c043c063f7fd7b81cc951011c5f/lib/python3.10/site-packages/distributed/utils.py", line 452, in sync raise error File "/home/benchmark/prefix/takeshi/ubuntu/python_virtual_environments/python-3.10/5a2e1c043c063f7fd7b81cc951011c5f/lib/python3.10/site-packages/distributed/utils.py", line 426, in f result = yield future File "/home/benchmark/prefix/takeshi/ubuntu/python_virtual_environments/python-3.10/5a2e1c043c063f7fd7b81cc951011c5f/lib/python3.10/site-packages/tornado/gen.py", line 769, in run value = future.result() File "/home/benchmark/prefix/takeshi/ubuntu/python_virtual_environments/python-3.10/5a2e1c043c063f7fd7b81cc951011c5f/lib/python3.10/site-packages/distributed/deploy/spec.py", line 396, in _correct_state_internal await asyncio.gather(*worker_futs) File "/home/benchmark/prefix/takeshi/ubuntu/python-3.10.gcc/69716d1907a38bef273dea4dd5921890/lib/python3.10/asyncio/tasks.py", line 650, in _wrap_awaitable return (yield from awaitable.__await__()) File "/home/benchmark/prefix/takeshi/ubuntu/python_virtual_environments/python-3.10/5a2e1c043c063f7fd7b81cc951011c5f/lib/python3.10/site-packages/distributed/core.py", line 536, in start raise RuntimeError(f"{type(self).__name__} failed to start.") from exc RuntimeError: Nanny failed to start. ====================================================================== ERROR: test_rosetta_scripts (pyrosetta.tests.distributed.test_dask.TestDaskDistribution) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/benchmark/prefix/takeshi/ubuntu/python_virtual_environments/python-3.10/5a2e1c043c063f7fd7b81cc951011c5f/lib/python3.10/site-packages/distributed/core.py", line 528, in start await wait_for(self.start_unsafe(), timeout=timeout) File "/home/benchmark/prefix/takeshi/ubuntu/python_virtual_environments/python-3.10/5a2e1c043c063f7fd7b81cc951011c5f/lib/python3.10/site-packages/distributed/utils.py", line 1928, in wait_for return await asyncio.wait_for(fut, timeout) File "/home/benchmark/prefix/takeshi/ubuntu/python-3.10.gcc/69716d1907a38bef273dea4dd5921890/lib/python3.10/asyncio/tasks.py", line 408, in wait_for return await fut File "/home/benchmark/prefix/takeshi/ubuntu/python_virtual_environments/python-3.10/5a2e1c043c063f7fd7b81cc951011c5f/lib/python3.10/site-packages/distributed/nanny.py", line 369, in start_unsafe response = await self.instantiate() File "/home/benchmark/prefix/takeshi/ubuntu/python_virtual_environments/python-3.10/5a2e1c043c063f7fd7b81cc951011c5f/lib/python3.10/site-packages/distributed/nanny.py", line 452, in instantiate result = await self.process.start() File "/home/benchmark/prefix/takeshi/ubuntu/python_virtual_environments/python-3.10/5a2e1c043c063f7fd7b81cc951011c5f/lib/python3.10/site-packages/distributed/nanny.py", line 759, in start msg = await self._wait_until_connected(uid) File "/home/benchmark/prefix/takeshi/ubuntu/python_virtual_environments/python-3.10/5a2e1c043c063f7fd7b81cc951011c5f/lib/python3.10/site-packages/distributed/nanny.py", line 905, in _wait_until_connected raise msg["exception"] File "/home/benchmark/prefix/takeshi/ubuntu/python_virtual_environments/python-3.10/5a2e1c043c063f7fd7b81cc951011c5f/lib/python3.10/site-packages/distributed/nanny.py", line 956, in run worker = worker_factory() File "/home/benchmark/prefix/takeshi/ubuntu/python_virtual_environments/python-3.10/5a2e1c043c063f7fd7b81cc951011c5f/lib/python3.10/site-packages/distributed/worker.py", line 732, in __init__ ServerNode.__init__( TypeError: Server.__init__() got an unexpected keyword argument 'local_dir' The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/home/benchmark/rosetta/source/build/PyRosetta/Linux-5.4.0-84-generic-x86_64-with-glibc2.27/gcc-7.5.0/python-3.10/minsizerel.serialization.thread/build/pyrosetta/tests/distributed/test_dask.py", line 28, in setUp self.local_cluster = dask.distributed.LocalCluster( File "/home/benchmark/prefix/takeshi/ubuntu/python_virtual_environments/python-3.10/5a2e1c043c063f7fd7b81cc951011c5f/lib/python3.10/site-packages/distributed/deploy/local.py", line 256, in __init__ super().__init__( File "/home/benchmark/prefix/takeshi/ubuntu/python_virtual_environments/python-3.10/5a2e1c043c063f7fd7b81cc951011c5f/lib/python3.10/site-packages/distributed/deploy/spec.py", line 292, in __init__ self.sync(self._correct_state) File "/home/benchmark/prefix/takeshi/ubuntu/python_virtual_environments/python-3.10/5a2e1c043c063f7fd7b81cc951011c5f/lib/python3.10/site-packages/distributed/utils.py", line 376, in sync return sync( File "/home/benchmark/prefix/takeshi/ubuntu/python_virtual_environments/python-3.10/5a2e1c043c063f7fd7b81cc951011c5f/lib/python3.10/site-packages/distributed/utils.py", line 452, in sync raise error File "/home/benchmark/prefix/takeshi/ubuntu/python_virtual_environments/python-3.10/5a2e1c043c063f7fd7b81cc951011c5f/lib/python3.10/site-packages/distributed/utils.py", line 426, in f result = yield future File "/home/benchmark/prefix/takeshi/ubuntu/python_virtual_environments/python-3.10/5a2e1c043c063f7fd7b81cc951011c5f/lib/python3.10/site-packages/tornado/gen.py", line 769, in run value = future.result() File "/home/benchmark/prefix/takeshi/ubuntu/python_virtual_environments/python-3.10/5a2e1c043c063f7fd7b81cc951011c5f/lib/python3.10/site-packages/distributed/deploy/spec.py", line 396, in _correct_state_internal await asyncio.gather(*worker_futs) File "/home/benchmark/prefix/takeshi/ubuntu/python-3.10.gcc/69716d1907a38bef273dea4dd5921890/lib/python3.10/asyncio/tasks.py", line 650, in _wrap_awaitable return (yield from awaitable.__await__()) File "/home/benchmark/prefix/takeshi/ubuntu/python_virtual_environments/python-3.10/5a2e1c043c063f7fd7b81cc951011c5f/lib/python3.10/site-packages/distributed/core.py", line 536, in start raise RuntimeError(f"{type(self).__name__} failed to start.") from exc RuntimeError: Nanny failed to start. ---------------------------------------------------------------------- Ran 2 tests in 2.062s FAILED (errors=2) Encountered error(s) with exit code 1 while running: /home/benchmark/prefix/takeshi/ubuntu/python_virtual_environments/python-3.10/5a2e1c043c063f7fd7b81cc951011c5f/bin/python -m unittest pyrosetta.tests.distributed.test_dask Terminating... Finished T900_distributed in 0:15:18.356167