Hi @toufikshit
thanks for the contribution! I have a few general remarks:

• I don't think the venv folder and files contained in it belong to pycomlink
• I think Pytorch_Utils_Tensors is not a good name for the submodule. 1. you are using tensorflow and 2. it does not really describe the functionality that you want to implement
• The notebook should go to notebooks/. Also, please make sure to clear all outputs before you commit the notebook to avoid large diffs.
• What is the intention of the notebook? I see that it loads cml_wd_pytorch so I assume it is unrelated to the tensorflow code you want to implement. I suggest to remove it from the PR

Some comments for discussion tomorrow:

1. The "maximilan method" does not belong to this PR because it is not using any tensorflow code (or deep learning method). I just uses the wet-indicator time series from two different source, e.g. CNN and PC, but is not limited to using CNN output.
2. The most important functionality of this PR should be a function to call one (for now) DL wet-dry classification tensorflow model, loading the saved model and its weights from disk and later (in the course of this PR) from a URL.
3. I am undecided regarding having the model training tools in pycomlink. For now this might be the easites way to have them documented, but I assume that over time and with having more models, this would clutter the codebase of pycomlink with many specific solution for DL model training. To be discussed...

Just some brief comments. Would be good if @jpolz could have a look at the DL code.

Is it ready for testing?

@toufikshit Are there specific things I should look into in this PR or do we need to discuss anything? Or is it ready to be merged from your side?

Not that I’m aware of. I think it’s ready to be merged. It would be better to test it on Windows/Linux or on your local machine to confirm. I tested it in an HPC environment.

...maybe it is best now to go back to the last working commit locally, do some small reasonable changes and then do a "force push" to change the commit history of the branch in this PR

Done!

@jpolz Please note that the version of the branch you have is maybe now out of sync with the branch here in the PR because the force push removed the last commit that you were using. @toufikshit will now manually add back some of his updates.

Unused imports were removed and the requirements.txt was updated. The GPU issue still; on Christian’s machine, my local setup, HPC and on Google Colab the code runs. The failure on Julius’s local envrionment and HPC environment appears to be due to an incompatibility or misconfiguration in the CUDA/cuDNN setup, this could be dealt with in an other pull request, I think we merge for now.

Unused imports were removed and the requirements.txt was updated. The GPU issue still; on Christian’s machine, my local setup, HPC and on Google Colab the code runs. The failure on Julius’s local envrionment and HPC environment appears to be due to an incompatibility or misconfiguration in the CUDA/cuDNN setup, this could be dealt with in an other pull request, I think we merge for now.

Can you give proper instructions on how to install everything? there are cudnn compatibility issues with newer versions that you get when doing this:

mamba env create -f environment_dev.yml -n pycomlink-tf
mamba install tensorflow
pip install -e ./

I get cudnn version 9 and this model only runs with version 8

I get cudnn version 9 and this model only runs with version 8

Just to know for the future. Is this a common problem that "older" models will not run with newer cudnn versions? How long till we run into the same problem which led to removing all tensorflow models in #151?

I get cudnn version 9 and this model only runs with version 8

Just to know for the future. Is this a common problem that "older" models will not run with newer cudnn versions? How long till we run into the same problem which led to removing all tensorflow models in #151?

I have no clue.

! pip install 'tensorflow[and-cuda]'
import tensorflow as tf
print("TF version:", tf.version)
print("CUDA version:", tf.sysconfig.get_build_info()["cuda_version"])
print("cuDNN version:", tf.sysconfig.get_build_info()["cudnn_version"])

TF version: 2.20.0
CUDA version: 12.5.1
cuDNN version: 9

I read about compiled version with XLA and without, maybe this is the cause here because apparently it runs with cudnn v9 for @toufikshit

Anyway, we can merge it, but it seems we will get a problem with this at some point in the future.

Can we merge now? Please add a 👍

TF version: 2.19.0
CUDA version: 12.5.1
cuDNN version: 9

this is the default version on google colab and it runs.

TF version: 2.19.0 CUDA version: 12.5.1 cuDNN version: 9

this is the default version on google colab and it runs.

I was not able to install this combination with conda on our hpc