Hmm, is this the error that Kris posted in a screenshot? If so, that appears to be from Quarto. Here is an older issue about it: quarto-dev/quarto-cli#9262 . Could we not solve it on the Quarto side? First check Quarto versions, then see if there are more recent mentions of the error, etc.

In any case, I don't think .env.example is ever meant to be loaded into the application even as a fallback. It's for documentation only. The idea is that everyone should copy/rename it and fill it out (if the env vars are not coming from the shared file). Also, here it overrides the real env vars coming from the real env files.

She "fixed" it by renaming to .env. I can't remember how I fixed my issue, but I also had an issue because it was not able to load/use the env variables within the code, even if I didn't need the env for what I was doing. Do you have another solution to keep that for documentation but also will fix those errors?

I think what Kris did (renaming to .env) is how .env.example is supposed to be used.

That said, based on the Quarto issue I don't think this is intended behaviour from Quarto, so it would be nice to find where that is going wrong. Do you get it by running just run-all without a .env file? I haven't been able to recreate it with the latest Quarto version and with a fresh repo clone.

If your issue was different, I'm not sure about that without knowing the details.

I used "fixed" in quotes because having to create an .env isn't a solution. It might be that her Quarto version is older. My issue came when trying to run pytask builds without an .env file present.

But I honestly don't even remember what I did to fix it, so 🤷 I'll try to recreate the issue with a fresh clone.

Okay, well, I think for the Quarto issue we should see if it can be recreated with the latest Quarto (I couldn't do it).

For the build.py issue, isn't that expected? It is hitting the API, so it tries to load the env vars.

No, if I'm running the code locally, I don't want the API key outside of GDK, nor do I want to pull any data. So some pytask build tasks (e.g. processing the metadata) should work without needing to be in GDK ☺️

Got it to now fail because API key is wrong, not because it doesn't exist, which is what I was aiming for. We could probably do something clever and have an env var for remote vs local to bypass those API calls.

We can make some pytask tasks conditional on whether the API keys are there, but first I think it would be helpful to describe how exactly we want the pipeline to work on GenomeDK vs locally. For example, we decided not to commit the downloaded metadata, so that will not be available locally and so the processing task won't be able to run even if the download task is skipped.

If we eventually refactor the metadata processing and form splitting tasks to depend on raw-to-resource-map.csv, then we get around this metadata downloading problem. We will still have the same situation with downloading data, but that can't be processed locally anyway.

We can make some pytask tasks conditional on whether the API keys are there, but first I think it would be helpful to describe how exactly we want the pipeline to work on GenomeDK vs locally. For example, we decided not to commit the downloaded metadata, so that will not be available locally and so the processing task won't be able to run even if the download task is skipped.

If we eventually refactor the metadata processing and form splitting tasks to depend on raw-to-resource-map.csv, then we get around this metadata downloading problem. We will still have the same situation with downloading data, but that can't be processed locally anyway.

Sorry, but I still don't see which problem is solved by this. I still think Kris' Quarto problem should be solved by looking at Quarto and it's not clear to me how this would solve it (does this code run before the Quarto command?).

As for the build pipeline, this makes the API keys exist, but they are still not correct, so as you said in the other comment, the download tasks still won't run.