run_train_pipeline.py writes a file $(Build.SourcesDirectory)/tmp.sh that is then sourced by the pipeline.

This is not explained, and not easy to understand.

Comments would help, but it would be cleaner if run_train_pipeline.py accepted an argument which is a file name to which it writes the pipeline id, and the pipeline code was changed to something like:

tmpfile=$(mktemp)
python $(Build.SourcesDirectory)/ml_service/pipelines/run_train_pipeline.py --pipeline-id "$tmpfile"
echo "##vso[task.setvariable variable=AMLPIPELINEID;isOutput=true]$(cat $tmpfile)"
rm "$tmpfile"