Merge qsub remodel - #250

@hhentschke, I have just sent a PR to your repo, which fixes a few small (unrelated) issues. After these fixes, I can approve this PR.

I still have some issues with parpool but only when the engine is also submitted as a job and aaq_parpool is a sub-job. When I run the engine on the login node, then it works and is, indeed, faster than batch probably due to the full (in)dependency check.

Thanks, @hhentschke. I am happy with this PR.
How about you, @jones-michael-s?

I've tested matlab_pct and it runs fine here (it seems faster than before -- is that possible?).

What I'd like to see is some documentation. @tiborauer suggested I try out the aaq_parpool option, but I have no idea how to do that.

I suggest looking at the README. Based on my experience, you do not have to do anything but set the aap.options.wheretoprocess accordingly. As usual, the value should be the engine name without 'aaq' (e.g. 'parpool' for 'aaq_parpool').

Ok, I have run the parpool version here. No crash and is wicked fast!

A minor complaint is that we don't get module info messages echoed to the command window. I assume this is the nature of the beast? Meh, I can live with it.

However, I'm still going to complain about documentation. The readme is a start, but it's not "documentation." And while I'm sure operation is obvious to the people who worked on the code, it isn't to the rest of us. I didn't even know how to select the new processing option. But also: does the new PCT use the same aap settings? Do I need to create a new local pool profile. Apparently not -- the analysis ran -- but what about optimization? What are the hardware requirements/recommendations? What versions of matlab are supported?

We should adopt the rule that if you write (or rewrite) a substantial piece of the code, then you need to write (or rewrite) documentation for it as part of the PR. Who better qualified to write the documentation than the developer?

Maybe this is in the works for the parallel project, but I checked the wiki and it doesn't seem to have been updated.

In summary: New PCT code works great, go ahead and merge but needs documentation.

Hi @jones-michael-s, I hear what you say and am with you on the need for more documentation for aa and especially for the cluster integration. However, this documentation that you miss is for aa(q) in general and not specific for this PR.
The new aaq engines work the same way as the previous ones and require the same hardware/software configuration/optimization. And there is only a slight change in MATLAB compatibility because parpool and parfeval requires R2103b rather than R2013a as opposed to what the website says; which I can rectify easily.
Only HTCondor requires MATLAB b2020b, which is also indicated. HTCondor finally works again and with 'batch' only (which is due to a fundamental property of HTCondor no supporting communicating jobs in general). aaq_condor should be, perhaps, removed, because it might confuse users.

The README clearly describes aaq_batch and aaq_parpool with their most relevant features and some recommendation. It also indicates that they are there to replace aaq_qsub and aaq_matlab_pct, respectively; which are also still available as "legacy" versions.
Long story short, no matter whether you used aaq_qsub or aaq_matlab_pct, the README gives you a one-to-one mapping without any further need to change anything.