aacache, toolboxes - #213

FieldTrip has a nasty problem of shadowing functions in the Matlab signal processing toolbox (cf. ./spm12/external/fieldtrip/external/signal) and some of their versions are inconsistent with the Mathworks implementation. (I've actually removed FieldTrip from my SPM install because of this).

Just curious: would your Toolbox fix change this behavior?

M

The solution is that you can load an unload toolboxes (such as FieldTrip) from the path easily.

Apart from SPM, the other toolboxes are not pre-loaded (i.e. not added to the path during aa_init), but they are rather loaded and unloaded within the modules if needed.

Happy to combine this with some work on my fork and run the test suite, once we have clarity on #214.

Ok, I've dropped MPM. Now crashing on fieldmap2vdm:

MODULE aamod_fieldmap2VDM RUNNING: Create VDMs in fieldmaps for /imaging/jc01/aa/aa_demo/fmri/aamod_fieldmap2VDM_00001/CBU090973/Loc
Loaded datecheck was d2+E6JDeV1oRt4x4QIbs3Q== and calc d2+E6JDeV1oRt4x4QIbs3Q==
Loaded datecheck was p75ermqZkFxi/uWnGtdK9w== and calc p75ermqZkFxi/uWnGtdK9w==
Loaded datecheck was 6024EQNM9k5aktKcxl5O/Q== and calc 6024EQNM9k5aktKcxl5O/Q==
Loaded datecheck was InZI+fz2+Ola6Q5CKPaGaA== and calc InZI+fz2+Ola6Q5CKPaGaA==
 retrieve stream epi [checksum match, not recopied] from /imaging/jc01/aa/aa_demo/fmri/aamod_convert_epis_00001/CBU090973/Loc to /imaging/jc01/aa/aa_demo/fmri/aamod_fieldmap2VDM_0000
1/CBU090973/Loc
 retrieve stream epi_dicom_header [checksum match, not recopied] from /imaging/jc01/aa/aa_demo/fmri/aamod_convert_epis_00001/CBU090973/Loc to /imaging/jc01/aa/aa_demo/fmri/aamod_fiel
dmap2VDM_00001/CBU090973/Loc
 retrieve stream fieldmap [checksum match, not recopied] from /imaging/jc01/aa/aa_demo/fmri/aamod_convert_fieldmaps_00001/CBU090973/Loc to /imaging/jc01/aa/aa_demo/fmri/aamod_fieldma
p2VDM_00001/CBU090973/Loc
 retrieve stream fieldmap_dicom_header [checksum match, not recopied] from /imaging/jc01/aa/aa_demo/fmri/aamod_convert_fieldmaps_00001/CBU090973/Loc to /imaging/jc01/aa/aa_demo/fmri/
aamod_fieldmap2VDM_00001/CBU090973/Loc
 executing
Load stream from file /imaging/jc01/aa/aa_demo/fmri/aamod_fieldmap2VDM_00001/CBU090973/Loc/stream_fieldmap_inputto_aamod_fieldmap2VDM_00001.txt...
Load stream from file /imaging/jc01/aa/aa_demo/fmri/aamod_fieldmap2VDM_00001/CBU090973/Loc/stream_epi_inputto_aamod_fieldmap2VDM_00001.txt...
Load stream from file /imaging/jc01/aa/aa_demo/fmri/aamod_fieldmap2VDM_00001/CBU090973/Loc/stream_epi_dicom_header_inputto_aamod_fieldmap2VDM_00001.txt...
Load stream from file /imaging/jc01/aa/aa_demo/fmri/aamod_fieldmap2VDM_00001/CBU090973/Loc/stream_fieldmap_dicom_header_inputto_aamod_fieldmap2VDM_00001.txt...
WARNING: Error during retrieving Fieldmap Echo Times: Reference to non-existent field 'EchoTime1'.
WARNING: Manual settings are used:  ms and


**** automatic analysis failed - see reason and line numbers below
ERROR: No value is specified
for help, see the <a href="https://github.com/rhodricusack/automaticanalysis/wiki">aa wiki</a>
Error using aas_log (line 65)
aa error:
see above (or set aap.options.verbose

Error in aamod_fieldmap2VDM (line 88)
                if isempty(job.defaults.defaultsval.et), aas_log(aap,true,'ERROR: No value is specified'); end
Error in aa_feval (line 15)
    [varargout{:}]=feval(funcname,varargin{:});
Error in aa_feval_withindices (line 4)
[aap,resp]=aa_feval(mfile_alias,aap,task,ci{:});
Error in aa_doprocessing_onetask (line 189)
            [aap,resp]=aa_feval_withindices(mfile_alias,aap,task,indices);
Error in aaq_qsub_debug (line 42)
                aa_doprocessing_onetask(obj.pool.Jobs(eii).Tasks.InputArguments{:})
65          error('aa:internal','aa error:\n%s\n','see above (or set aap.options.verbose')

Thanks, @ethanknights. It was indeed the issue. I have corrected it.

Hm, now crashing in aa_user_ASL on seemingly a much more fundamental problem with how toolboxes are coded in the defaults XML. Note that in the CBU defaults we have multiple toolbox entries, which does not seem to play nicely with validatepaths - it looks like mystruct.(fs{f}) should also be indexed by structind?

Error using isfield                                                            
Too many input arguments.                                                                                    
Error in aas_validatepaths>validatepaths (line 14)                                                           
    if (~isfield(mystruct.('ATTRIBUTE'),'ui'))                   
Error in aas_validatepaths>validatepaths (line 48)                                                           
                            validatepaths(aap,mystruct.(fs{f}),values.(fs{f})(structind),[nme '.' fs{f}]);
Error in aas_validatepaths>validatepaths (line 48)                       
                            validatepaths(aap,mystruct.(fs{f}),values.(fs{f})(structind),[nme '.' fs{f}]);
Error in aas_validatepaths (line 7)                                                                      
validatepaths(aap,aap.schema,aap,'aap');                                                                 
Error in aa_doprocessing (line 87)                                                                           
aap=aas_validatepaths(aap);                                                                              
Error in aa_user_ASL (line 57)                                                                  
aa_doprocessing(aap);                                                          
14      if (~isfield(mystruct.('ATTRIBUTE'),'ui'))   

@jooh, Can you point me to the script/analysis folder? I can see /imaging/jc01/aa/aa_demo/ASL, but it seems to be completed. Also, please, make sure you are using the latest version because e.g. the ASL used version a4ca590, which refers to your commit correcting the FreeSurfer installation.

This crash happens before the pipeline runs, so I don't think you'll get the right answer by looking at those old outputs.

I made another attempt. First I double-checked that I'm on the the latest version of the PR, and attempted to re-generate the user parameters XML. I get the following exception during the creation of the new XML (this is after selecting aap_parameters_defaults_CBSU.xml in the GUI). The exception is different, but notice that the problem seems to be handling a struct array with multiple elements again.

aa_user_ASL

Please wait a moment, adding <a href = "matlab: cd /home/jc01/matlab/aa5"></a> to the path
Welcome to aa version 5.4.0 (2f06ec377301b1446516ff7b0fdae55a80847e7a) Jul 2020
 If you publish work that has used aa, please cite our manuscript:
 <a href = "http://dx.doi.org/10.3389/fninf.2014.00090">Cusack R, Vicente-Grabovetsky A, Mitchell DJ, Wild CJ, Auer T, Linke AC, Peelle JE (2015)</a>
 <a href = "http://dx.doi.org/10.3389/fninf.2014.00090">Automatic analysis (aa): Efficient neuroimaging workflows and parallel processing using Matlab and XML</a>
 <a href = "http://dx.doi.org/10.3389/fninf.2014.00090">Frontiers in Neuroinformatics 8:90</a>

Please visit <a href = "http://automaticanalysis.org">The aa website</a> for more information!

Here you can find example <a href = "matlab: cd /home/jc01/matlab/aa5/aa_parametersets">parameter sets</a> and <a href = "matlab: cd /home/jc01/matlab/aa5/examples">examples</a>.
Ready.
Cannot find parameters file aap_parameters_user.xml, opening user interface to generate a new file
Undefined function or variable 'yID'.
Error in xml_read>mergeStructs (line 469)
            itemmatch = strcmp(xIDs,yID);
Error in xml_read>mergeStructs (line 482)
            res.(names{fnum}) = mergeStructs(x.(names{fnum}),y.(names{fnum}),Pref);
Error in xml_read>mergeStructs (line 482)
            res.(names{fnum}) = mergeStructs(x.(names{fnum}),y.(names{fnum}),Pref);
Error in xml_read>expand_tree (line 446)
    otree = mergeStructs(otree,itree.local,Pref);
Error in xml_read (line 195)
tree = expand_tree(tree,DPref);
Error in aarecipe (line 56)
    xml=xml_read(seedparam,Pref);
Error in aa_user_ASL (line 17)
aap=aarecipe('aap_tasklist_ASL.xml'); 
469             itemmatch = strcmp(xIDs,yID);
xIDs
xIDs =
  1x2 cell array
    {'spm'}    {'eeglab'}

It looks like this is also caused by a problem with how the toolboxes are coded as struct arrays in the defaults now. Would you mind checking if CBSU_defaults is compliant with how things are meant to work now? I copied from aap_parameters_defaults.xml so if CBSU is wrong it's possible this file is wrong too.

Hope you can reproduce this issue.

@jooh, the issue was not the parameterset but that I have not updated aarecipe for creating new usersettings.

Ok, good, so now I'm back to the exception I reported above in validatepaths.

@jooh, Your initial guess about indexing was a good lead, but we should not index with structind because the schema and the actual aap structure may not have the same amount of items. On the other hand, for validation, we can rely on any schema item i.e. the first one, which always exists.
How about now?

That did it. I already completed the general tests. Running the cbu tests now...

Now crashing on aamod_meg_denoise_ICA_1:

MEG:detect_ICA_artefacts:Undefined function 'runica' for input arguments of type 'double'.

In fact it looks like the problem is that rik_runica isn't found (L44), which leads to looking for runica instead, and eventually a crash on L55.

That is a known bug I am afraid. We can discuss how we want to proceed. I remember Rik was reluctant to give out his scrip, and in this case, I prefer removing it completely.
Then we/I just have to make sure that EEGLAB (containing runica) is loaded.

Hm, well the tests must have passed in previous version by falling back to runica then, because I don't think rik_runica has ever been on my path. So I think it's fine to just ensure EEGLAB is available in this module.

Unless @RikHenson would like to contribute this function to AA? It's a little weird that a public AA module references a personal function.

@jooh, I have updated the path to EEGLAB at the CBSU and its usage in the ICA module.

I think we're close now. I managed to get through the CBU test suite once, but after so many updates I thought it would be safest to do a clean re-run. Now I get stuck on aamod_coreg_extended_2 in the aa_user_MTI pipeline:

identifier: 'MATLAB:audiovideo:VideoWriter:invalidDimensions'
message: 'Frame must be 729 by 1051'

aas_checkreg_avi, line 101.

The images that are being saved are /imaging/jc01/aa/aa_demo/mti/aamod_MTI_coreg_extended_2_00001/S05/structurals/msCAMCAN_CALIBRATIONS-0003-00001-000192-01.nii,1 and /imaging/jc01/aa/aa_demo/mti/aamod_MTI_coreg_extended_2_00001/S05/MTI/mti_MTR.nii,1. The GUI state looks ok in my VNC viewer.

The Matlab version is R2019a. I see the following warning from spm_check_registration before the exception: Warning: figure JavaFrame property will be obsoleted in a future release. For more information see <a href="http://www.mathworks.com/javaframe">the JavaFrame resource on the MathWorks web site</a>.

I am not sure what the issue might be but I have an educated guess.
The report ran without any issue for me (see /imaging/ta02/aa/aa_demo/mti).
The error message you received complains about inconsistent window sizes, which can happen if you accidentally change the SPM figure generating the overlay video. It is usually strongly advised no to touch the whole VNC session while running aa_report. You can, perhaps, try submitting report generation as a job, but I am not sure whether jobs can create figures at the CBU. At Surrey, they can.

Ok, just had an intriguing exception where aas_shell appears to have started to treat my shell as CSH rather than bash half way through the tests, with all kind of errors as a result (can't use setenv etc). Can't reproduce on a fresh Matlab session so I'm going to try running the two affected tests back to back and see if the exception re appears. Stay tuned...

I'm going to be away for a bit so going merge today so we can finish this very extended review. I'm satisfied that all the user scripts run in isolation, which is going to have to be good enough.

There are definitely weird dependencies between the different user scripts when you run them in the same Matlab session during aatest, but I don't have the time to get to the bottom of this now. For instance, the GUI error I had above is probably caused by one script putting the SPM GUI in a state where another script no longer works, and I think the exception I had with CSH above is a similar thing, that only occurs when one particular script follows another in the Matlab session and the cache somehow ends up in an invalid state.

The underlying cause of all this is that AA has a lot of hidden state, both in terms of the cache and the SPM GUI. For now it remains good advice to start a new Matlab session whenever you try to run a new user script, but this is going to cause confusion for new users because it runs a little counter to how people typically run Matlab (as a continuous session, not a one-off 'application'). Longer term it would be good to ensure that AA starts and closes more cleanly.

Thank you, @jooh, for the exhaustive and persistent testing. 🥇