Module Development Guide

Automatic Analysis can be extended by adding modules that implement custom functionality. Although a module is really nothing more than a Matlab function, some specialized coding is required in order to properly interact with the scheduler and other aa internals. The purpose of this document is to describe module creation in sufficient detail so that users may implement their own extensions should the need arise.

Before creating a new module, you should thoroughly review the current contents of the aa distribution, as replication of functionality should be kept to a minimum. Analysis modules are located in the aa_modules directory. Useful utilities may also be found in the aa_tools, external, and extrafunctions directories.

Terminology: In this document, the code you create to implement new functionality (typically an mfile/xml header pair) will be called the module. The term script (or userscript) will refer to the mfile created by an end user which makes use of your new module (as well as other modules ) in an analysis. The xml file created by the user (technically, the modules listed therein) is called the tasklist.

1. The aa Module

An aa module consists of two files: 1) a Matlab mfile and 2) an associated xml header. These have the same name, but a different extension (.m and .xml, respectively). Occasionally, a new module is implemented by reusing an existing mfile in the distribution and creating a new header for it (see the mfile_alias attribute described in Section 1.2). However, you will usually create both. The files should be placed in the aa_modules directory of the aa distribution. If you wish to make your module available to the aa userbase, you can fork the aa repo and open a pull request that includes your new code.

Your module is not called directly from a user’s mfile script. Rather, a user accesses your module’s functionality by including its name in their analysis tasklist. The aa engine is responsible for parsing the tasklist, and locating and calling the mfile function associated with each module listed in the tasklist, in the proper sequence and with the proper input arguments. This arrangement introduces coding practices peculiar to aa development. To explain these, it’s necessary to introduce the module domain, the module working directory, and the concept of streams.

The Module Domain

Each module must declare an analysis domain in its xml header. This is specified using the domain attribute for the currentlist tag (see Section 1.2). Although about a dozen custom domains are currently defined in the aa distribution, most modules will use a domain of study, subject, or session. There are important differences in calling convention and the data passed to a module defined each of these domains. Generally speaking, a subject domain is used for modules that process only structural data, and either subject or session can be used in modules that process functional data. During an analysis run, a subject domain module will be called once per subject (defined in the userscript using aas_addsubject). A session domain module will be called once per session per subject (defined using aas_addsession). Modules with a domain of study will be called exactly once, and typically implement second-level statistics.

Conceptually, think of each subject as having only one structural scan but as many functional scans as there are protocols included in the study. Each of these functional scans is one session. This distinction will be reflected in the contents of the module working directory, which will contain one subdirectory (typically called “structural”) holding all of the structural data files (if any), and a number of subdirectories containing functional data, one for each defined session (if any). The contents of these directories can be a single 3D (structural) file, a series of 3D or one 4D nifti file (epi), or a collection of DICOM files.

The Module Working Directory

During an analysis, aa generates a rather extensive directory tree which it uses to distribute files used/created by modules listed in the tasklist. We will refer to the directory (and all subdirectories) containing files related to a given module as the _module working directory _(MWD). The aa toolkit provides a suite of functions used for reading and writing files in your module’s working directory. These functions are described in a later section.

Streams

Most file interaction in aa is handled using a data abstraction called a stream. A stream name is a short self-explanatory string you define in the xml header (e.g., structural, epi, meanepi, ROI, firstlevel_betas) which provides a convenient alias you use in the module mfile code instead of files names. The aa engine handles the details of managing the file(s) that are associated with a given stream.

There are two primary stream types: input and output. These are listed in the outputstream and inputstream sections of the xml header (see Section 1.2) — usually at least one of each are defined in a module. The input stream(s) are generated by a previous module in the tasklist and supplies data your module requires. The output stream(s) created by your module supply data to module(s) appearing later in the tasklist.

We now describe the organization of the module mfile and xml header.

1.1 The Module mfile

The module mfile is a standard Matlab function with features customized for aa. The overall structure is as follows:

function [aap, resp] = aamod_mymodule(aap, task, subject_index, session_index)

switch task
	case ‘report’
	   < case ‘report’ code goes here >
	case ‘doit’
	   < case ‘doit’ code goes here >
	case ‘checkrequirements’
	   < case ‘checkrequirements’ code goes here >
	otherwise
	   < error code goes here >
end

end

By convention, the module name should begin with the prefix aamod_ (the file name must be the same as the function name). The function must take four input parameters: the aap structure, the current task, the current subject index, and the current session index. Depending on your module domain, some of these variables may be undefined when your function is called.

A subject domain module will be passed a valid subject index when invoked. This is an integer indexing of the subjects in the order they were defined and can be used to locate and read subject data (described in the next section). A session domain module will be passed a subject index and a session index, the latter is an integer indexing the subject’s session(s) in the order added. These are also used to locate and read data. The session index is undefined if your module’s domain is specified as subject and neither the subject nor session index is defined in a study domain module. Any attempt to access an undefined variable in your module will cause aa to crash.

The module should return the aap structure and a response string (resp). However, many aa modules in the current distribution just return an empty string.

The module is organized around a switch construct. A complete and proper aa module implements the following cases:

`report`
	implements functionality (if any) specific to report generation.

`doit`
	the primary analysis code goes here.

`checkrequirements`
	code that is run at the beginning of an analysis to check requirements

You may find older modules that include cases such as 'domain', 'whentorun', 'description', or 'summary'. These have been depreciated and should not appear in new module development.

The doit Case

The bulk of the module coding effort goes into the doit case. The code here is structured as follows:

1) extract data from one or more input streams
2) process the data
3a) write results to one or more output files and associate those files with an output stream
3b) (optional) save graphical results to a file

We now consider each of these tasks in turn.

Extracting Data from an Input Stream

Extracting input stream data is a two-step process: 1) get the stream name, then 2) get the filenames(s) associated with the stream name. Generally, speaking, there are three options for getting the stream name: 1) using aas_getstreams, 2) using the currentask field in the aap structure passed to your module, or 3) hardcoding the stream name as a literal string.

Method #1: Use aas_getstreams

The function aas_getstreams takes a stream type (which is almost always either ‘input’ or ‘output’) and returns a cell array of stream names defined in the xml header for that type. This will be made clearer by considering a few examples.

Example 1: Extract all input streams for a module (i.e., all streams defined in the inputstream section in the xml header):

inputStreams = aas_getstreams(aap,'input');

Example 2: Extract all output streams for the module:

outputStreams = aas_getstreams(aap,’output’);

If multiple streams are defined for the specified stream type, access them individually using standard cell indexing:

  streams = aas_getstreams(aap,’input’); 
  firstStream = streams{1};
  secondStream = streams{2};

Method #2: Use the currenttask field in aap

The second method for obtaining stream names is to extract them from the aap structure passed to your module. Drilling down through the various levels in the aap structure gives you access to different stream information. At the lowest level, the information is stored as a stream struct — a Matlab structure with fields CONTENT and ATTRIBUTE. The name of the stream is in the CONTENT field, and the ATTRIBUTE field contains additional information such as if the stream is renameable. If the stream has no attributes, as string (containing the name) rather than a struct is returned.

Example 1: Extract all input streams defined in the header

istreams = aap.tasklist.currenttask.settings.inputstreams;

Example 2: Extract all output streams

ostreams = aap.tasklist.currenttask.settings.outputstreams;

Example 3: Extract the first input stream

s1 = aap.tasklist.currenttask.settings.inputstreams(1).stream;

Example 4: Extract the first input stream struct

s1 = aap.tasklist.currenttask.settings.inputstreams(1).stream{1};

Example 5: Extract the first input stream name

s1 = aap.tasklist.currenttask.settings.inputstreams(1).stream{1};
stream_name = s1.CONTENT;

Method #3: Hardcode the stream name

Finally, you may simply hardcode the name of streams in your module code as they appear in the xml header (an example is included in the next section). This is the simplest approach, but it makes your module less flexible. Specifically, retrieving a stream name using aas_getstream or the aap struct allows the stream to be renameable. For example, aamod_histogram constructs a voxel histogram of the input stream. However, it’s impossible to know the stream of interest in advance — t1, t2, epi, or some other image data. As such, a generic input stream (simply called ‘input’) is defined in aamod_histogram.xml and tagged as renameable. The user assigns this stream to the data of interest using aas_renamestream in their analysis script. The code in aamod_histogram.m extracts the stream from the passed aap struct, which aa has initialized to the assigned name and so the module is able to operate on whatever data the user specifies.

Reading the File associated with the Stream

Once a stream has been retrieved, the aa toolkit provides functions for locating the file(s) associated with the stream. Two will suffice for most applications: aas_getfiles_bystream and aas_getimages_bystream. These functions take a stream-name or a stream-struct and return the full paths of the file(s) associated with the stream. These can then be passed to Matlab functions such as load or fopen, or SPM functions such as spm_vol to read the file contents.

It is recommended aas_getimages_bystream be used for reading functional data (i.e., “epi” streams) and aas_getfiles_bystream be used for all other stream types. As such, the code examples shown below only pass a subject index to aas_getfiles_bystream, although the function will accept a both a subject and session index (recall the earlier discussion of subject versus session module domain). You may come across examples of this usage in the distribution.

_Using aas_getfiles_bystream _

The function aas_getfiles_bystream offers a great deal of flexibility in the input parameters it can accept. To avoid confusion, we’ll consider only the following usage:

   filename = aas_getfiles_bystream(aap, subject_index, stream_name);

The function returns one or more filenames which can then be passed to a Matlab or SPM function to read the file contents. Here are three examples taken from the current distribution:

Example #1: Using aas_getstreams

streams = aas_getstreams(aap,'input');
img = aas_getfiles_bystream(aap, subjind, streams{1});
V = spm_vol(img);
Y = spm_read_vols(V);

Example #2: Using the currenttask field

% from: aamod_mask_fromstruct.m (note aamod_mask_fromstruct.xml
% defines two input streams: ‘structural’ and ‘segmentation’)

inStreams = aap.tasklist.currenttask.inputstreams;
Simg = aas_getfiles_bystream(aap,subj,inStreams.stream{1});
SEGimg = aas_getfiles_bystream(aap,subj,inStreams.stream{2});

Example #3: Using a hardcoded stream name

spmName = aas_getfiles_bystream(aap, subjInd, 'firstlevel_spm');
load(spmName);

Note you can also pass aas_getfiles_bystream a stream structure rather than a stream name:

filename = aas_getfiles_bystream(aap, subject_index, stream_struct);

The value returned (a filename or list of filenames) is the same as when passing in a stream name.

Using aas_getimages_bystream

The syntax of aas_getimages_bystream is similar to aas_getfiles_bystream, the difference being the addition of a session index. We’ll consider only the following calling syntax:

filename = aas_getimages_bystream(aap, subject_index, session_index, stream_name);

The function returns one or more file paths. The stream name is almost always epi but can also be an epi-related stream such as epi_dicom_header. Here’s an example from aamod_movie.m (i and j are the subject index and session index passed to the mfile):

imgs = aas_getimages_bystream(aap,i,j,'epi');
V = spm_vol(imgs);

Note “images” here is a generic term — it does not refer to reading the old Analyze .img file format.

A Note on Multi-file Streams

If a stream comprises multiple files (e.g., 3D epi), the functions aas_getimages_bystream and aas_getimages_bystream will return the file pathnames in a character array (each row is the full path of one file). As Matlab character arrays cannot have variable row lengths, the entries in the array will be padded if necessary. This trailing whitespace can confuse other functions, an so it is typically removed using deblank( ) when (or before) using the pathnames elsewhere.

Data Processing

Once you have identified the filename(s) associated with a given stream, you are free to read these files and process the data using any valid Matlab code, including SPM or other installed toolbox functionality (at minimum, your code will probably make calls to spm_vol to read the data).

Output

Two types of output can be generated by your module: 1) one or more output streams and 2) graphical results which are usually included in the aa report.

Output Streams

An output stream is created by 1) defining an output stream in the header, 2) saving the data associated with the stream to a file in the module working directory, and 3) associate the file with the output stream. The last step is done by describing the file using aas_desc_output. This generates a unique identifier aa will use to make your output stream available to other modules that list it an input stream (here, “describe” should be interpreted in the sense of “designate” or “label” — it does not involve a literal description of the file).

The function aas_desc_output will accept a variety of input parameters, which can be rather confusing. The basic syntax is as follows:

aap = aas_desc_outputs(aap, domain, indices, streamname, filename);

here, domain is the domain of your module, streamname is a string containing the name of the output stream as defined in the xml header, and filename is the name of the output file to be associated with the stream. If domain is subject, then indices is a scalar equal to the subject index (if the subject index is 1, this can be omitted). If domain is session, then indices is a vector [subject_index session_index]. If domain is study, the indices are omitted. Some existing code uses an old syntax in which domain is omitted or which passes separate subject and session indices rather than a vector. Here’s a few example calls taken from the current distribution:

From aamod_convert_epis.m (complete parameter set):

spm_write_vol(V,Ymean);
aap = aas_desc_outputs(aap, domain, indices,'epi_mean', V.fname);

From aamod_structuralstats.m (note domain is omitted):

outfile = fullfile(fileparts(img), 'structuralstats.mat');
save(outfile,'S'); % ’S’ is created earlier in the code
aap = aas_desc_outputs(aap, subjind, 'structuralstats', outfile);

From aamod_convert_fieldmaps.m (note domain is omitted, and subject and session are passed separately)

dcmhdrfn=fullfile(sesspath,'fieldmap_dicom_header.mat');
save(dcmhdrfn,'dcmhdr');
aap=aas_desc_outputs(aap, subj, sess,'fieldmap_dicom_header',dcmhdrfn);

From aamod_secondlevel_threshold.m (note domain and indices are omitted):

Outputs.thr = strvcat(Outputs.thr, V.fname);
aap = aas_desc_outputs(aap,'secondlevel_thr', Outputs.thr);

An easy point of confusion is that it is possible to associate more than one file with a stream. This is done by passing a (vertically concatenated) list of file names to aas_desc_outputs instead of a single file name. For example, aamod_norm_noss associates two files with the structural output stream using the following code:

aap=aas_desc_outputs(aap,domain,indices,'structural’,strvcat(Simg,Sout));

The variables Simg and Sout are paths to the raw and bias corrected structural image. The former was input to the module, the latter was generated during module execution.

A more elaborate example is provided by the definition of the segmentation output stream, also in aamod_norm_noss. Filenames for the native and normalized grey, white, and CSV segmentations (c1*, c2*, c3*, wc1* wc2*, and wc3*, in SPM parlance) are concatenated using a for loop, then the six files are associated with a single segmentation output stream:

while ~isnan(d)
 d = d+1;
 if exist(fullfile(Spth,sprintf('c%d%s',d,['m' Sfn Sext])), 'file')
  outSeg=strvcat(outSeg,fullfile(Spth,sprintf('c%d%s',d,['m'Sfn Sext])));
  outSeg=strvcat(outSeg,fullfile(Spth,sprintf('wc%d%s',d,['m'Sfn Sext])));
else
  d = NaN;
end
end

aap = aas_desc_outputs(aap, domain, indices,'segmentation', outSeg);

These examples also nicely illustrate the bookkeeping scheme used by the aa for stream management. Following execution of aamod_norm_noss, a plaintext file is generated in the module working directory named stream_structural_outputfrom_aamod_norm_noss. The contents of this file consists of an MD5 header followed by two relative paths pointing to the raw and bias-corrected structural files. Similarly, the contents of stream_segmentation_outputfrom_norm_noss are relative paths pointing to the six tissue segmentation files. The streamname is parsed from the text file name by the aa scheduler, which it will then translate into the literal file paths for you as the need arises (for example, by a call to aas_getfiles_bystream in a subsequent module). In this way, your code need not deal with the details of file naming, including the transformation of filenames generated by SPM over the course of processing, but rather need only deal with a single immutable stream name.

Graphical Results

Graphical results are generated in a module usually for the purpose for including them in the analysis report (the report file is generated by a call to aa_report in the userscript). It appears to be standard practice to generate graphical results during analysis (i.e., the doit case) which are saved to disk. These results will then exist in the module working directory even if the user does not choose to generate a report.

Locating a Module’s Working Directory

As a rule, a module should save any files it creates in the module working directory or in a subdirectory created under it. This will be <root>/<analysisid>/current-analysis-stage for study-level modules, <root>/<analysisid>/current-analysis-stage/current-subject-name for subject modules, or <root>/<analysisid>/current-analysis-stage/current-subject-name/current-session-name for session modules. Here, <root> is the directory specified by the userscript in aap.acq_details.root and <analysisid> is the subdirectory under root specified in aap.directory_conventions.analysisid.

There are several aa utility functions available to help you identify the module working directory. Here’s an annotated list of such functions currently in use in the aa distribution:

path = aas_getsubjpath(aap, subject_index)

This will return the MWD (e.g., <root>/<analysisid>/aamod_mymodule_00001/<current_subject_name>) for your module. You can pass a third argument to get the MWD for a previous stage. For example:

path = aas_getsubjpath(aap, subject_index, 1) 

will return <root>/<analysisid>/aamod_autoidentifyseries_timtrio_00001/<current-subject-name>, assuming the first stage of your analysis script was autoidentifyseries_timetrio. The index ignores the modules specified in the initialisation block of the header.

path = aas_getsesspath(aap, subject_index, session_index);

will return the MWD for a domain subject module (e.g., <root>/<analysisid>/aamod_mymodule_00001/<current_subject_name>/<current_session_name). This will also work for a subject domain module by setting session_index equal to 1 (remember the session index passed to your module will only be defined if the domain of the module is declared as session in the xml header). That being said, if your module is domain subject, it’s probably best to use aas_getsubjpath instead.

path = aas_getpath_bydomain(aap, domain, [ indices ] )

This is the most general directory utility, and will work with any domain type module. Common usage include:

path = aas_getpath_bydomain(aap, ‘subject’, 4)

which would return the MWD for the fourth subject in a subject domain module.

path = aas_getpath_bydomain(aap, ‘session’, [2 3])

which returns the MWD for the third session of the second subject in a session domain module. In general, we have:

path = aas_getpath_bydomain(aap, domain, [ indices ], module_index)

where module_index can be used to specify a previous analysis stage module, and the content of indices must be appropriate for the domain specified (see the distribution for use of domains other than session or subject). Note the domain of your module is stored in the variable aap.tasklist.currenttask.domain.

You can also obtain the path to the current module directory using getstudypath:

path = aas_getstudypath(aap);

This returns the variable defined in aap.acq_details.root.

Finally, two additional utility functions are useful when working with session modules:

session_name = aas_getsessname(aap, session_index)

returns the session name (i.e., the string specified in aas_addsession ) for the specified session_index.

session_description=aas_getsessdesc(aap, subject_index, session_index);

returns a description of the current session including the subject and analysis modality (e.g., ‘MRI’). This is handy for figure captions and log files.

Output File Naming Conventions

There currently appears to be no standardized naming convention for image files. Some modules create a “diagnostics” subdirectory in the module working directory and place graphics files there. Other modules simply place graphics files in the working directory and prefix filenames with the identifier “diagnostics_”. Some extant modules use a combination of both approaches. For example, aamod_realign saves plots of the rigid body corrections generated by SPM in a diagnostics subdirectory using the file prefix “aamod_realign_”. It also saves a plot of summary statistics in the aamod_realign analysis directory using the prefix “diagnostic_”.

In the long term, we should enforce a consistent naming convention across the aa distribution. However, any convention will work in your module as long as you use it consistently in the doit and report code sections. The only aa module that needs to locate your module’s graphical results is aa_report, and you control that internally (cf. the report case). All that is required is the code in the report case that reads the results for report generation is consistent with the code in the doit case that wrote them.

Should you decide to use the first approach mentioned above, aa provides a convenience function to create a “diagnostics” subdirectory in the current analysis directory:

subjectName = aas_prepare_diagnostic(aap, subjectIndex); 

The function will check if a “diagnostics” directory already exists before attempting to create one, so you need not worry about overwriting any previously created files. It returns a string containing the name that was assigned to the subject in aas_addsubject.

Should you instead choose the second approach, you simply need to prepend ‘diagnostic_’ to the names of any files you create. There is some merit to this, in that it will simplify the task of identifying all diagnostic images generated by aa in a given analysis directory tree for other kinds of postprocessing (e.g, as might be done via shell script using find).

Generating and Saving Graphical Results

You can generate necessary graphical results using any Matlab functionality, including that provided by a third-party toolbox such as SPM or FSL (e.g., spm_figure or spm_orthviews or spm_render). The aa engine also provides a few functions (e.g., aas_checkreg, aas_realign_graph) that generate graphical results that can be included in a report. Once the graphics are displayed, there are a number of options available to save the figure contents to a file. The preferred format appears to be a 150 dpi resolution jpeg. This file can be generated using the Matlab print command. You may want to optimize the rendering technique before calling print. Here is an example from aamod_listspikes. The code plots results in a window created by a figure(2) call (aa modules use figure 2 because figure 1 is reserved for the SPM graphics window); the window contents are then saved as follows:

set(2,'Renderer','zbuffer');
print(2,'-djpeg','-r150',fullfile(aap.acq_details.root,'diagnostics', …
		[mfilename '__' subjname '.jpeg']));

(Note there are minor two errors here. First, the correct file extension should be “.jpg” not “.jpeg”. Second, zbuffer rendering is depreciated in the current release of Matlab. Use opengl or painters instead. The painters renderer draws using vector graphics (slower, better quality); opengl draws using raster graphics (faster, lower quality). For most figures you probably won’t notice a difference.)

If you omit the figure handle in the call to print, the current figure will be printed (which should be the intended figure assuming it was the last figure created). Note print is notorious for altering content (aspect ratio, line weights, etc) when saving a figure to file. Text is especially problematic. Calling:

set(findall(gcf,'Type','text'),'FontUnits','normalized’);

in your code immediately before the print command will permit font scaling, which can improve text appearance when graphics are saved to a file.

If you attempt to generate graphics in the SPM graphics window and the window is not currently open, the result will likely be an empty jpeg. You can force SPM to open the graphics window (and create it if it does not exist) by calling spm_figure:

    h = spm_figure('GetWin', 'Graphics');

You should close a figure once the contents have been saved. This can be done by passing the figure handle to the Matlab close function. You can do something like this:

< generate and save SPM graphics here >
close(h);

NB: Closing the SPM Graphics window may prevent other modules from creating graphical results (if they don’t check that the window has been closed). It may be safer to not close the window.

Adding Graphical Results to the Report

The aa report is a standard html file. There are two functions for adding content to it: aas_report_add and aas_report_addimage. Both of these functions take three parameters:

	aap = aas_report_add(aap, <subject index>, html_string);
or
	aap = aas_report_add(aap, <section label>, html_string);


        aap = aas_report_addimage(aap, <subject index>, fullpathtoimage);
or
        aap = aas_report_addimage(aap, <section label>, fullpathtoimage);

The second parameter designates the section of the report to which the content should be added. This can be either the subject index (an integer) or a section label (a string). Section labels currently recognized are moca (motion correction), reg (registration), and Cxx (a contrast number). If an empty field [ ] is passed, the contents are added to the main body of the report.

The html string passed to aas_report_add can contain any valid html string, including formatting tags. A common use is to open and close a table when inserting an image into the report. For example:

aap=aas_report_add(aap,subjectIndex,'<table><tr><td>');
sesspath=aas_getsesspath(aap, subjectIndex, sessionIndex);
aap=aas_report_addimage(aap,subjectIndex,fullfile(sesspath,’aresult.jpg'));
aap=aas_report_add(aap,subjectIndex,'</td></tr></table>');

This code snippet assumes that the file someresult.jpg exists in the proper directory (this will have been generated in the doit section of the same module). Note the image is preceded by a html tags that open a table, a row in the table, and a data cell within the row. The image is then added using aas_report_addimage, and a second call to aas_report_add is made which closes the tags in the opposite order. If incorrect html is passed, an error will be generated when attempting to subsequently open the report.

These functions may be called as many times as needed.

1.2 The Module xml Header

The module xml header file is written using Extensible Markup Language. The overall structure is as follows:

<?xml version="1.0" encoding="utf-8"?>
<aap>

<tasklist>

<currenttask domain='subject' desc=‘helpful description’ modality='MRI'>

<permanenceofoutput>2</permanenceofoutput>
                    
<!— module parameters and their default values go here —>

<param_1 desc=‘helpful description of param_1’>0.111</param_1>
<param_2 desc=‘helpful description of param_2’>0.222</param_2>
      
<!— input streams —>

<inputstreams>
	<stream>input_stream</stream>
	<stream>another_input_stream</stream>
	<stream isessential=‘0’>an_optional_input_stream</stream>
	<stream isrenameable=‘1’>a_renameable_input_stream</stream>
</inputstreams>
      
<!— output streams —>

<outputstreams>
	<stream>an_output_stream</stream>
	<stream>another_output_stream</stream>
	<stream>still_another_output_stream</stream>
</outputstreams>
					
</currenttask>

</tasklist> 
</aap>

The domain and desc entries in currenttask are required. Modules typically have one or more input and output streams* as well as optional parameters. The details of each header file are of course task specific. (* currently, aa will crash if any module in a script other than the last has no output streams. This is a bug in the aa internals).

It is good programming practice to include a description of parameters defined in the header by utilizing the desc field in the definition. Although the meaning of a given module parameter is obvious to the module developer, it may be less-so to an end user. Making your code self-documenting helps others to understand the proper use of your module. Parameter naming should always err on the side of clarity; entries like “verbose” and “provenance” are unhelpful.

currenttask Attributes

domain

	This field is required. The value will probably be either subject or session.
				
	A partial list of domain names currently in use in aa distribution include:

		scan
		study
		subject
		session
		isc_session
		meg_session
		diffusion_session
		splitsession_cv_fold
		searchlight_package
		hyperalignment_subject
		splitsession_cv_fold_hyper
		diffusion_session_bedpostx

desc

	This field is required.  A helpful description of the module functionality. 
	This text is displayed in the command window when the module is called by aa.

modality

	aa currently recognizes the modalities ‘MRI’ and ‘MEG’ 
	If the modality is not specified, MRI is assumed.


mfile_alias

Existing module functionality can be customized for a new application without the need to change
the mfile code. By default, aa expects to find an mfile having the same name as the .xml file 
(but with a ‘.m’ extension). This default behavior can be overridden using the mfile_alias tag.

Other Header Content

timebase
memorybase

	THese tags are related to qsub cluster processing.

permanenceofoutput

	This tag is used in garbage collection.

Custom Parameters

You may include custom parameters in your xml header simply by including the name of the parameter as a tag. A description and a default value should be included. The advantage of listing these parameters in the header is that the default value can be easily overridden in the userscript (see Appendix B). For example:

<samp desc='Sampling distance (mm)'>1</samp>

NB: It appears to be possible to restrict values of a parameter by using the ui tag. Possible variants include dir (requiring the user specify a directory), double (i.e., a real number), text (i.e., a string) or optionlist (accepted values are listed explicitly). For example:

<resolution options=‘low|med|high’ ui='optionlist'>low</resolution>

The intent here is to restrict values of resolution to low, med, or high. However, it appears to be up to you to implement parameter checking in the module’s mfile code; the ui tag alone does not accomplish this automatically. Many modules in the current aa distribution specify parameter restrictions in the xml header which have no effect.

You access parameters declared in the header from your module’s mfile using:

aap.tasklist.currenttask.settings.parameterName;

where parameterName is the string used as the tag for the parameter in the .xml file. You may come across an alternate way to code parameter access using aap.tasksettings instead of the tasklist.currenttask field:

aap.tasksettings.(MODULENAME).parameterName;	% don’t do this

As explained on the aa website, this approach should be avoided for two reasons. If there is more than one instance of (modulename) in a tasklist, you won’t know which one holds the settings for the current job. Second, aap.tasklist.currenttask can be customised by extraparameters in the tasklist.

In brief, always use aap.tasklist.currenttask.settings to access parameters defined in your module’s header.

Stream Attributes

A number of optional attributes can be applied to input and output streams.

isessential

Many processing tasks can take optional input that is useful but not essential for the task at hand. 
For example, spatial normalization can be performed using only GM, but SPM provides the option
to use both GM and WM. 

To determine whether an optional input stream is available, include the following in your code:
	
	if aas_stream_has_contents(aap, streamname)
		< retrieve and process streamname >
	end

Here, _streamname_ is the name of a nonessential stream. This is specified setting isessential=0 
in the script tag. For example:

		<stream isessential=‘0’>t2</stream>

Note the 0 value is a string not a Boolean, and so it must be enclosed in quotes. 

If the `isessential` attribute is omitted, the stream is assumed to be essential. When a
userscript is run, aa analyzes module dependencies and will halt execution with an error
message if it identifies a module having an undefined essential input stream.

Important: A module must define at least one essential input stream, otherwise the aa
scheduling algorithm will crash.

ismodified
	
	If your module does not modify a given input stream, you should tag the stream
	as unmodified (i.e., apply the tag `ismodified=‘0’`). This will allow aa to create 
	(hard) link(s) to the file(s) associated with the stream rather than making duplicates, 
	which can result in substantial disk space savings.

isrenameable

	A renameable stream allows you to define a generic stream in your module that
	the user can reassign as they see fit (an example is described in Section 1.1). 
	Although the implementation of a module determines whether a stream is 
	functionally renameable, aa prevents stream renaming by default. As such, 
	you must explicitly tag a stream `isrenameable=‘1’` if you wish to make it
	renameable. Note ‘1’ is a string, not a Boolean.

diagnostic

	This tag indicates the stream is used in reporting. It is not necessary to designate
	such steams, however it will prevent garbage collection from deleting it.

forcedomain

	The purpose of this tag is unclear.

named streams

According to the aa website, you can request streams from a specific module. For example:

		<stream><name>aamod_realign.epi</name></stream>

refers to the epi stream generated by `aamod_realign` and not the epi stream output created
by any subsequent processing of the epi stream in the userscript. This can be useful when
implementing functionality that requires an input stream having a known state.

Error Handling

Proper error handling is critical to improving the accessibility of aa, especially for new users. Your module should check the parameters and data passed to it in as much detail as possible and flag incorrect or suspicious input. At best, failing to do so will cause aa to crash, often at a later analysis stage making it extremely difficult to identify and correct the error. At worst, improper data may generate wrong results with no warning at all. As the module developer, you are in the best position to evaluate the data passed to your module and judge its correctness. Your code should alert the user immediately if usage appears suspect and suggest how best to correct the problem.

The programming tool for implementing error handling in aa is aas_log, which allows your module to print messages to the Matlab command window and optionally halt execution:

aas_log(aap, flag, message, style);

The Boolean flag is true if execution should stop and false otherwise. The Matlab function sprint can be used to include data values and formatting in message (examples to follow). The optional style string can be used to specify the formatting of the text.

As an example, suppose your module accepts a parameter called baseline, which will be used to normalize data in a subsequent calculation. If the value is zero, it will eventually cause execution to halt with a divide by zero. Although Matlab will generate an error message when that occurs, it is better to check the value of the parameter in your module and provide a informative message if an improper value is encountered:

if (baseline == 0)
  aas_log(aap, true, … 
 sprintf(’\n%s: Baseline cannot be zero. Exiting.’, mfilename), ‘r’);
end

Note mfilename is a Matlab built-in variable that identifies the m-file in which the error occurred. This can be useful for debugging purposes.

If the value of a parameter is not technically wrong but still suspicious, your module can warn the user:

if (baseline < 0)
  aas_log(aap, false, … 
  sprintf(’Warning: Negative baseline passed to %s.’, mfilename));
end

Note false is passed as the error flag so that execution does not halt.

Ideally, the error message should include instructions for correcting the problem if the solution is not obvious. For example, suppose your module uses the first covariate for global normalization. If no covariates are specified, the module should generate an error and instruct the user what to do:

if (total_covariate_count == 0)
  aas_log(aap, true, …
  sprintf('\n%s: You must supply data for global normalization
  (use aas_addcovariate to add as 1st covariate)\n', mfilename));
end

Checking input to your module is tedious and anticipating all eventualities may not be possible, but including proper error handling in your module will help the user obtain correct analysis results in a timely fashion with the minimal amount of frustration.

Summary

Custom aa module development in allows the end user to add functionality not currently available in the toolkit, or to tailor current functionality to the needs of a specific application. The typical custom module consists of an mfile and an associated xml header. Both of these files should be copied to the aa_modules subdirectory of the aa installation directory; both files should have the same name (except for extension) and the file names should begin with the prefix aamod_. The key content of the xml header is the module domain, and the module’s input and output streams. The key content of the mfile is the analysis code contained within a doit case branch. The analysis code typically reads data from one or more input streams, processes the data, then writes results to one or more output streams. It may optionally generate graphical or diagnostic results. A report case will be called at the end of the analysis, which can retrieve output generated by the module and add this content to the aa report using the appropriate html embedding.

Appendix A: Simple Example Module

XML Header

<?xml version="1.0" encoding="utf-8"?>
<aap>
<tasklist>
    <currenttask domain='subject' desc=‘aa example module’ modality='MRI'>
    
        <permanenceofoutput>2</permanenceofoutput>
                    			
        <thresh desc='threshold'>0.123</thresh>
        
        <inputstreams>
            <stream>structural</stream>	
        </inputstreams>
      
        <outputstreams>
            <stream>voxel_histogram</stream>
        </outputstreams>
		
    </currenttask>
</tasklist> 
</aap>

Matlab mfile

function [aap,resp] = aamod_TESTMODULE(aap, task, subjectIndex, sessionIndex)
%
% AAMOD_TESTMODULE -- Test module creation for aa
%
% [aap,resp] = aamod_TESTMODULE(aap, task, subjectIndex, sessionIndex)
%
%

resp='';

switch task
		
case 'report'
   		
% add the voxel histogram we created in 'doit' to the report
	
voxogram = fullfile(aas_getsubjpath(aap, subjectIndex),'voxogram.jpg');
aap = aas_report_add(aap, subjectIndex, '<table><tr><td>');
aap = aas_report_addimage(aap, subjectIndex, voxogram);
aap = aas_report_add(aap, subjectIndex, '</td></tr></table>');
	
	
case 'doit'
    		
% 'thresh' is currently unused -- this just shows how to access a 
% parameter defined in the xml file:
	
thresh = aap.tasklist.currenttask.settings.thresh;
	
inputstreamname = aap.tasklist.currenttask.inputstreams(1).stream{1};
	
inputImg = aas_getfiles_bystream(aap, subjectIndex, inputstreamname);	
[pth, nm, ext] = fileparts(inputImg);
    
V = spm_vol(inputImg);		
	
[Y, xyz] = spm_read_vols(V);

% make and save a voxel histogram
	
h = figure;	
temp = histogram(Y);
	
% save histogram figure so we can add it to the report later
	
voxogram = fullfile(aas_getsubjpath(aap, subjectIndex),'voxogram.jpg');
set(h,'Renderer','painter');
print(h, '-djpeg', '-r150', voxogram);
close(h);
	
% save the histogram data
	
S = temp.Data;
outfile = fullfile(fileparts(inputImg), 'voxel_histogram.mat');
save(outfile, 'S');

% describe the output stream
	
aap = aas_desc_outputs(aap, subjectIndex, 'voxel_histogram', outfile);


case 'checkrequirements'

if ~aas_cache_get(aap,'spm'), aas_log(aap,true,'SPM is not found'); end
   
end % switch

end % function

Appendix B: Miscellaneous Useful aa Code Tricks

1. Locate a file in the SPM distribution (such as a mask)

   fp = spm_select('FPListRec',aap.directory_conventions.spmdir,filename);

This will return the fullpath to the named file if it exists in any folder in the SPM distribution. Directories are searched recursively. For example:

fp = spm_select('FPListRec', aap.directory_conventions.spmdir, 'avg152T1.nii);

Returns a full path to the MNI averaged T1 template included with the SPM distribution in the “canonical” directory. This file can then be loaded by passing fp to spm_vol.

2. Running an FSL command

   [err w] = aas_runfslcommand(aap,sprintf('feat %s',fsffn));
   

   if (err) < report error > end

3. Write a message to the Matlab command window, optionally halting the analysis

   Function syntax: aas_log(aap, haltanalysis, msg, optionaltextstyle);

   Example usage:

   aas_log(aap, true, [‘Fatal error in module: ’ mfilename ], ‘r’);

4. Override the default value for a module parameter defined in its xml header

Parameters defined in a module’s .xml header file usually include a default value. For example, aamod_segment8_multichan defines a sampling parameter having a default value of 1 (mm):

<samp desc='Sampling distance (mm)'>1</samp>

The default value can be changed in a tasklist using the extraparameters tag. The following changes samp to 3:

<module><name>aamod_segment8_multichan</name>
	<extraparameters>
		<aap><tasklist><currenttask><settings>
			<samp>3</samp>
		</settings></currenttask></tasklist></aap>
	</extraparameters>
</module>

To reiterate, this code is included in the user's tasklist, not in the aamod_segment8_mutichan header.

Appendix C: Miscellaneous aa Programming Topics

Stream Management Internals

The details of stream management are usually not important for module coding, but a basic understanding can be helpful for the aa developer (and user) to demystify the process. If nothing else, it will help explain the many files generated by aa during an analysis run.

The crux of stream management is associating a named stream with the physical file or files that comprise it. The aa engine does this using a system of plaintext files (suffix: .txt) created in each module working directory. One such file will be created having a name:

stream_<streamname>_inputto_<modulename>.txt 

for each stream defined in the inputstream section of the module header and another having a name:

stream_<streamname>_outputfrom_<modulename>.txt 

for each stream defined in the outputstream section of the module header. Note all module names include a five digit suffix so that each can be uniquely identified even if it is repeated in a tasklist. For example, if aamod_imcalc appears twice in a given tasklist, the first occurrence will be identified as aamod_imcalc_00001 and the second as aamod_imcalc_00002 in all aa internals. Notably, aa uses this naming convention when creating module working directories.

The contents of these stream management files can be examined using any text editor. Here is an example from aamod_smooth (slightly reformatted here for readability):

MD5	 Jw4KC3jegCYB1ew73tptLw == srfMSC12_42580_20170628.nii

The string of random characters is a unique identifier aa generates which is used internally to identify the named data file (this example is a smoothed resliced epi nifti file — not the s and r prefixes in the filename and the .nii extension). This file contains the stream data that the module refers to using a named stream (‘epi’ in this example). The identifier is generated using the MD5 hash algorithm, which is why the file begins with the identifier MD5. If the stream comprises more than one file (e.g., a 3D rather than 4D epi), then the text to the right of == would be a list of filenames.

When a module is executed, aa will look up an identifier file for any input streams for the module, and convert the identifier to the proper filename. It will then either copy the file(s) into the module working directory or create a link in the MWD so that the functions aas_getfiles_bystream and aas_getimages_bystream can find the file. The choice to copy or link is determined by the ismodified tag you assigned to the stream: If a stream is explicitly specified as not modified (i.e., ismodified=‘0’) a link is used, otherwise the file is coped. Note aa uses hard links (see: man ln) not symbolic links (the latter is what the Finder will show as an alias). For all practical purposes, a hard link is identical to the original file except that it does not occupy additional disk space. To verify a file is a hard link and not a copy, check the files inode (serial number) using ls -i.

An analogous process occurs for the output streams named in your module. An MD5 textfile is created when you describe your output stream — that is, when you pass the streamname and filename to aas_desc_outputs in your module.

The cmap file

At the beginning of an analysis, aa constructs a list of data dependencies based on the chain of input and output streams present in the modules appearing in the user tasklist. It passes this information to the scheduling engine, which determines the order in which modules must execute (including parallel execution of modules when using a computing cluster). This list can be viewed in the file <aap.acq_details.root>/aa_cmap.txt.

The “done” file

When your module has finished running, the aa engine will create a donefile in its working directory to indicate module execution completely normally. This is so aa can restart an analysis that did not run to completion without rerunning unnecessary stages of the analysis. The name of the donefile has the format: done_<streamname> where streamname will include a five digit suffix. Like the MD5 stream management files, the donefile is a plain text file (although it does not have a .txt extension). The contents are typically the module execution elapsed time expressed as a single floating point number, although an IP address or other information may be present.

The donefile is generated automatically by the aa engine and does not require any module code.

The .aa Worker Directory

AutomaticAnalysis maintains a (hidden) entry in your home directory named .aa which it populates with a collection of subdirectories called workers. These are generated by aa during analysis and are used for internal bookkeeping purposes. Usually a worker is empty, however some may contain files returned by aa jobs submitted to a computing cluster.

Ordinarily, you will not interact with workers in any way. However, as of this writing, they create a harmless (but annoying) bug that occasionally requires you to restart a crashed analysis twice, the first restart failing with a non existent directory message. Your options are to delete the worker(s) created by the failed analysis and restart or, alternatively, simply restart the analysis a second time which generally fixes the problem. You can have aa automatically delete workers by specifying a nonzero value for the parameter aap.options.aaworkercleanup.

Note the directory ~/.aa does not appear in the Finder. In Terminal, you must use the “all” option (ls -a) for it to be listed.

Modules with no Output Streams

There is no a priori reason a module must define an output stream. For example, a plotting module might take some input stream and plot a figure for the report and nothing else.

Unfortunately, there is currently a bug in the aa scheduler that may cause it to crash if it encounters a module with no output streams. The solution is to simply define a “dummy” output stream in your module:

<outputstreams>
	<stream>dummy</stream>
</outputstreams>

The actual name of the dummy stream is arbitrary. Your script does not have to describe or even create data for the stream; the definition in the xml header is simply a placeholder needed to placate the scheduler.

Multi-domain Modules

Although most modules naturally operate in a single analysis domain (e.g.,study, subject, or session), processing tasks exist that are used in more than one. For example, it is possible to spatially smooth an epi file (domain = session), a structural image (domain = subject), or a parametric map generated by a first- or second-level GLM (domain = subject or session). Rather than providing multiple versions of a module, each operating in a different domain, it is possible to implement multi-domain processing in a single module if the necessary code modifications are included.

In most cases, the key to writing a multi-domain module is proper consideration of the subject and session identifiers. A “session” level module is called once per session per subject, and the module is passed a valid subject and session identifier. A “subject” level module is called once per subject, and is passed only a subject identifier. A “study” level module is called once per analysis and is passed no identifiers. (A subject “identifier” is simply an integer, usually 1,2 … corresponding to the order in which the subjects were defined using aas_addsubject. Similarly, session identifiers are integers corresponding to the order in which sessions were defined using aas_addsession.)

The primary modification concerns retrieval of module inputs and properly describing outputs. The former is handled by aas_getimages_bystream or aas_getfiles_bystream and the latter by aas_desc_outputs. As described earlier, these functions can take a variety of input arguments: session-level modules will pass in a session and subject identifier, subject-level modules will pass only the subject identifier, and study-level module will pass neither. By including appropriately-modified calls to these functions, input and output can be handled properly across domains.

As an example, consider the following code from aamod_smooth.m, which can be used either in the subject or session domain:

function [aap,resp] = aamod_smooth(aap, task, subj, sess)

. . .

if (exist('sess','var'))
P = aas_getfiles_bystream(aap,aap.tasklist.currenttask.domain,
			[subj sess],streams{streamind});
else
P = aas_getfiles_bystream(aap,subj,streams{streamind});
end

By checking whether the session index is defined, the code can determine whether it is currently operating at the subject or session domain. It then calls aas_getfiles_bystream with arguments appropriate for the analysis domain.

The same check is done later in the module when describing its outputs:

if (exist('sess','var'))
  aap = aas_desc_outputs(aap,aap.tasklist.currenttask.domain,
			[subj sess],streams{streamind},outputfns);
else
  aap = aas_desc_outputs(aap,subj,streams{streamind},outputfns);
end

With such modifications, it is then possible to use the same code in different analysis domains. This is usually done by creating multiple xml headers defined for task of interest and (if necessary) aliased to a the mfile. For example, aamod_smooth.m. is used for both epi and parametric map smoothing. Access to this functionality is provided by the headers aamod_smooth.xml and aamod_smooth_spmts.xml, respectively.

Here is the currenttask specifier from aamod_smooth.xml:

<currenttask domain='session' …

Here is the specifier from aamod_smooth_spmts.xml:

<currenttask domain='subject' mfile_alias='aamod_smooth' …

By specifying a subject domain and an mfile alias in the header, the code in aamod_smooth.m can be used to spatially smooth a first-level parametric map (these are generated at the subject level). The remainder of the header may also include definitions specified to the task.