docs: 🏗️ add a flow diagram and text about processing steps - #207
lwjohnst86 wants to merge 8 commits into
Conversation
f4ae8ac to
e511674
Compare
| completely missing or not. This is to make sure that during joining with other | ||
| forms that no data is overwritten. |
There was a problem hiding this comment.
As you say, visit_id would be null for some resources. It's not immediately clear to me how including this column of nulls would prevent overwriting data. Maybe I don't understand what kind of overwriting or joining you mean here.
There was a problem hiding this comment.
Hmm, I can't recall exactly why I put this. It might not be a problem, but there was something I noticed during reviewing the variables that made me think that we needed this. I should have put the reason here 😛 well, I'll remove for now and we can always fix later.
| - All staged resources will need a `participant_id`, `study_week_id`, and | ||
| `visit_id` during processing, regardless of if those columns will be |
There was a problem hiding this comment.
The second point in this list makes a similar but less strict statement. Are they different points or is only one correct?
| renaming process (after first applying the map between old and new column | ||
| names from the CSV file). Form prefixes or suffixes are indicated by name in | ||
| the column `form_abbrev` of the `raw-to-resource.csv` mapping file. If that | ||
| row is empty, than the name is kept in the `new_form_name` column. |
There was a problem hiding this comment.
| row is empty, than the name is kept in the `new_form_name` column. | |
| column is empty, than the name is kept in the `new_field_name` column. |
Does this simply mean if there is no abbreviation there is nothing to remove?
There was a problem hiding this comment.
This needs to be updated based on your comments from another PR.
| tar["feasibility-data_VERSION.tar"] | ||
| md_json["datapackage.json"] | ||
| resources["resources/*.parquet"] | ||
| staging["staging/**/*.parquet"] |
There was a problem hiding this comment.
Should we also add the raw-to-resource-map.csv file as input to the staging process? If understand correctly, we will split the raw data and make the resources based on this.
One extra piece of information we will need to know when creating the resources is which forms repeat. Those will need a submission_id column added. This could be added to raw-to-resource-map.csv as an extra row for the repeating resource with redcap_repeat_instance as the current_field_name and submission_id as the new_field_name.
Another question is what happens to checkbox fields. In the REDCap metadata, checkbox fields are given as one field with a series of options under select_choices_or_calculations. In the data, each option gets its own column (with a value of checked or not checked). (Same discussion #32 (comment).) Will raw-to-resource-map.csv match the data and list the expanded checkbox fields? See this comment for a concrete example.
There was a problem hiding this comment.
Hmm, good idea about using the REDCap pulled dictionary rather than the original form creation CSV file. There are more columns created from the form than in the raw to resource mapping file (which is based on the REDCap form creator).
| the participant within a form are included in all resources generated from | ||
| that form. They have the resource `*` in the mapping CSV. | ||
|
|
||
| <!-- TODO: Not sure if `*` is the right thing to use here...? --> |
There was a problem hiding this comment.
I think this depends on how we want to use raw-to-resource-map.csv.
If, when splitting the forms, we want to say: for each resource, select all fields belonging to that resource, then it would be nice to have all fields listed for all resources in the CSV. That's why I suggested listing these date fields in separate rows for each resource.
| which needs to be converted to `-10`. | ||
| - `medicine_changes` fields end in `_rep`, which need to be removed during | ||
| renaming. | ||
| - `gruppemde_*` (`group_meetings_*`) end in |
github-project-automation
Bot
moved this from In Review
to In Progress
in Data development
Co-authored-by: martonvago <[email protected]>
This branch has not been deployed
2 participants
Description
This can help us keep an overview of what's going on within the data package, and what things we need to do to get them there.
Needs a thorough review.
Checklist
just run-all