Skip to content

Add EChemDBReader for reading digitized CV data from echemdb.org#194

Merged
ScottSoren merged 48 commits into
mainfrom
echemdb
Aug 25, 2025
Merged

Add EChemDBReader for reading digitized CV data from echemdb.org#194
ScottSoren merged 48 commits into
mainfrom
echemdb

Conversation

@FrederikLizakJohansen

@FrederikLizakJohansen FrederikLizakJohansen commented Aug 5, 2025

Copy link
Copy Markdown
Contributor

Summary

This PR introduces a new EChemDBReader class to support loading CV reference data from echemdb.org, a curated open-access repository for digitized electrochemical datasets (digitized from SVG extractions). This data is versioned and distributed via GitHub releases and PyPI metadata.

This PR also includes:

  • Minor CI improvements (Python 3.7 discontinued support on Ubuntu >22.04, at least through GitHub actions)
  • A new cross-platform caching utility (get_default_cache_dir) in tools.py

Key Features

EChemDBReader class

  • Fetches digitized CV data by identifier (e.g. "alves_2011_electrochemistry_6010")
  • Supports latest or pinned versions of the dataset (version="0.4.1")
  • Downloads and caches only necessary files (CSV, JSON, bib) at read-time.
  • Uses metadata schema to infer CV axes and set appropriate aliases (potential, current)
  • Converts EChemDB data into TimeSeries + ValueSeries using schema. Each read-call fetches a single CV.

Demo script (demo_echemdb_reader.py)

  • Shows loading of CVs from EChemDB.
  • Demonstrates version-switching.

Additional Improvements

Tools

  • get_default_cache_dir() (in src/ixdat/tools.py)
    • Determines appropriate cache directory based on platform:
      • Windows --> LOCALAPPDATA
      • macOS --> ~/Library/Caches
      • Linux --> $XDG_CACHE_HOME or ~/.cache
    • Supports optional subdirectory input for namespacing/versioning
    • Used by EChemDBReader to store cached datasets

CI / Testing

  • Restricting testing on Python 3.7 to Ubuntu 22.04
    • Needed due to setup-python compatibility issues
    • Ensures continued test coverage for Python 3.7

CI-related changes are required for the EChemDBReader to pass tests. If preferred, they could be split into a separate PR.

For Review

  • New reader: src/ixdat/readers/echemdb.py
  • Demo: development_scripts/reader_demonstrators/demo_echemdb_reader.py
  • Caching utility: src/ixdat/tools.py --> get_default_cache_dir()
  • CI updates in workflows

ScottSoren and others added 25 commits August 4, 2025 11:12
@ScottSoren
skeleton echemdb reader
3b2ee72
@FrederikLizakJohansen
tools: add get_default_cache_dir() for cross-platform user caching
5a4d1f5
@FrederikLizakJohansen
demo: add script to demonstrate loading echemdb dataset with local ca…
20130e6 
…ching
@FrederikLizakJohansen
demo: refine EChemDBReader test to compare local and remote CVs from …
350e2e3 
…echemdb
@FrederikLizakJohansen
fix: remove trailing blank line
d39a2a6
@FrederikLizakJohansen
ci: pin Python 3.7 to 3.7.17 to fix Github Actions setup issue
8cf9486
@FrederikLizakJohansen
style: apply black formatting to echemdb scripts
29bda40
@FrederikLizakJohansen
ci: switch to setup-python@v4 to enable Python 3.7
49d1cf7
@FrederikLizakJohansen
ci: switch to setup-python@v4 to enable Python 3.7
ba79568
@FrederikLizakJohansen
style: apply black formatting to echemdb scripts
a905a6a
@FrederikLizakJohansen
Fix CI workflow: restrict Python 3.7 to Ubuntu 22.04 due to runner in…
294053c 
…compatibility

- Switched to actions/setup-python@v4
- Explicitly mapped Python versions to compatible OS using matrix.include
- Ensured Python 3.7 runs only on ubuntu-22.04 since ubuntu-latest (24.04) drops support
@FrederikLizakJohansen
feat(reader): add EChemDBReader for accessing echemdb.org CV datasets
c67a717 
- Supports fetching latest version from PyPI
- Selectively extracts only required files from release zip
- Caches data locally using versioned directories
- Parses metadata and bibtex citation
- Builds Measurement with alias mapping for CV compatibility
@FrederikLizakJohansen
demo: feat on script demonstrating measured vs. reference CV comparis…
c5deb84 
…on with EChemDBReader
@FrederikLizakJohansen
demo: feat on script demonstrating measured vs. reference CV comparis…
9ea176c 
…on with EChemDBReader
@FrederikLizakJohansen
fix(reader/echemdb): resolve lint errors and remove obsolete methods
9a12fbf
@FrederikLizakJohansen
fix(reader/echemdb): replace | with Optional and Union for Python 3.7…
786e45c 
… compatibility
@FrederikLizakJohansen
fix(reader/echemdb): replace builtin generics with typing.List for Py…
a59af8e 
…thon 3.7
@FrederikLizakJohansen
refactor(echemdb): generalize get_meta_value to support fallback keys
69c47c9
@FrederikLizakJohansen
refactor(echemdb): use Path.open with encoding for metadata loading
43c1373
@FrederikLizakJohansen
feat(reader/echemdb): add fallback aliases for potential and current …
9a8e7fd 
…fields
@FrederikLizakJohansen
feat(reader/echemdb): include full raw metadata in
c34ed86
@FrederikLizakJohansen
fix(reader/echemdb): add constant cycle series to avoid false splitti…
b1fa5b6 
…ng of CV data
@FrederikLizakJohansen
chore(demo): apply minor improvements and cleanup
d9d205b
@FrederikLizakJohansen
chore(demo): apply minor linting
99b2edc
@FrederikLizakJohansen
update: bring EChemDBReader to ixdat 0.3.1
de483de
@FrederikLizakJohansen FrederikLizakJohansen mentioned this pull request Aug 6, 2025
Open
@FrederikLizakJohansen
fix(reader/echemdb): improve transparency and readability of metadata…
c869939 
…-derived aliases; resolve potential issue of circular alias lookup
@FrederikLizakJohansen FrederikLizakJohansen mentioned this pull request Aug 18, 2025
Open
ScottSoren
ScottSoren requested changes Aug 19, 2025
View reviewed changes

@ScottSoren ScottSoren left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fantastic work, @FrederikLizakJohansen !
It works well on my computer and all the code fits well with ixdat
I've requested a few additions to comments and docstrings, plus very minor changes of code in the demo script and the reader.

Comment thread development_scripts/reader_demonstrators/demo_echemdb_cache_load_entry.py Outdated
Comment thread development_scripts/reader_demonstrators/demo_echemdb_cache_load_entry.py Outdated
Comment thread development_scripts/reader_demonstrators/demo_echemdb_cache_load_entry.py Outdated
Comment thread development_scripts/reader_demonstrators/demo_echemdb_reader.py Outdated
Comment thread src/ixdat/readers/echemdb.py Outdated
Comment thread src/ixdat/readers/echemdb.py
Comment thread src/ixdat/readers/echemdb.py
Comment thread src/ixdat/readers/echemdb.py
Comment thread src/ixdat/readers/echemdb.py
Comment thread src/ixdat/tools.py
@FrederikLizakJohansen
style(reader/echemdb): rename TypeVar T->M (Measurement) for clarity (#…
9f4695d 
…194)
@FrederikLizakJohansen
style(reader/echemdb): remove spyder author header (#194)
8809d58
@FrederikLizakJohansen
feat(tools): add request_with_retries helper with timeouts, retries, …
2632100 
…and Ctrl-C support
@FrederikLizakJohansen
fix(reader/echemdb): use request_with_retries for robust PyPI/zip dow…
6f4b319 
…nloads with errormsg
@FrederikLizakJohansen
chore(tools): minor flake8 linting
0fca0a6
@FrederikLizakJohansen
chore(readers/echemdb): minor flake8 linting, add notes on doc string
70da4b0
@FrederikLizakJohansen
fix(reader/echemdb): drop from public API; find_cached_paths->_find_c…
b03bd33 
…ached_paths
@FrederikLizakJohansen
fix(reader/echemdb): remove explicit 'return None', not needed
79e3c6e
@FrederikLizakJohansen
fix(tools): remove appname default. appname is now required
b24d4b1
@FrederikLizakJohansen
fix(reader/echemdb): remove fallback aliases. handle explicitely in f…
9670e08 
…uture
@FrederikLizakJohansen
docs(reader/echemdb): clarify EChemDB zip layout and selective extrac…
72e548a 
…tion
@FrederikLizakJohansen
docs(reader/echemdb): document tstamp default at class and use site
abba0aa
@FrederikLizakJohansen
docs(reader/echemdb): comment on metadata locations
797cfef
@FrederikLizakJohansen
remove(demo/echemdb/load): remove old dev-level test script, not actu…
984b3d1 
…al demo for users
@FrederikLizakJohansen
feat(demo/echemdb): show how to calibrate to RHE using metadata
48cb321
@FrederikLizakJohansen
chore(demo/echemdb): apply minor linting
b147be5
@FrederikLizakJohansen
docs(reader/echemdb): clarify metadata split into schema fields, eche…
d977b4e 
…mdb metadata, and bib
@FrederikLizakJohansen
docs(reader/echemdb): clarify metadata split into schema fields, eche…
0bcf7d1 
…mdb metadata, and bib
@ScottSoren

ScottSoren commented Aug 25, 2025

Copy link
Copy Markdown
Member

Looks great! Thanks for the contribution :)
Merging.

@ScottSoren ScottSoren merged commit 40b7ef2 into main Aug 25, 2025
9 checks passed
@FrederikLizakJohansen FrederikLizakJohansen mentioned this pull request Feb 23, 2026
Merged
@DunklesArchipel

DunklesArchipel commented Apr 26, 2026

Copy link
Copy Markdown

Just found out about this integration. Nice to see the reuse of the database. :)

It seems though that you write some code that might be covered by our API. Maybe that simplifies a bit the data extraction.
https://echemdb.github.io/unitpackage/usage/echemdb_usage.html
Feel free to open issues, if our API is missing something that you mare interested in.

@FrederikLizakJohansen FrederikLizakJohansen mentioned this pull request Apr 27, 2026
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ScottSoren ScottSoren Awaiting requested review from ScottSoren

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@FrederikLizakJohansen @ScottSoren @DunklesArchipel