Local Engine for Structured Analysis of Data Exports (named after the minor Sherlock Holmes character, Inspector Lestrade) is an open-source visualization tool that helps users understand their account security history using data exports from online platforms.

Instead of uploading user data files to a server, this project processes everything locally in the browser using Pyodide, a port of CPython to WebAssembly that runs a full Python environment in the web browser.

The Vue frontend is forked and heavily modified from Google's Timesketch, specifically the timesketch/frontend-ng (link) directory. See the License section below.

This repository also includes a set of evaluation scripts (entity_resolution_evaluation/) for measuring how well our Device Entity Resolution pipeline (see python_core/device_grouping2/) determines if two authentication or session records originate from the same identity. The datasets are very large and it's unnecessary run these scripts if you are just interested in exploring the web application. See Device Entity Resolution Evaluation

You can either visit a hosted version of the static LEStrADE site at https://wispr-lab.github.io/data-export-gui/, or run it locally on your own machine. To do the latter:

This project relies on Git submodules for user-agent parsing. You must initialize them first:

git clone --recurse-submodules https://github.com/WISPR-lab/data-export-gui/
# or if already cloned:
git submodule update --init --recursive

Prerequisites: Docker or Docker Desktop installed and running.

docker compose up --build web

The web application will be live at http://localhost:5001.

Prerequisites:

• Node.js 22+ with Yarn v1
• Python 3.12+ (standard CPython)
• uv (brew install uv or pip install uv)

1. Install and run local version of the web app:

   cd webapp
   yarn install
   yarn serve

   Under the hood, this runs sync_assets.sh which automatically builds the UA-Extract-purepy wheel using uv. The frontend runs at http://localhost:5001.

Currently, the tool includes parsing manifests for:

• Google - Fully Supported
• Apple/iCloud - Fully Supported

We are working on support for:

• Facebook - Beta
• Instagram - Beta
• Discord - Beta
• Snapchat - Beta

For instructions on how to request your data exports, see the How to Request Data Guide on our hosted site (or http://localhost:5001/#/how-to-request when running locally). We'll put some anonymized sample data up soon.

When you import your data export file, it is never transmitted over the network; all unzipping, parsing, and database transactions happen entirely inside your local browser sandbox. The codebase does not make external API requests containing your data (such as querying a remote service to parse User Agents or geolocate IP addresses).

Note that the site is hosted via GitHub Pages, which may collect connection logs or track cookies. Furthermore, the Vue app currently loads some CSS assets and Pyodide package wheels from public CDNs, which implies an outbound network request. We are working on bundling these assets from the source and self-hosting our own version of the project soon with better privacy guarantees.

This is optional and requires substantial disk and memory resources; skip it if you only want to run the web app. (todo better explanation)

Running outside of Docker is not recommended. You must change your Docker/VM settings to ensure you have sufficient disk space and memory available.

• Absolute Minimum: 4 GB RAM, 15 GB free disk space.
  • This might crash and will be significantly less efficient.
• Recommended: 8 GB RAM, 25 GB free disk space.

• If you are running Docker Desktop (most users), follow these instructions to change your Memory limit and Disk usage limit to the settings specified above.
• If you are using a different VM, use the instructions below:
  • Colima
  • OrbStack
  • WSL

• If you are not using a VM to run Docker, proceed (but add the --memory="8g" flag").
• If you are using a VM, ensure disk/memory limits meet the requirments. But you probably already know how to do that...

1. Start the container:

      # from data-export-gui dir
      mkdir -p entity_resolution_evaluation/data/{raw,normalized}
      
      # mac/windows OR linux with VM
      docker compose run --rm eval
   
      # if linux w/o VM
      docker compose run --rm eval --memory="8g" # or "4g", etc.

   This will put you in an interactive bash session inside the container (at /workspace). The raw datasets will be downloaded to Docker named volume (data_raw) mounted at entity_resolution_evaluation/data/raw to speed up the SQLite writes.

2. Inside the container shell, download the datasets:

   uv run python -m entity_resolution_evaluation.fetch_data

   The script autodetects container RAM to pick an appropriate chunk size. Override via --chunksize <size> or -c <size> if needed.*

3. TODO...

Feel free to submit UI bugs under Issues or post there if you're interested in contributing to the project. To add support for a new platform (or augment supported keys for an existing one), follow the instructions in the Manifests Schema Guide.

• webapp/: Vue 2 / Vuetify frontend with some JS utility files.
• python_core/: Python parsing and database logic (runs inside Pyodide in the browser).
• manifests/: Platform YAML configurations defining mappings to ECS.
• schema.sql: SQLite database schema. Both JS and Pyodide read/write to this DB, but never at the same time.
• tests/: Vitest (JS) and Pytest (Python) integration tests.

1. Vue worker downloads Pyodide WASM + package wheels (pandas, regex, sqlite3) on load.
2. After the user imports their data export ZIP file in the UI, it is unzipped locally by JS and files are written to the Origin Private File System (OPFS)
3. Python (python_core/) parses the data into a standard representation defined by the YAML schemas in manifests/.
4. Python saves normalized rows to a WASM SQLite database synced to OPFS defined by schema.sql.
5. Vue queries the local SQLite DB to render views.

If you make changes to the python_core logic and would like to run unit tests, run

# via Docker
docker compose run --rm test tests/python  # all python tests
docker compose run --rm test tests/python/test_device_grouping2.py # or a specific test

# without Docker
uv sync
uv run pytest tests/python # all python test
uv run pytest tests/python/test_device_grouping2.py # or a specific test

The Vue frontend is forked and heavily modified from Google's Timesketch, specifically the timesketch/frontend-ng (link) directory., which is licensed under the Apache License 2.0. Files that have been modified from the original repository have been documented as such. This project, too, is protected by the same license. See LICENSE for details.