The goal of this project is to automate the process of extracting data, correlations, key features, and conclusions from a database of research papers. Data is parsed in the form of text as well as quantitative charts and tables.
The model draws conclusions from the text, and converts any numeric data into data structures for further analysis via machine learning.
The textual conclusions are compared with the output of the analytical model to verify results and provide key insights, as well as citations pointing to specific areas in the source material that can be used as evidence.
The model is written in Python with the following dependencies:
pdf2image
pytesseract
tabula
- Install Dependencies using pip/conda/brew
- Place papers in single folder called Papers (or rename in code as needed)
- Create empty folder called ParsedPapers (or rename in code as needed)
- Run readPaper.py
PDF text (as .txt) and Table data (.csv) files will be saved to ParsedPapers folder as ' (TEXT).txt' and ' (TABLES).csv'
- ChatGPT API to summarize findings (piecemeal)
- Pandas DF to run correlation analysis