Skip to content

Commit dae7ea4

Browse files
jsgreenwelljsgreenwell
committed
Changed project structure
0 parents  commit dae7ea4

8 files changed

Lines changed: 82 additions & 0 deletions

File tree

‎ETL/__init__.py‎

Whitespace-only changes.

‎ETL/__pycache__/__init__.cpython-37.pyc‎

134 Bytes
Binary file not shown.

‎ETL/__pycache__/fileparser.cpython-37.pyc‎

695 Bytes
Binary file not shown.

‎ETL/data/student_poll.json‎

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,28 @@
1+
[
2+
{
3+
"grade": "freshman",
4+
"TA": false,
5+
"degree": "Software Engineering",
6+
"gender": "N",
7+
"professor_traits": [
8+
"passion", "knowledge", "honest", "caring", "technology"],
9+
"peer_traits": [
10+
"honest", "hard working", "determined", "intelligent", "technology"],
11+
"working": false,
12+
"GPA": 3.1,
13+
"add_anything": ""
14+
},
15+
{
16+
"grade": "sophmore",
17+
"TA": false,
18+
"degree": "Civil Engineering",
19+
"gender": "F",
20+
"professor_traits": [
21+
"knowledge", "experience", "intelligent", "caring", "group tasks"],
22+
"peer_traits": [
23+
"intelligent", "determined", "healthy", "creative", "caring"],
24+
"working": false,
25+
"GPA": 3.7,
26+
"add_anything": ""
27+
}
28+
]

‎ETL/data_notes.md‎

Lines changed: 35 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,35 @@
1+
## Notes on various schema
2+
-----
3+
4+
### JSON - List of dictionaries
5+
Based on results of poll given to students (quanitative except last question).
6+
* grade
7+
- freshman to senior (full word)
8+
- gr = graduate
9+
- pro = professor
10+
- staff (academic based)
11+
* degree
12+
- Not offered to Staff
13+
- Professor degree = main degree teaching
14+
* TA
15+
- boolean value: True for TAs & RAs
16+
* gender options:
17+
- M = Male
18+
- F = Female
19+
- B = Non-binary
20+
- N = Not disclosed
21+
* professor_traits:
22+
- ranking of various words (traits) from provided list of most important to least
23+
* peer_traits:
24+
- ranking of traits from list from most to least
25+
- only students were offered this option
26+
* Working
27+
- both Boolean value which indicate working (full or part)
28+
* GPA
29+
- floating point current GPA value
30+
* add_anything:
31+
- Final open-ended question: data varies
32+
33+
ADD more later
34+
35+
-----

‎ETL/fileparser.py‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
import pathlib
2+
3+
class FileParser:
4+
def __init__(self):
5+
self._data_dir = pathlib.WindowsPath(r'C:\Spark\DataAnalysis\ETL-basic\ETL\data')
6+
7+
def check_file(self, file="survey.txt"):
8+
file = self._data_dir / file
9+
return file.exists()

‎ETL/readme.md‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
# Basic-ETL
2+
3+
Full project for basic ETL - Add notes later (licensed under MIT)

‎ETL/test_FileParser.py‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
import pytest
2+
from ETL.fileparser import FileParser
3+
4+
# This will be replaced by specific fileParser unit tests (as it will be the abstract)
5+
def test_check_file():
6+
fp = FileParser()
7+
assert fp.check_file("student_poll.json") is True

0 commit comments

Comments
 (0)