pavanjava · Oct 12, 2024 · Oct 10, 2024 · Oct 10, 2024 · Oct 11, 2024 · Oct 11, 2024
diff --git a/README.md b/README.md
@@ -1,6 +1,6 @@
 ![Banner](https://raw.githubusercontent.com/pavanjava/bootstrap-rag/refs/heads/main/assets/bootstrap-rag.png)
 # bootstrap-rag
-this project will bootstrap and scaffold the projects for specific semantic search and RAG applications along with regular boiler plate code.
+This project will bootstrap and scaffold the projects for specific semantic search and RAG applications along with regular boilerplate code.
 
 ### Architecture
 ![Arch](assets/architecture.png)

diff --git a/bootstraprag/templates/evaluations/mlflow_evals/basic_evaluations.py b/bootstraprag/templates/evaluations/mlflow_evals/basic_evaluations.py
@@ -0,0 +1,34 @@
+import mlflow
+import openai
+import pandas as pd
+from dotenv import load_dotenv, find_dotenv
+
+load_dotenv(find_dotenv())
+
+eval_data = pd.read_csv("input_data.csv")
+
+with mlflow.start_run() as run:
+    system_prompt = "Answer the following question in two sentences"
+    # Wrap "gpt-4" as an MLflow model.
+    logged_model_info = mlflow.openai.log_model(
+        model="gpt-4",
+        task=openai.chat.completions,
+        artifact_path="model",
+        messages=[
+            {"role": "system", "content": system_prompt},
+            {"role": "user", "content": "{question}"},
+        ],
+    )
+
+    # Use predefined question-answering metrics to evaluate our model.
+    results = mlflow.evaluate(
+        logged_model_info.model_uri,
+        eval_data,
+        targets="ground_truth",
+        model_type="question-answering",
+    )
+    print(f"See aggregated evaluation results below: \n{results.metrics}")
+
+    # Evaluation result for each data record is available in `results.tables`.
+    eval_table = results.tables["eval_results_table"]
+    print(f"See evaluation table below: \n{eval_table}")
diff --git a/bootstraprag/templates/evaluations/mlflow_evals/input_data.csv b/bootstraprag/templates/evaluations/mlflow_evals/input_data.csv
@@ -0,0 +1,3 @@
+,inputs,ground_truth
+0,What is MLflow?,"MLflow is an open-source platform for managing the end-to-end machine learning (ML) lifecycle. It was developed by Databricks, a company that specializes in big data and machine learning solutions. MLflow is designed to address the challenges that data scientists and machine learning engineers face when developing, training, and deploying machine learning models."
+1,What is Spark?,"Apache Spark is an open-source, distributed computing system designed for big data processing and analytics. It was developed in response to limitations of the Hadoop MapReduce computing model, offering improvements in speed and ease of use. Spark provides libraries for various tasks such as data ingestion, processing, and analysis through its components like Spark SQL for structured data, Spark Streaming for real-time data processing, and MLlib for machine learning tasks"
diff --git a/bootstraprag/templates/evaluations/mlflow_evals/requirements.txt b/bootstraprag/templates/evaluations/mlflow_evals/requirements.txt
@@ -0,0 +1,3 @@
+mlflow==2.16.2
+python-dotenv==1.0.1
+pandas==2.2.3
diff --git a/bootstraprag/templates/evaluations/phoenix_evals/readme.md b/bootstraprag/templates/evaluations/phoenix_evals/readme.md
@@ -1,2 +1,23 @@
 ## Phoenix Evaluations
-- Under development
+
+This repository provides a script for evaluating model-generated responses using Phoenix's `HallucinationEvaluator` and `QAEvaluator`.
+To start evaluating run `bootstraprag create phoenix_evals`
+select the specific template shown on cli
+
+```text
+? Which technology would you like to use? standalone-evaluations
+? Which template would you like to use? 
+  deep-evals
+  mlflow-evals
+❯ phoenix-evals
+  ragas-evals
+```
+
+just replace the `input_data.csv` with your own data, the file has following columns
+`id,reference,query,response`.
+
+### How to execute?
+- run `python basic_evaluations.py`
+
+### What to expect?
+- At the end of process, you can see the `evaluation_report.csv` is created and kept in the parent folder where you can see different aspects of evaluations carried on your input data.