Skip to content

Add S3 Connection to the ragapp docker and reindex api #221

Open
mrt2081 wants to merge 62 commits into
ragapp:mainfrom
mrt2081:main
Open

Add S3 Connection to the ragapp docker and reindex api #221
mrt2081 wants to merge 62 commits into
ragapp:mainfrom
mrt2081:main

Conversation

@mrt2081
Copy link
Copy Markdown
Contributor

@mrt2081 mrt2081 commented Sep 27, 2024

  1. Add Ability to connect to S3 on the docker boot process:
    image
  2. Add S3 Path and Hide the upload file when the docker is connected to S3
    image
  3. Add ReIndex functionality on frontend and backend

MoeDigilite and others added 12 commits September 20, 2024 16:18
- Updated `FileHandler.upload_file` to accept `fileIndex` and `totalFiles` parameters and index data only when the last file is uploaded.
- Modified `add_file` endpoint in FastAPI to accept `fileIndex` and `totalFiles` as form data.
- Enhanced the file upload logic in the frontend to include `fileIndex` and `totalFiles` in the form data.
- Improved user feedback during file operations in the frontend.

These changes ensure better handling of file uploads and removals, providing a more robust and user-friendly experience.
- Introduced new fields in RAGAppContainerConfig for S3 integration:
  - connectToExternalData
  - s3BucketName
  - s3AccessKey
  - s3SecretKey
  - s3Url

- Updated ragappFormSchema to include validation for new S3 fields.
- Enhanced CreateAgentForm to conditionally render S3 fields based on external data connection.
- Modified service creation to include connectToExternalData option.
- Updated RAGAppContainerConfig to include additional Docker creation parameters for external data connections.
- Added methods to manage S3 environment variables and configurations.
- Modified Dockerfile to install FUSE for S3FS support.
- Integrated S3 configuration into the management router with new API endpoints.
- Updated entrypoint script to mount S3 bucket if enabled.
- Improved code formatting and comments for better readability.
- Updated Dockerfile to copy backend patch files.
- Enhanced S3ConfigSchema in `s3Config.ts` to include `s3_enabled`, `s3_bucket`, and `s3_url`.
- Modified `fileLoader.tsx` to conditionally render components based on S3 configuration and added reindexing button.
- Corrected error message in `s3Config.tsx` and updated form fields to reflect S3 configuration.
- Added `data_dir` field to `FileLoader` model in `loader.py`.
- Extended `S3Config` model in `s3_config.py` to include `s3_enabled`, `s3_bucket`, and `s3_url`.
- Updated `config.py` to set environment variable for S3 path and update loader configuration.
- Added reindexing endpoint in `files.py`.
- Modified `entrypoint.sh` to ensure S3 keys are present in the `.env` file.
@changeset-bot
Copy link
Copy Markdown

changeset-bot Bot commented Sep 27, 2024

🦋 Changeset detected

Latest commit: 35e9fd7

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 1 package
Name Type
ragbox Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

MoeDigilite and others added 17 commits September 27, 2024 12:39
- Unpacked the list `parts` using the `*` operator in `os.path.join` to fix the error.
- Added `s3_path_meta_files` to S3ConfigSchema in `s3Config.ts`.
- Updated `S3Config` component to include a form field for `s3_path_meta_files`.
- Added `s3_path_meta_files` to `S3Config` model in `s3_config.py`.
- Renamed `default_file_metadata_func` to `extract_file_metadata_func` in `generate.py`.
- Implemented logic to read metadata from a JSON file in the S3 path if it exists.
- Updated `generate_datasource` to use `extract_file_metadata_func`.

This commit addresses the issue where `os.path.join` was incorrectly used, causing errors in file path construction.
- Updated .gitignore to include Docker-related files.
- Modified Makefile to add a new target for the API wrapper.
- Enhanced .env file with a new variable for WRAPPER_DOMAIN.
- Added configuration for the API wrapper service in docker-compose.yml.
- Updated full_documentation.json with a new generation date.
- Adjusted schema to include 'configured' property in various sections.
- Removed 'Grok' from the assistant type options for consistency.

This commit introduces the API wrapper service and improves project documentation and configuration.
- Update @strapi/strapi dependency from "5.1.0" to "^5.1.0" to allow minor version updates
- Implement new chat function in container.js controller with RAG integration
- Add validation for required chat fields
- Integrate service category and province information retrieval
- Add RAG API communication with OpenAI payload formatting
- Implement error handling for API requests

Technical Details:
- Updated package.json and package-lock.json with new Strapi version
- Added chat endpoint in container controller
- Implemented token-based authentication for RAG API
- Updated documentation generation date

Breaking Changes: None
- Add comprehensive JSDoc documentation for the chat function
- Clean up commented authorization code
- Improve code formatting for container endpoint construction
mrt2081 and others added 30 commits January 18, 2025 16:34
- Added a new delete method in the assistant controller to handle comprehensive assistant and container deletion
- Implemented a list method to retrieve assistants with detailed related data
- Created a new route for listing assistants
- Extended RAG API service with a deleteRAGApi method for service deletion
- Updated documentation generation date

This commit improves assistant resource management by providing more robust deletion and retrieval capabilities.
- Implemented a new `updateAgents` method in the assistant controller to update agent instructions
- Created a new route `/assistants/:id/update-agents` for updating agent instructions
- Added comprehensive error handling and result tracking for agent updates
- Updated assistant and container metadata with new instructions
- Ensured atomic updates across multiple containers and agents

This commit enhances the assistant management capabilities by providing a flexible mechanism to update agent instructions across multiple containers.
- Adjusted formatting in docker-compose.yml for better readability.
- Enhanced App.jsx to support WebSocket connections with updated CORS settings.
- Refactored message handling logic to improve clarity and maintainability.
- Updated connection status display and ensured consistent string formatting throughout the file.

These changes improve the application's real-time chat functionality and maintain code consistency.
- Separated HTTP and WebSocket services and routers for better clarity and management.
- Defined specific rules and TLS configurations for both HTTP and WebSocket traffic.
- Enhanced service naming for consistency and easier identification.

These changes optimize the Traefik setup for handling different types of traffic, improving the application's deployment structure.
- Activated the WebSocket service configuration by uncommenting relevant lines in docker-compose.yml.
- Ensured the WebSocket service uses the correct scheme and transport settings for compatibility with the backend.

These changes improve the Traefik setup for handling WebSocket traffic, enhancing real-time communication capabilities.
…nce CORS settings

- Refined the HTTP and WebSocket service configurations in docker-compose.yml for better clarity and management.
- Adjusted router priorities to ensure proper traffic handling between HTTP and WebSocket services.
- Enhanced CORS settings in index.js to allow requests from specific origins, improving security and compatibility.

These changes optimize the deployment structure and enhance the application's real-time communication capabilities.
…date API client settings

- Consolidated HTTP and WebSocket service configurations into a single service for the api-wrapper in docker-compose.yml, improving clarity and management.
- Updated the App.jsx file to point to the local Strapi URL for development purposes.
- Adjusted the generation date in full_documentation.json to reflect the latest updates.

These changes streamline the deployment setup and enhance the development experience for the API client.
…ts export

- Changed the CORS origin in App.jsx from "http://localhost:5174" to "http://localhost:5173" for correct frontend URL configuration.
- Simplified the requirements export command in the Dockerfile by removing the format specification.

These changes ensure proper development environment setup and streamline the Docker build process.
…es in docker-compose.yml and update Strapi URL in App.jsx

- Separated HTTP and WebSocket router configurations in docker-compose.yml for improved clarity and management.
- Updated the Strapi URL in App.jsx to use HTTPS for secure connections.
- Applied WebSocket middlewares specifically to the WebSocket router, ensuring proper handling of WebSocket traffic.

These changes optimize the deployment setup and enhance the application's real-time communication capabilities.
- Modified the main HTTP router rule for the api-wrapper to simplify traffic handling by removing WebSocket exclusions.
- This change enhances the clarity of the routing configuration and ensures all HTTP traffic is properly managed.

These updates contribute to a more streamlined deployment setup.
- Added a dedicated router for Strapi's node_modules to handle requests separately from other HTTP traffic.
- Adjusted the main HTTP router rule to exclude the node_modules path, ensuring proper routing for all other requests.
- Applied headers middleware to both the main HTTP router and the new modules router for consistent request handling.

These changes improve the routing clarity and enhance the management of HTTP traffic in the deployment setup.
…roller

- Introduced console logging for service category and province to aid in debugging and monitoring.
- Updated the retrieval of the active provider setting to streamline the process and improve clarity.

These changes enhance the visibility of key variables during execution and simplify the configuration retrieval process.
…active provider retrieval in container.js

- Changed serviceCategoryId from "1" to "20" and provinceId from "1" to "2" in App.jsx for correct configuration.
- Improved the retrieval of the active provider setting in container.js by safely accessing its value.

These updates ensure accurate configuration and enhance the robustness of the active provider setting retrieval.
…vider retrieval in container.js

- Changed the Strapi URL in App.jsx to point to the local development server.
- Updated serviceCategoryId from "20" to "2" and provinceId from "2" to "9" for correct configuration.
- Improved the retrieval of the active provider setting in container.js for better safety and clarity.

These changes ensure accurate configuration for local development and enhance the robustness of the active provider setting retrieval.
- Updated Keycloak service command to include Java options for IPv4 preference and memory settings.
- Added resource limits and reservations for memory to optimize performance.

These changes improve the Keycloak service configuration for better resource management and stability.
- Removed Java options from the Keycloak command and added them as environment variables for better clarity and management.
- This change enhances the configuration by separating command execution from environment settings, improving maintainability.
- Implemented a restart policy set to 'unless-stopped' for the traefik, keycloak, manager, and api-wrapper services.
- This change ensures that the services automatically restart unless explicitly stopped, improving reliability and uptime in the deployment environment.
- Updated the assistant creation logic to handle general assistants by introducing an `isGeneral` flag.
- Modified validation to require province and service category only for non-general assistants.
- Adjusted the assistant name generation to differentiate between general and specific assistants.
- Improved S3 path handling in the agent service based on the assistant type.

These changes improve the flexibility and correctness of the assistant creation process.
…sistants

- Introduced an `isGeneral` flag in both the assistant and container controllers to differentiate between general and specific assistants.
- Modified validation logic to require service category and province only for non-general assistants.
- Adjusted endpoint generation in the container controller based on the assistant type, improving flexibility in handling requests.

These changes enhance the functionality and correctness of the assistant and container management processes.
- Modified npm install command to include --legacy-peer-deps for compatibility.
- Added a step to clear existing Strapi cache before building the admin panel.
- Changed the CMD to use "start" instead of "develop" for a production-like environment.

These changes enhance the Docker setup for the Strapi application, ensuring better dependency management and optimized build processes.
- Commented out the Traefik router rules for Strapi's node_modules and main HTTP traffic to prevent conflicts.
- Added a PUBLIC_URL configuration and an admin URL to the server settings for better deployment flexibility.

These changes improve the routing configuration and enhance the server setup for the Strapi application.
- Removed the commented-out rule for Strapi's node_modules in the main HTTP traffic router configuration.
- This change simplifies the routing setup for the api-wrapper service, ensuring proper handling of HTTP traffic.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants