Welcome to the SEMOSS Backend Documentation. This collection of documents aims to provide developers with a comprehensive understanding of the SEMOSS platform's architecture, core concepts, services, engine integrations, and development practices.

1. Introduction to SEMOSS
   • What is SEMOSS?
   • Core Capabilities
2. Backend Architecture Overview
   • Key Backend Components
   • Data Storage and Interaction
   • High-Level API Request Flow

1. Pixel: The SEMOSS Query Language
   • Core Purposes, Syntax Basics, Data Types, Control Flow, Execution Model
   • Pixel and Reactor Interaction
2. Reactor Framework
   • Core Interfaces/Classes (IReactor.java, AbstractReactor.java)
   • Key Reactor Examples and Roles
   • Reactor Inputs and NounStore
   • Reactor Outputs and NounMetadata
   • Reactor Results and UI Interaction
3. Engine Abstraction in SEMOSS
   • The IEngine Interface
   • Engine Management (Configuration, Registration, Instantiation, Usage)
4. DataFrames and QueryStructs
   • ITableDataFrame and AbstractTableDataFrame
   • Key Frame Implementations
   • SelectQueryStruct Overview
   • Interaction between Engines and DataFrames
5. The Insight Object
   • Purpose and Key Responsibilities
   • Key Information Held
   • How Insights are Used

(Located in the ./engines/ directory)

1. DATABASE Engines
   • Core Concepts, SelectQueryStruct Deep Dive, Examples, Extension Guide
2. STORAGE Engines
   • Core Concepts, Examples, Extension Guide
3. MODEL Engines
   • Core Concepts, Examples, Extension Guide
4. VECTOR Engines
   • Core Concepts, Document Ingestion Flow, Custom Processors, Examples, Extension Guide
5. FUNCTION Engines
   • Core Concepts, Examples, Extension Guide
6. PROJECT Engines

1. SEMOSS Internal Databases
   • Overview, LocalMasterDatabase, PromptDatabase, ModelInferenceLogsDatabase, Configuration/Access.
2. Authentication and Authorization
   • Core Security Objects, AuthN/AuthZ Flows, Key Utilities.
3. Java-Python Communication
   • Overview & Visual Flow Placeholder
   • Java-Side Components
   • Python-Side Components (including chroot & SymlinkHelper details)
   • Communication Protocol

This section describes how SEMOSS interacts with other related systems or components.

• SEMOSS Monolith Interaction: Explains the relationship and data flow between the core SEMOSS platform and the semoss/monolith web application, including details on the runPixel API endpoint.

1. Overview of SEMOSS in Cloud/Cluster Environments
   • Rationale: Scalability, High Availability, Shared Access
   • Key Components: Central Storage, ZooKeeper Synchronization
2. Centralized Cloud Storage for Assets
   • CentralCloudStorage.java: Purpose and Design
   • Storage Backend Abstraction (S3, Azure, GCP, etc.)
   • Asset Synchronization (Engines, Projects, Insights)
   • Locking and Coordination
3. Cluster Synchronization with ZooKeeper
   • ClusterUtil.java and ClusterSynchronizer.java
   • Role of ZooKeeper: Change Notification, State Coordination
   • Synchronization Workflow and Configuration
4. Docker Deployment Guide
   • Introduction to Dockerized SEMOSS
   • Details on Core Dockerfiles
   • Build Process, Environment Variables, Volume Mounts, Example Usage
5. Docker Configuration Details
   • Configuring SEMOSS via Docker, semoss-artifacts, Environment Variables.
6. GitHub Actions Workflows for CI/CD
   • CI, Python/Tomcat Builders, Application Image Builders.

The genai_client Python package provides a standardized interface for interacting with various generative AI models and services, including text generation, embedding generation, and tokenization.

• GenAI Client Package Documentation
  • Details on:
    • Core Files (constants.py, utils.py)
    • Model Keys (ModelKeysEnum)
    • Model Limits (Context Windows, Max Tokens)
    • Client Initializers & Wrappers
    • Embedders Overview
    • Text Generation Clients Overview
    • Tokenizers Overview

This collection of Python modules provides the tools and underlying server infrastructure for SEMOSS's Generative AI Agent Services. It includes engines for database, model, function, storage, and vector interactions, as well as server components and security utilities.

• GAAS Tools & Components Documentation

• Development Guides
  • Overview of configurations, onboarding, etc.

This section provides practical guides and code-centric examples for developing with and using SEMOSS. Topics include writing custom Reactors, interacting with databases, working with DataFrames, advanced Pixel scripting, and more.

• SEMOSS How-To Guides