Skip to content

Latest commit

 

History

History
172 lines (135 loc) · 7.86 KB

File metadata and controls

172 lines (135 loc) · 7.86 KB

fe-foundation Module Design Document

1. Background & Motivation

The existing fe-common module has accumulated heavy dependencies over time (Guava, Hadoop, ANTLR, Alibaba MaxCompute SDK, etc.), making it unsuitable as a lightweight shared library for the plugin/SPI ecosystem. When SPI plugin authors depend on fe-common, they are forced to pull in dozens of transitive dependencies that have nothing to do with their plugin logic.

We need a zero-dependency foundation module that sits below fe-common and fe-extension-spi in the dependency hierarchy, providing only the most essential, general-purpose utilities that any module — including SPI plugins — can safely depend on.

2. Naming Decision

Candidate Verdict
fe-foundation Chosen. Clear "below-common" semantics. No ambiguity with existing modules.
fe-base Too generic; widely used in other contexts.
fe-essentials Good semantics but verbose.
fe-primitives Implies primitive types; misleading.
fe-toolkit "Toolkit" implies a heavier helper library rather than a minimal foundation layer.
fe-kernel Conflicts with database "kernel" terminology.

3. Module Positioning

fe-foundation          ← Zero third-party dependencies. Pure JDK utilities.
    │
    ├── fe-extension-spi       ← Plugin contracts (Plugin, PluginFactory, PluginContext)
    │       │
    │       └── fe-extension-loader    ← Plugin classloading & discovery
    │
    ├── fe-common              ← Heavier shared code (Gson, Guava, Hadoop, etc.)
    │
    └── fe-core                ← Main FE module (optimizer, catalog, transaction)

Key principle: fe-foundation has ZERO compile-scope third-party dependencies. Only JDK.

4. Admission Criteria

A class qualifies for fe-foundation if it meets ALL of:

  1. Zero third-party runtime dependencies — only java.* imports
  2. No coupling to Doris business logic — no references to catalog, optimizer, planner, etc.
  3. General-purpose — useful across multiple modules (SPI plugins, fe-common, fe-core, etc.)
  4. Stable API — unlikely to change frequently

5. Package Structure

org.apache.doris.foundation/
├── property/
│   ├── ConnectorProperty.java           # @annotation: marks config fields
│   ├── ConnectorPropertiesUtils.java    # Reflection-based KV → Bean binder
│   ├── StoragePropertiesException.java  # Property-related RuntimeException
│   └── ParamRules.java                  # Fluent parameter validation DSL
├── type/
│   └── ResultOr.java                    # Rust-style Result<T, E> type
├── format/
│   └── FormatOptions.java               # Immutable data formatting config
└── util/
    ├── BitUtil.java                     # Bit manipulation helpers
    ├── ByteBufferUtil.java              # Unsigned ByteBuffer reads
    ├── SerializationUtils.java          # Deep clone via Java serialization
    └── PathUtils.java                   # URI path comparison utilities

6. Classes Included (First Iteration)

6.1 Property Framework (from fe-core)

Class Original Location External Deps Description
ConnectorProperty o.a.d.datasource.property None Runtime annotation for connector config fields
ConnectorPropertiesUtils o.a.d.datasource.property None (Guava/commons-lang3 removed) Reflection-based KV→Bean binding
ParamRules o.a.d.datasource.property None Fluent validation DSL with chained rules
StoragePropertiesException o.a.d.datasource.property.storage.exception None RuntimeException for property errors

6.2 Type Utilities (from fe-common)

Class Original Location External Deps Description
ResultOr<T,E> o.a.d.common None Success-or-error result type

6.3 Format Utilities (from fe-common)

Class Original Location External Deps Description
FormatOptions o.a.d.common None Immutable formatting configuration

6.4 General Utilities (from fe-core)

Class Original Location External Deps Description
BitUtil o.a.d.common.util None log2, power-of-2 rounding
ByteBufferUtil o.a.d.common.util None Unsigned byte buffer reads
SerializationUtils o.a.d.common.util None Deep clone via serialization
PathUtils o.a.d.common.util None URI path comparison with S3 handling

7. Classes NOT Included (and Why)

Class Reason
Pair, Triple Depend on @SerializedName (Gson) for persistence. Moving would require adding Gson or breaking persistence compatibility.
Writable, Codec, CountingDataOutputStream, DataInputBuffer, DataOutputBuffer, etc. Deeply embedded in the persistence layer (194+ importers for Writable). High-risk migration better done in a separate phase.
CloudCredential Depends on commons-lang3. Could be migrated after trivial refactoring.
GZIPUtils, EnvUtils Depend on commons-io / Guava. Could be migrated after trivial refactoring.

8. Serialization Safety Analysis

All 10 classes are SAFE to move (package rename):

  • None implement java.io.Serializable (except StoragePropertiesException via Throwable, but it is never serialized to disk)
  • None are registered in RuntimeTypeAdapterFactory (no class name stored in JSON)
  • None have serialVersionUID
  • All are either annotations, static utility classes, or transient runtime objects
  • No class name is stored in any editlog, metadata image, or checkpoint

9. Migration Strategy

Phase 1 (This PR): Backward-Compatible Migration

  1. Create fe-foundation module with zero dependencies
  2. Copy classes to new org.apache.doris.foundation.* packages
  3. In original locations, replace class bodies with extends/delegation to the foundation class:
    // fe-core: org.apache.doris.datasource.property.ParamRules
    // Now just re-exports the foundation class
    package org.apache.doris.datasource.property;
    public class ParamRules extends org.apache.doris.foundation.property.ParamRules {}
  4. No existing code needs to change import statements
  5. New code should prefer importing from org.apache.doris.foundation.*

Phase 2 (Future): Gradually update imports across the codebase

  • Update import statements in fe-core to use foundation packages directly
  • Deprecate and eventually remove the re-export shims
  • Migrate more classes from fe-common (IO utilities, Pair/Triple after decoupling Gson)

10. Build Configuration

<!-- fe-foundation/pom.xml -->
<artifactId>fe-foundation</artifactId>
<packaging>jar</packaging>
<name>Doris FE Foundation</name>
<description>Zero-dependency foundation utilities for Doris FE modules and SPI plugins</description>

<dependencies>
    <!-- Intentionally empty. This module has ZERO third-party dependencies. -->
</dependencies>

11. Dependency Graph After Migration

fe-foundation (0 deps)
    ↑
    ├── fe-extension-spi (depends on fe-foundation)
    │       ↑
    │       └── fe-extension-loader
    │
    ├── fe-common (depends on fe-foundation + Guava + Gson + Hadoop + ...)
    │       ↑
    │       └── fe-core
    │
    └── fe-core (depends on fe-foundation + fe-common + ...)