GraphGen: Enhancing Supervised Fine-Tuning for LLMs with Knowledge-Driven Synthetic Data Generation

GraphGen is a framework for synthetic data generation guided by knowledge graphs.

It begins by constructing a fine-grained knowledge graph from the source text，then identifies knowledge gaps in LLMs using the expected calibration error metric, prioritizing the generation of QA pairs that target high-value, long-tail knowledge. Furthermore, GraphGen incorporates multi-hop neighborhood sampling to capture complex relational information and employs style-controlled generation to diversify the resulting QA data.

After data generation, you can use the generated data to finetune your LLMs.

We support various LLM inference servers, API servers, inference clients, input file formats, data modalities, output data formats, and output data types. Users can flexibly configure according to the needs of synthetic data.

Inference Server	Api Server	Inference Client	Data Source	Data Modal	Data Type
HF SGLang vllm	Silicon OpenAI Azure	HTTP Ollama OpenAI	Files(CSV, JSON, PDF, TXT, etc.) Databases(UniProt, NCBI, RNAcentral) Search Engines(Bing, Google) Knowledge Graphs(Wikipedia)	TEXT IMAGE	Aggregated Atomic CoT Multi-hop VQA

1. Install uv

   # You could try pipx or pip to install uv when meet network issues, refer the uv doc for more details
   curl -LsSf https://astral.sh/uv/install.sh | sh

2. Clone the repository

3. Create a new uv environment

    uv venv --python 3.10

4. Configure the dependencies

   uv pip install -r requirements.txt

python -m webui.app

For hot-reload during development, run

PYTHONPATH=. gradio webui/app.py

1. Configure the environment

   • Create an .env file in the root directory

     cp .env.example .env

   • Set the following environment variables:

      # Tokenizer
      TOKENIZER_MODEL=
      
      # LLM
      # Support different backends: http_api, openai_api, ollama_api, ollama, huggingface, tgi, sglang, tensorrt
      # Synthesizer is the model used to construct KG and generate data
      # Trainee is the model used to train with the generated data
     
      # http_api / openai_api
      SYNTHESIZER_BACKEND=openai_api
      SYNTHESIZER_MODEL=gpt-4o-mini
      SYNTHESIZER_BASE_URL=
      SYNTHESIZER_API_KEY=
      TRAINEE_BACKEND=openai_api
      TRAINEE_MODEL=gpt-4o-mini
      TRAINEE_BASE_URL=
      TRAINEE_API_KEY=
      
      # azure_openai_api
      # SYNTHESIZER_BACKEND=azure_openai_api
      # The following is the same as your "Deployment name" in Azure
      # SYNTHESIZER_MODEL=<your-deployment-name>
      # SYNTHESIZER_BASE_URL=https://<your-resource-name>.openai.azure.com/openai/deployments/<your-deployment-name>/chat/completions
      # SYNTHESIZER_API_KEY=
      # SYNTHESIZER_API_VERSION=<api-version>
      
      # # ollama_api
      # SYNTHESIZER_BACKEND=ollama_api
      # SYNTHESIZER_MODEL=gemma3
      # SYNTHESIZER_BASE_URL=http://localhost:11434
      #
      # Note: TRAINEE with ollama_api backend is not supported yet as ollama_api does not support logprobs.
      
      # # huggingface
      # SYNTHESIZER_BACKEND=huggingface
      # SYNTHESIZER_MODEL=Qwen/Qwen2.5-0.5B-Instruct
      #
      # TRAINEE_BACKEND=huggingface
      # TRAINEE_MODEL=Qwen/Qwen2.5-0.5B-Instruct
      
      # # sglang
      # SYNTHESIZER_BACKEND=sglang
      # SYNTHESIZER_MODEL=Qwen/Qwen2.5-0.5B-Instruct
      # SYNTHESIZER_TP_SIZE=1
      # SYNTHESIZER_NUM_GPUS=1
      
      # TRAINEE_BACKEND=sglang
      # TRAINEE_MODEL=Qwen/Qwen2.5-0.5B-Instruct
      # SYNTHESIZER_TP_SIZE=1
      # SYNTHESIZER_NUM_GPUS=1
      
      # # vllm
      # SYNTHESIZER_BACKEND=vllm
      # SYNTHESIZER_MODEL=Qwen/Qwen2.5-0.5B-Instruct
      # SYNTHESIZER_NUM_GPUS=1
      
      # TRAINEE_BACKEND=vllm
      # TRAINEE_MODEL=Qwen/Qwen2.5-0.5B-Instruct
      # TRAINEE_NUM_GPUS=1

2. (Optional) Customize generation parameters in config.yaml .

   Edit the corresponding YAML file, e.g.:

     # examples/generate/generate_aggregated_qa/aggregated_config.yaml
     global_params:
       working_dir: cache
       graph_backend: kuzu # graph database backend, support: kuzu, networkx
       kv_backend: rocksdb # key-value store backend, support: rocksdb, json_kv
   
     nodes:
       - id: read_files # id is unique in the pipeline, and can be referenced by other steps
         op_name: read
         type: source
         dependencies: []
         params:
           input_path:
             - examples/input_examples/jsonl_demo.jsonl # input file path, support json, jsonl, txt, pdf. See examples/input_examples for examples
   
     # additional settings...

3. Generate data

   Pick the desired format and run the matching script:

   Format	Script to run	Notes
   atomic	bash examples/generate/generate_atomic_qa/generate_atomic.sh	Atomic Q&A pairs covering basic knowledge
   aggregated	bash examples/generate/generate_aggregated_qa/generate_aggregated.sh	Aggregated Q&A pairs incorporating complex, integrated knowledge
   multi-hop	examples/generate/generate_multi_hop_qa/generate_multi_hop.sh	Multi-hop reasoning Q&A pairs

4. Get the generated data

   ls cache/output

1. Build the Docker image

   docker build -t graphgen .

2. Run the Docker container

    docker run -p 7860:7860 graphgen