Skip to content
BranchesTags

Repository files navigation

GROBID Client Python

PyPI version SWH License

A simple, efficient Python client for GROBID REST services that provides concurrent processing capabilities for PDF documents, reference strings, and patents.

📋 Table of Contents

✨ Features

  • Concurrent Processing: Efficiently process multiple documents in parallel
  • Flexible Input: Process PDF files, text files with references, and XML patents
  • Configurable: Customizable server settings, timeouts, and processing options
  • Command Line & Library: Use as a standalone CLI tool or import into your Python projects
  • Coordinate Extraction: Optional PDF coordinate extraction for precise element positioning
  • Sentence Segmentation: Layout-aware sentence segmentation capabilities
  • JSON Output: Convert TEI XML output to structured JSON format with CORD-19-like structure
  • Markdown Output: Convert TEI XML output to clean Markdown format with structured sections
  • Type Hints: Ships inline type annotations and a py.typed marker (PEP 561) for static type checking
  • Archive Streaming: Process files directly from .zip/.tar/.tar.gz archives without fully decompressing them
  • S3 Streaming: Read PDFs and zips straight from s3:// (range-streamed, no full download) with the optional [s3] extra

📋 Prerequisites

  • Python: 3.8 - 3.13 (tested versions)
  • GROBID Server: A running GROBID service instance

Important

GROBID supports Windows only through Docker containers. See the Docker documentation for details.

🚀 Installation

Choose one of the following installation methods:

PyPI (Recommended)

pip install grobid-client-python

# to stream inputs directly from S3 (s3:// URIs), install the optional 's3' extra:
pip install "grobid-client-python[s3]"

Development Version

pip install git+https://github.com/kermitt2/grobid_client_python.git

Local Development

git clone https://github.com/kermitt2/grobid_client_python
cd grobid_client_python
pip install -e .

⚡ Quick Start

Command Line

# Process PDFs in a directory
grobid_client --input ./pdfs --output ./output processFulltextDocument

# Process with custom server
grobid_client --server https://your-grobid-server.com --input ./pdfs processFulltextDocument

Python Library

from grobid_client.grobid_client import GrobidClient

# Create client instance
client = GrobidClient(config_path="./config.json")

# Process documents
client.process("processFulltextDocument", "/path/to/pdfs", n=10)

📖 Usage

Command Line Interface

The client provides a comprehensive CLI with the following syntax:

grobid_client [OPTIONS] SERVICE

Available Services

Service Description Input Format
processFulltextDocument Extract full document structure PDF files
processHeaderDocument Extract document metadata PDF files
processReferences Extract bibliographic references PDF files
processCitationList Parse citation strings Text files (one citation per line)
processCitationPatentST36 Process patent citations XML ST36 format
processCitationPatentPDF Process patent PDFs PDF files

Common Options

Option Description Default
--input Input directory path Required
--output Output directory path Same as input
--server GROBID server URL http://localhost:8070
--n Concurrency level 10
--config Config file path Optional
--force Overwrite existing files False
--skip_errors Also skip documents that failed in a previous run False
--verbose Enable verbose logging False

Processing Options

Option Description
--generate_ids Generate random XML IDs
--consolidate_header Consolidate header metadata
--consolidate_citations Consolidate bibliographic references
--include_raw_citations Include raw citation text
--include_raw_affiliations Include raw affiliation text
--tei_coordinates Add PDF coordinates to XML
--segment_sentences Segment sentences with coordinates
--flavor Processing flavor for fulltext extraction
--json Convert TEI output to JSON format
--markdown Convert TEI output to Markdown format

Examples

# Basic fulltext processing
grobid_client --input ~/documents --output ~/results processFulltextDocument

# High concurrency with coordinates
grobid_client --input ~/pdfs --output ~/tei --n 20 --tei_coordinates processFulltextDocument

# Process with JSON output
grobid_client --input ~/pdfs --output ~/results --json processFulltextDocument

# Process with Markdown output
grobid_client --input ~/pdfs --output ~/results --markdown processFulltextDocument

# Process citations with custom server
grobid_client --server https://grobid.example.com --input ~/citations.txt processCitationList

# Force reprocessing with sentence segmentation and JSON output
grobid_client --input ~/docs --force --segment_sentences --json processFulltextDocument

# Resume an interrupted run without retrying the documents that already failed
grobid_client --input ~/docs --output ~/results --skip_errors processFulltextDocument

# Process PDFs directly from a zip or tar.gz archive (streamed, not fully decompressed)
grobid_client --input ~/papers.zip --output ~/results processFulltextDocument
grobid_client --input ~/papers.tar.gz --output ~/results processFulltextDocument

# --input also accepts glob patterns (quote them so the shell does not expand them)
grobid_client --input "~/papers/*.zip"    --output ~/results processFulltextDocument   # many archives
grobid_client --input "~/data/**/*.pdf"   --output ~/results processFulltextDocument   # PDFs in subdirectories

Note

--input accepts a directory, a single file, an archive, or a glob pattern:

  • Archives (.zip, .tar, .tar.gz/.tgz, .tar.bz2/.tbz2) are streamed: eligible entries are extracted in chunks of batch_size to a temporary directory, sent to GROBID, written to --output, and deleted before the next chunk. The archive is never fully decompressed, so disk usage stays bounded. If --output is omitted, results go to a directory named after the archive (e.g. papers.zip → papers/).
  • Glob patterns (paper.zip, paper*.zip, **/paper*.zip, **/*.pdf, …) are expanded with ** recursion; each match is handled by type (archive → streamed, directory → recursed, file → processed). Quote the pattern so your shell passes it through to the client unexpanded.
  • S3 (requires pip install "grobid-client-python[s3]"): pass an s3:// object, prefix or glob. A remote zip is range-streamed (only its central directory and the entries are fetched — never the whole object); loose remote PDFs are fetched a batch at a time. Credentials use the standard AWS chain (env vars / ~/.aws / IAM role). 
    grobid_client --input "s3://my-bucket/papers/2021.zip"  --output ~/out processFulltextDocument   # one remote zip
grobid_client --input "s3://my-bucket/pdfs/*.pdf"        --output ~/out processFulltextDocument   # loose PDFs
grobid_client --input "s3://my-bucket/zips/"             --output ~/out processFulltextDocument   # every object under a prefix

A manifest of paths (local, glob or s3://, one per line, # comments allowed) can be processed together via --input-list paths.txt (combinable with --input).

Note

Skipping already handled documents. By default a re-run skips a document only when its TEI output already exists, so documents that failed are sent to GROBID again. Since a failed document generally fails again unless something changed, --skip_errors also skips the documents for which a previous run left an error file (<name>_<status>.txt, e.g. paper_500.txt) next to the expected TEI output. Drop the flag (or use --force) to retry them. Error files are kept in sync automatically: the marker is deleted once the document is processed successfully, and replaced when the same document fails again with a different status code.

Python Library

Basic Usage

from grobid_client.grobid_client import GrobidClient

# Initialize with default localhost server
client = GrobidClient()

# Initialize with custom server
client = GrobidClient(grobid_server="https://your-server.com")

# Initialize with config file
client = GrobidClient(config_path="./config.json")

# Process documents
client.process(
    service="processFulltextDocument",
    input_path="/path/to/pdfs",
    output_path="/path/to/output",
    n=20
)

Advanced Usage

# Process with specific options
client.process(
    service="processFulltextDocument",
    input_path="/path/to/pdfs",
    output_path="/path/to/output",
    n=10,
    generate_ids=True,
    consolidate_header=True,
    tei_coordinates=True,
    segment_sentences=True
)

# Process with JSON output
client.process(
    service="processFulltextDocument",
    input_path="/path/to/pdfs",
    output_path="/path/to/output",
    json_output=True
)

# Process with Markdown output
client.process(
    service="processFulltextDocument",
    input_path="/path/to/pdfs",
    output_path="/path/to/output",
    markdown_output=True
)

# Re-run without retrying the documents that failed before
client.process(
    service="processFulltextDocument",
    input_path="/path/to/pdfs",
    output_path="/path/to/output",
    force=False,
    skip_errors=True
)

```python
# Process citation lists
client.process(
    service="processCitationList",
    input_path="/path/to/citations.txt",
    output_path="/path/to/output"
)

Standalone Conversion Tools

The library includes standalone scripts to convert TEI XML files to other formats without using the main client or server.

TEI to JSON Converter

Converts TEI XML files to the structured JSON format (similar to --json option).

# Convert a single file
python -m grobid_client.format.TEI2LossyJSON_cli --input path/to/file.tei.xml --output path/to/output.json

# Convert with verbose logging
python -m grobid_client.format.TEI2LossyJSON_cli --input path/to/file.tei.xml --verbose

TEI to Markdown Converter

Converts TEI XML files to Markdown format (similar to --markdown option).

# Convert a single file
python -m grobid_client.format.TEI2Markdown_cli --input path/to/file.tei.xml --output path/to/output.md

⚙️ Configuration

Configuration can be provided via a JSON file. When using the CLI, the --server argument overrides the config file settings.

Default Configuration

{
  "grobid_server": "http://localhost:8070",
  "batch_size": 1000,
  "sleep_time": 5,
  "timeout": 60,
  "coordinates": [
    "persName",
    "figure",
    "ref",
    "biblStruct",
    "formula",
    "s"
  ]
}

Configuration Parameters

Parameter Description Default
grobid_server GROBID server URL http://localhost:8070
batch_size Thread pool size. Tune carefully: a large batch size will result in the data being written less frequently 1000
sleep_time Wait time when server is busy (seconds) 5
timeout Client-side timeout (seconds) 180
coordinates XML elements for coordinate extraction See above
logging Logging configuration (level, format, file output) See Logging section

Tip

Since version 0.0.12, the config file is optional. The client will use default localhost settings if no configuration is provided.

Warning

Citation consolidation and the timeout setting. When --consolidate_citations (or consolidate_citations=True) is enabled, GROBID queries external services (e.g. CrossRef) to enrich the extracted references. This is considerably slower than a plain extraction, and a low timeout frequently causes HTTP 408 (Request Timeout) errors. Set the timeout to at least 120 seconds (2-3 minutes recommended) when consolidating citations. The client emits a warning when consolidation is requested with a timeout below 120 seconds. See issue #54.

Logging Configuration

The client provides configurable logging with different verbosity levels. By default, only essential statistics and warnings are shown.

Logging Behavior

  • Without --verbose: Shows only essential information and warnings/errors
  • With --verbose: Shows detailed processing information at INFO level

Always Visible Output

The following information is always displayed regardless of the --verbose flag:

Found 1000 file(s) to process
Processing completed: 950 out of 1000 files processed
Errors: 50 out of 1000 files processed
Processing completed in 120.5 seconds

Verbose Output (--verbose)

When the --verbose flag is used, additional detailed information is displayed:

  • Server connection status
  • Individual file processing details
  • JSON conversion messages
  • Detailed error messages
  • Processing progress information

Examples

# Clean output - only essential statistics
grobid_client --input pdfs/ processFulltextDocument
# Output:
# Found 1000 file(s) to process
# Processing completed: 950 out of 1000 files processed
# Errors: 50 out of 1000 files processed
# Processing completed in 120.5 seconds

# Verbose output - detailed processing information
grobid_client --input pdfs/ --verbose processFulltextDocument
# Output includes all essential stats PLUS:
# GROBID server http://localhost:8070 is up and running
# JSON file example.json does not exist, generating JSON from existing TEI...
# Successfully created JSON file: example.json
# ... and other detailed processing information

Configuration File Logging

The config file can include logging settings:

{
    "grobid_server": "http://localhost:8070",
    "logging": {
        "level": "WARNING",
        "format": "%(asctime)s - %(levelname)s - %(message)s",
        "console": true,
        "file": null
    }
}

Note: The --verbose command line flag always takes precedence over configuration file logging settings.

🔬 Services

Fulltext Document Processing

Extracts complete document structure including headers, body text, figures, tables, and references.

grobid_client --input pdfs/ --output results/ processFulltextDocument

JSON Output Format

When using the --json flag, the client converts TEI XML output to a structured JSON format similar to CORD-19. This provides:

  • Structured Bibliography: Title, authors, DOI, publication date, journal information
  • Body Text: Paragraphs and sentences with metadata and reference annotations
  • Figures and Tables: Structured JSON format for tables with headers, rows, and metadata
  • Reference Information: In-text citations with offsets and targets

JSON Structure