Skip to content

bug(indexer): Non-UTF-8 source files are silently dropped for Python, JS, TS, TSX, Go, CSS, HTML #1390

Closed

Description

@Shashankss1205

tools/languages/python.py:150, javascript.py:187, typescript.py:181, typescriptjsx.py:84, go.py:180, css.py:20, html.py:42

with open(path, "r", encoding="utf-8") as f:      # no errors= parameter
    source_code = f.read()

The other 20 language parsers (c, cpp, java, kotlin, rust, php, ruby, swift, scala, dart, perl, csharp, haskell, elixir, elisp, lua, …) all pass errors="ignore". The seven most-used languages do not.

UnicodeDecodeError is caught by graph_builder.py:1053, which returns {"path": …, "error": …} with no unsupported key. pipeline.py:149-155 then treats it as a generic file and creates a bare File node — indistinguishable from a .md file.

Reproduced. A repo with latin.py (latin-1 ç), ok.py, bom.py:

Total scanned files: 3 | Function nodes: 4
Successfully finished indexing

Graph:  File node for latin.py       → exists
        Function nodes for latin.py  → NONE

No warning; the summary table has no "files failed" row. A codebase with legacy latin-1 files loses them entirely and reports success.

Fix. Use errors="replace" consistently across all 23 parsers, and add a failed_files count to build_index_summary.


Audit environment
Version codegraphcontext 0.5.2 (PyPI)
Commit df3af04
Backend FalkorDB Lite (falkordblite 0.9.0), Python 3.12.3, Linux
Corpus the CodeGraphContext repo itself — 784 files, 23 languages, 5,501 Function nodes

Audit ID I-H02 · severity high · from an end-to-end audit of CGC 0.5.2 (63 findings). Reported with the reproduction as run.

Activity

  1. added
    bugSomething isn't working
    severity: highWrong graph content, data loss, or a broken workflow with no clear signal
    area: indexerParsing, discovery, resolution, persistence
    on Jul 26, 2026
  2. github-actions commented on Jul 26, 2026

    @github-actions
    Contributor

    Hi! 👋 Join our CodeGraphContext Discord channel to collaborate: https://discord.gg/dR4QY32uYQ

  3. moved this from Backlog tasks to In progress in CGC Progress Boardon Jul 26, 2026
  4. moved this from In progress to Done in CGC Progress Boardon Jul 26, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area: indexerParsing, discovery, resolution, persistencebugSomething isn't workingseverity: highWrong graph content, data loss, or a broken workflow with no clear signal

    Type

    No type

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions