Skip to content

When the compiler get's killed by OOM/SIGTERM the cache-db.json file is not updated #3995

Description

@gorbak25

Description

When building large purescript codebases such as https://github.com/erlscripten/erlps-stdlib it's very easy to get killed by OOM - The modules are compiled in parallel usually exceeding the amount of RAM available on the system(I will open separate issues with the results of my investigation/profiling data - for instance the lexer right now uses up a lot of memory on large files). What is really troublesome is that if the compilation gets killed by OOM after compiling a lot of heavy modules then on the next run they will get recompiled although they compiled successfully. After investivating the purescript codebase the cache-db.json file is only updated after all of the compiler tasks either succeed or failed - this is not a problem for small projects but when the compilation times become significant(https://github.com/erlscripten/erlps-stdlib) it's really troublesome :(

To Reproduce

  1. Compile a project with modules M_1, M_2, ..., M_N
  2. The compiler compiles M_1, M_2, ..., M_N-1
  3. While compiling M_N the compiler gets killed either by SIGTERM or OOM
  4. After restarting the build modules M_1, M_2, ..., M_N-1 are being recompiled again although the generated code is already present on the drive

Expected behavior

  1. Compile a project with modules M_1, M_2, ..., M_N
  2. The compiler compiles M_1, M_2, ..., M_N-1
  3. While compiling M_N the compiler gets killed either by SIGTERM or OOM
  4. After restarting the build only M_N gets build

PureScript version

Tried both on 0.13.8 and on the current master branch :(

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions