Current State:
Polynote persists notebooks locally on the server’s file system. The lifecycle of the saved notebooks is thus tightly coupled with that of the polynote server itself and does not lend itself well to sharing notebooks, operating them in a distributed environment, backup and recovery as well as versioning.
This raises below issues:
Unavailability of notebooks if the polynote server is down
Having to re-upload notebooks if a fresh polynote server is stood up
Loss of data if the operational model is to deploy polynote servers as ephemeral containers.
Having an external store shall help remediate these issues. A related issue is open to address similar concerns by virtue of persisting to external S3 Storage.
Description:
Here we propose adding support for an open-source SQL database, such as Postgres as an additional storage backend.
In addition to moving the persistence model outside of local file-system, this delivers on additional benefits:
Ability to add additional metadata outside of the notebook itself.
Ability to search on additional metadata.
Given Postgres supports JSON field types, one could search on notebook content as well.
Proposed Path:
One may consider making the storage backend design more modular to allow for additional backends. Recommend we add support for one database and solicit community feedback before we consider a more modular design.
Proposed implementation would be to use the same file based data model for notebooks, but persist in a Postgres DB, wherein the choice b/w filesystem and postgres would be a configuration in the config-template.yml.
Current State:
Polynote persists notebooks locally on the server’s file system. The lifecycle of the saved notebooks is thus tightly coupled with that of the polynote server itself and does not lend itself well to sharing notebooks, operating them in a distributed environment, backup and recovery as well as versioning.
This raises below issues:
Unavailability of notebooks if the polynote server is down
Having to re-upload notebooks if a fresh polynote server is stood up
Loss of data if the operational model is to deploy polynote servers as ephemeral containers.
Having an external store shall help remediate these issues. A related issue is open to address similar concerns by virtue of persisting to external S3 Storage.
Description:
Here we propose adding support for an open-source SQL database, such as Postgres as an additional storage backend.
In addition to moving the persistence model outside of local file-system, this delivers on additional benefits:
Ability to add additional metadata outside of the notebook itself.
Ability to search on additional metadata.
Given Postgres supports JSON field types, one could search on notebook content as well.
Proposed Path:
One may consider making the storage backend design more modular to allow for additional backends. Recommend we add support for one database and solicit community feedback before we consider a more modular design.
Proposed implementation would be to use the same file based data model for notebooks, but persist in a Postgres DB, wherein the choice b/w filesystem and postgres would be a configuration in the config-template.yml.