Repository navigation
[DISCUSSION]: move sqlparser to Apache (DataFusion) governance #1294
Description
(disclaimer: I am biased being the one who merges sqlparser prs and also am the Apache DataFusion PMC chair)
Problem Statement
sqlparser seems to have become the defacto sql parsing library in Rust (5.5M downloads at the time of this writing) 🎉
However the sqlparser-rs project doesn't have sufficient maintainer capacity. I (@alamb) do enough to keep it from going entirely dormant, but that is really not sufficient for a healthy project.
Here are the specific problems:
- Having contributors wait weeks for review feedback is a bad experience for everyone involved and for that I apologize.
- There is not enough capacity to drive large projects (e.g. token locations) forward
Challenges with current governance structure (or lack thereof)
- There is no clear way to add additional maintainers
- Some employers (for example Apple) only permit contributions to explicitly vetted projects with clear governance (e.g. ASF)
Past discussions:
- @andygrove has brought this up on the arrow mailing list: https://lists.apache.org/thread/q80j49poyg99x2c01900312qz7ps9wgp
- Discussed adding @lovasoa @jmhain and @iffyio as maintainers here Discussion: adding iffyio, jmhain, and/or lovasoa as maintainer of sqlparser-rs #1243 (reactions but no real discussions)
- Discussion adding @AugustoFKL as maintainer: Propose adding AugustoFKL as maintainer of sqlparser-rs #808
When DataFusion was part of the Apache Arrow project, we didn't have the correct space to bring SQL parser at that time
Now that DataFusion is its own top level project (with @andygrove and myself on the PMC) there is a natural space to do thos
Specific Proposal:
- Move the sqlparser-rs code (and commit history) into the Apache DataFusion project and under its Governance. This would require an IP clearance process to run and would take time.
- Move sqlparser-rs repository to
apache/datafusion-sqlparser - Archive this repository, and leave links to
apache/datafuson-sqlparser - Continue to release sqlparser versions approximately monthly.
Benefits of ASF governance;
- More people can approve/merge PRs (committers to DataFusion)
- Clear governance structure (rather than sqlparser-rs today which seems to be mostly me)
- Clear path to add additional maintenaners (e.g. committers)
Drawbacks
- There is a danger that sqlparser becomes "captured" by DataFusion and only accepts features needed for DataFusion
- There is additional overhead to the ASF process (releases, in particular, take additional non trivial overhead)
There is plenty of experience with the ASF release process in DataFusion so I don't think that is a major hurdle. I also think DataFusion in general and sqlparser in particular has a long history of accepting features that benefit all users not just maintainers, so I am not worried about this either (but I am of course biased)
Activity
andygrove commentedon May 30, 2024
Thank you for restarting this conversation @alamb. I am also biased as the original author of sqlparser-rs and DataFusion, but I do think the project will have more success under ASF governance, so I am in favor of this proposal.
lovasoa commentedon May 30, 2024
Hello! I'm not against it, and would love to be added as a maintainer under the asf governance, if we need some balance to avoid too much bias towards datafusion.
sqlparser-rs is a crucial component of SQLPage, and I'll keep making pull request either way.
vaibhawvipul commentedon May 30, 2024
I am in support of this.
alamb commentedon May 30, 2024
Yes, thank you @lovasoa -- I would expect to discuss adding committers to DataFusion who focused on sqlparser-rs (as we have committers focused on another subproject, comet, and we had committers focused on datafusion when it was part of arrow)
jmhain commentedon May 30, 2024
I just realized I never actually responded to #1243. So: I'm happy to participate as a maintainer of
sqlparser-rsregardless of where it ends up. I have a slight preference for keeping it independent but the arguments in favor of moving it under DataFusion seem reasonable.maxcountryman commentedon May 30, 2024
My preference would be to keep the project independent but I'm not able to commit much time to it and it's admittedly a preference mostly based on the idea of how things might appear were it to be moved under the DataFusion project.
nickolay commentedon May 30, 2024
I like how @maxcountryman put it, but I believe the decision must be done by those maintaining the project.
I will take the opportunity to express my gratitude to @alamb for keeping the project going for as long as he has. Thanks for doing this, Andrew!
tobyhede commentedon May 30, 2024
You have my full support.
I am personally so thankful for the endless hours you have put into sqlparser @alamb.
I would love to be involved in any way I can.
cisaacson commentedon Jun 3, 2024
@alamb I think this is a very good idea. One question though: If a fork is created now, it is just of the
sqlparser-rsproject. If you go this way would it mean creating a fork of the entire DataFusion project? That could be OK but I would like to know what would be involved to make custom changes tosqlparser(or other components for that matter). Since the entire DataFusion project changes fairly frequently it may be some work to keep up with those changes.In looking into this further I see the workspace is defined in the datafusion/Cargo.toml workspace. So it looks like someone will need a fork of the whole DataFusion project to make something work in any case. This should be manageable. Unless advised otherwise that's what I will do.
Regardless I am fully supportive of the idea of moving
sqlparser-rsunder the DataFusion umbrella so it is a full-fledged Apache project.alamb commentedon Jun 4, 2024
No I would expect that sqlparser-rs remains in its own repository (just the github organization would be different)
alamb commentedon Jul 9, 2024
@andygrove has started a vote thread here: https://lists.apache.org/thread/7tnq1v0dg2f2xw8c7611bk8mw8lz5dff
dharanad commentedon Jul 13, 2024
I am in favor of this proposal
alamb commentedon Jul 29, 2024
We have started the ip clearance process: https://docs.google.com/document/d/1XPp1zG8yfkCfjusQH7_qr8FMBDDEQ5psR7HhZ1LuVC8/edit#heading=h.nlygv2z3fdg2
Mailing list thread: https://lists.apache.org/thread/1s1qv6gxy0ltzrqq9skxt7lomlq9w4gj
lewiszlw commentedon Aug 20, 2024
I think moving sqlparser-rs into datafusion repo will get more visibility and seems reasonable. Not very sure separate repo can substantially improve the current situation. But we can do this in followup action.
alamb commentedon Sep 4, 2024
@iffyio @lovasoa and @jmhain -- we are trying to move this process along and part of doing so is to get documentation in place for the Apache Software Foundation that all the code in the repo is ok from an ip clearance perspective.
As the most active contributors recently, would you be willing to submit an "Individual Contributor Licence Agreement" (ICLA )to the ASF? The document can be found here: https://www.apache.org/licenses/contributor-agreements.html
This document re-affirms that you have donated all code under the Apache Software License, and is the first step in becoming a committer on the project.
There is more details in @andygrove 's doc here:
https://docs.google.com/document/d/1XPp1zG8yfkCfjusQH7_qr8FMBDDEQ5psR7HhZ1LuVC8/edit
12 remaining items
git-hulk commentedon Sep 19, 2024
Yes, it can. I did migrate another project into ASF: https://issues.apache.org/jira/browse/INFRA-25024, hope this would help.
alamb commentedon Sep 19, 2024
Thanks @git-hulk and @lovasoa -- I have filed https://issues.apache.org/jira/browse/INFRA-26136 to request the transfer.
andygrove commentedon Sep 19, 2024
Here is the XML form for the IP clearance process:
https://svn.apache.org/repos/asf/incubator/public/trunk/content/ip-clearance/datafusion-sqlparser.xml
As part of this process I would like to remind all contributors that they are responsible for ensuring that a Corporate CLA is recorded if such is required to authorize their contributions under their individual CLA
andygrove commentedon Sep 19, 2024
HTML form: https://incubator.apache.org/ip-clearance/datafusion-sqlparser.html
We need to do a couple more things before we can start the IP clearance vote:
andygrove commentedon Sep 19, 2024
Don't we need to complete the IP clearance process before moving the code into the new repo?
alamb commentedon Sep 19, 2024
We certainly need to complete the IP clearance process before making a new release. I am not sure about moving into the apache github organization
I will create a PR later today to add Apache License headers to the files in sqlparser-rs
alamb commentedon Sep 19, 2024
Filed #1436 to track migration issues
alamb commentedon Sep 20, 2024
The PR is here: #1437
andygrove commentedon Sep 26, 2024
The IP clearance vote has passed. Thanks to everyone who helped with this process!
https://lists.apache.org/thread/4qhyk23d0g01g3ccsffsxygwd6vj9b8k
andygrove commentedon Sep 30, 2024
I see that the repo has now moved 🎉
alamb commentedon Sep 30, 2024
I plan to review / update #1436 tomorrow with next steps
alamb commentedon Oct 2, 2024
Now that the repo has moved and the project has been accepted, let's track the transition work in #1436. Gere is the proposed plan going forward:
#1436 (comment)
Comments more than welcome
mobuchowski commentedon Nov 6, 2024
@alamb as maintainer of other OSS project that uses sqlparser-rs (OpenLineage) just wanted to thank you for the continued maintanence, support and merging a few of our PRs :)
alamb commentedon Nov 6, 2024
@mobuchowski -- thank you for the kind words
The last few months @iffyio has really been the key maintainer. 🙏 Thank you!