Add parallelized restore capability to ghe-restore-storage - #635
Merged
Conversation
maclarel
marked this pull request as ready for review
September 14, 2020 13:57
omgitsads
approved these changes
Sep 17, 2020
omgitsads
left a comment
omgitsads
left a comment
Member
There was a problem hiding this comment.
Sorry for the delay on this. I think it looks good to me 👍 .
Have you tried this out with a large amount of data, to get an idea of the improvement? Given this is just rsync'ing data from /data/user/storage, you could generate a decent chunk of random data with dd and confirm it's doing what you expect.
Contributor
Author
|
Was originally bottlenecked by my home connection speed for effectively testing this, but wound up spinning up an EC2 instance with a 5Gbit NIC to remove that problem :) With that said, disk performance on my backup host will still likely be a bottleneck as I'm seeing transfer and disk speeds plummet the more I use my test instance. Added 6GB worth of 1GB files to random locations for the storage restore into the Time without parallelism (~9 min): Time with parallelism (3 min 40 sec): I'll note that I burned a fair bit of time on this getting I've also verified that restores to an HA environment still function (with parallelism enabled or disabled) and are otherwise unaffected by these updates 🎉 |
Member
|
@maclarel I think you can just install |
jianghao0718
changed the title
jianghao0718
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This introduces a parallelized restore of
storagedata, with a number of rsync threads equal to the number of storage nodes. This is the same logic used forghe-restore-repositories, simply ported over toghe-restore-storage.For customers that are heavy users of LFS in a clustered environment, this can have significant performance improvements, with a reduction in run time equivalent to the number of nodes. Specifically,
rsynconly utilizes a single thread ofsshd, so when high transfer speeds are possible it is likely thatsshdwill become CPU bound resulting in limited transfer speed.For example, restoring ~5TB of data across 5 storage nodes would complete in approximately 16 hours assuming a transfer speed of 100MB/s (roughly where we see
sshdbecome CPU bound) as the restores would be run sequentially. Assuming sufficient bandwidth for transfers at 500MB/s (achievable on a 10Gbit connection) this could reduce the overall time to approximately 3 hours as all 5rsyncinvocations would be run simultaneously and would utilize 1 thread per server effectively quintupling performance.Verbose log confirms that all 3 are being kicked off at the same time, which aligns what what is seen for
ghe-restore-repositoriesbehaviour: