This repository was archived by the owner on Apr 23, 2026. It is now read-only.
Make SDG batch size configurable via system profile#3157
Merged
mergify[bot] merged 1 commit intoinstructlab:mainfrom Feb 26, 2025
Merged
Make SDG batch size configurable via system profile#3157mergify[bot] merged 1 commit intoinstructlab:mainfrom
mergify[bot] merged 1 commit intoinstructlab:mainfrom
Conversation
mergify
Bot
added
testing
cdoern
reviewed
Feb 17, 2025
Contributor
cdoern
left a comment
cdoern
left a comment
There was a problem hiding this comment.
looks good! Technically, if you are using the default value you don't need to add this to all of the profiles manually. One comment about the default in the click flag.
| default=None, | ||
| help="Number of elements to process in each batch through the SDG pipeline. Enabled by default for the vLLM serving backend, with a batch size of 8 chosen based on experiments to optimize for throughput. Use 0 to disable.", | ||
| cls=clickext.ConfigOption, | ||
| default=DEFAULTS.BATCH_SIZE, |
Contributor
There was a problem hiding this comment.
the default comes from the config when you set cls=... so no default is necessary here.
Contributor
Author
There was a problem hiding this comment.
Thanks @cdoern, hmm I think I am missing something. If I run ilab data generate --debug-params. I get all the values of the parameter correct except for the batch size for which I get
batch_size: None [type: None, src: default] (if I remove the default)
npalaska
force-pushed
the
configurable_batch_size
branch
from
576aa76 to
f4e8c50
Compare
cdoern
reviewed
Feb 17, 2025
| default=None, | ||
| help="Number of elements to process in each batch through the SDG pipeline. Enabled by default for the vLLM serving backend, with a batch size of 8 chosen based on experiments to optimize for throughput. Use 0 to disable.", | ||
| cls=clickext.ConfigOption, | ||
| default=DEFAULTS.BATCH_SIZE, |
Contributor
There was a problem hiding this comment.
Suggested change
| default=DEFAULTS.BATCH_SIZE, |
removing this line and re-installing I get the following:
ilab data generate --debug-params
Parameters:
model_path: '/Users/charliedoern/.cache/instructlab/models/mistral-7b-instruct-v0.2.Q4_K_M.gguf' [type: str, src: default_map]
num_cpus: 10 [type: int, src: default_map]
chunk_word_count: 1000 [type: int, src: default_map]
num_instructions: -1 [type: int, src: default]
sdg_scale_factor: 30 [type: int, src: default_map]
taxonomy_path: '/Users/charliedoern/.local/share/instructlab/taxonomy' [type: str, src: default_map]
taxonomy_base: 'origin/main' [type: str, src: default_map]
output_dir: '/Users/charliedoern/.local/share/instructlab/datasets' [type: str, src: default_map]
quiet: False [type: bool, src: default]
endpoint_url: None [type: None, src: default]
api_key: 'no_api_key' [type: str, src: default]
yaml_rules: None [type: None, src: default]
server_ctx_size: 4096 [type: int, src: default]
tls_insecure: False [type: bool, src: default]
tls_client_cert: '' [type: str, src: default]
tls_client_key: '' [type: str, src: default]
tls_client_passwd: '' [type: str, src: default]
model_family: None [type: None, src: default]
pipeline: 'full' [type: str, src: default_map]
batch_size: 8 [type: int, src: default_map]
enable_serving_output: False [type: bool, src: default]
gpus: None [type: None, src: default_map]
max_num_tokens: 4096 [type: int, src: default_map]
detached: False [type: bool, src: default]
Contributor
Author
There was a problem hiding this comment.
Thanks it indeed worked after I reinstalled it
Contributor
|
@Mergifyio rebase |
Contributor
✅ Branch has been successfully rebased |
cdoern
force-pushed
the
configurable_batch_size
branch
from
f4e8c50 to
bb2eb90
Compare
cdoern
approved these changes
Feb 19, 2025
mergify
Bot
added
one-approval
Signed-off-by: Nikhil Palaskar <[email protected]>
npalaska
force-pushed
the
configurable_batch_size
branch
from
bb2eb90 to
4a309ce
Compare
mergify
Bot
added
ci-failure
cdoern
approved these changes
Feb 24, 2025
nathan-weinberg
approved these changes
Feb 26, 2025
Contributor
|
@mergify backport release-v0.24 |
Contributor
✅ Backports have been createdDetails
|
Contributor
|
@mergify backport release-v0.23 |
Contributor
✅ Backports have been createdDetails
|
ktdreyer
added a commit
that referenced
this pull request
Mar 3, 2025
…-3157 Make SDG batch size configurable via system profile (backport #3157)
mergify Bot
added a commit
that referenced
this pull request
Mar 3, 2025
…3207) Currently, the batch size for SDG is only configurable via the CLI, but a single batch size across all hardware profiles is not optimal. Different hardware configurations have varying capabilities, and using a fixed batch size can lead to under-utilization or over-utilization of resources during the SDG process. To ensure efficient performance across different hardware, we should set the batch sizes independently in each system profile. <hr>This is an automatic backport of pull request #3157 done by [Mergify](https://mergify.com). Approved-by: courtneypacheco Approved-by: ktdreyer
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Currently, the batch size for SDG is only configurable via the CLI, but a single batch size across all hardware profiles is not optimal. Different hardware configurations have varying capabilities, and using a fixed batch size can lead to under-utilization or over-utilization of resources during the SDG process.
To ensure efficient performance across different hardware, we should set the batch sizes independently in each system profile.