fix: example dtype hardcode - #287
Conversation
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Summary of Changes
Hello @drunkcoding, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!
This pull request addresses an issue where data types (dtype or torch_dtype) were hardcoded in several example scripts for loading and saving models. The changes remove these explicit hardcoded values, enabling automatic data type conversion and improving compatibility with Hugging Face configurations, thereby resolving issue #220.
Highlights
- Remove Hardcoded Dtypes in Examples: Eliminated explicit
torch_dtype=torch.float16ordtype="float16"parameters from various model loading and saving example scripts, includingload_lora_adapter.py,load_quantized_transformers_model.py,load_transformers_model.py,load_vllm_model.py,save_lora_adapter.py, andsave_vllm_model.py. This allows the underlying libraries to automatically determine the appropriate data type. - Enable Automatic Dtype Conversion for Saving Transformers Models: Modified
save_transformers_model.pyto dynamically retrieve thetorch_dtypefrom the model'sAutoConfig(e.g.,config.torch_dtype) instead of hardcodingtorch.float16. This change ensures that models are saved with their native or configured data type, facilitating better compatibility with Hugging Face configurations.
Using Gemini Code Assist
The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.
Invoking Gemini
You can request assistance from Gemini at any point in your pull request via creating an issue comment (i.e. comment on the pull request page) using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands.
| Feature | Command | Description |
|---|---|---|
| Code Review | /gemini review |
Performs a code review for the current pull request in its current state. |
| Pull Request Summary | /gemini summary |
Provides a summary of the current pull request in its current state. |
| Comment | @gemini-code-assist | Responds in comments when explicitly tagged, both in issue comments and review comments. |
| Help | /gemini help |
Displays a list of available commands. |
Customization
To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.
Limitations & Feedback
Gemini Code Assist is currently in preview and may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments to provide feedback.
You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.
Footnotes
-
Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩
future-xy
added
the
ready-for-gpu-testing
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
Bugbot free trial expires on August 7, 2025
Learn more in the Cursor dashboard.
| model_name, | ||
| device_map="auto", | ||
| torch_dtype=torch.float16, | ||
| torch_dtype=config.torch_dtype, |
There was a problem hiding this comment.
Bug: Configuration Error Causes Runtime Failures
The explicit setting of config.torch_dtype = torch.float16 was removed. If AutoConfig.from_pretrained() returns a configuration where torch_dtype is None or an invalid value, subsequent operations that use config.torch_dtype (e.g., .to(config.torch_dtype) or passing it as an argument to load_model/load_lora) will fail at runtime.
Locations (3)
2 participants
Description
Auto dtype convertion on save with huggingface configurations
Motivation
Explain why this change is needed and what problem it solves.
If it fixes an issue, link it (e.g.,
close #123).Type of Change
Checklist