Conversation
|
Thanks for your contribution! |
|
/re-run all-failed |
Codecov Report❌ Patch coverage is ❌ Your patch status has failed because the patch coverage (59.30%) is below the target coverage (75.00%). You can increase the patch coverage or adjust the target coverage. Additional details and impacted files@@ Coverage Diff @@
## develop #3801 +/- ##
==========================================
Coverage ? 32.22%
==========================================
Files ? 433
Lines ? 82298
Branches ? 0
==========================================
Hits ? 26524
Misses ? 55774
Partials ? 0 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
| lora_module = RowParallelQuantizationLoRALinear(module, lora_config) | ||
| # Lora row parallel will spilt lora A matrix | ||
| self.add_lora_split_mapping(module_name + ".lora_A", is_column=False) | ||
| elif attribute_chain[-1] == "experts": |
There was a problem hiding this comment.
1.这个匹配规则是否具有通用性，会不会替换其他存量模型导致问题？
2. 需要考虑如果模型的expert写法比较特殊能够流一个接口适配自定义的loraexpert
3.是否能够匹配paddlefleet的expert?
There was a problem hiding this comment.
已修改匹配规则，并且保留接口用于适配自定义的lora expert
| @@ -1055,8 +1017,7 @@ def get_lora_model(self, model: Union[PretrainedModel, nn.Layer], lora_config: L | |||
| return model | |||
| if isinstance(lora_config.target_modules, str): | |||
| lora_config.target_modules = [lora_config.target_modules] | |||
| lora_config.target_modules = [lora_config.target_modules] | ||
| for i in model.named_sublayers(): | ||
| module_name = i[0] | ||
| for module_name, module in model.named_sublayers(): |
| @@ -1055,8 +1017,7 @@ def get_lora_model(self, model: Union[PretrainedModel, nn.Layer], lora_config: L | |||
| return model | |||
| if isinstance(lora_config.target_modules, str): | |||
| lora_config.target_modules = [lora_config.target_modules] | |||
There was a problem hiding this comment.
需要适配get_merge_state_dict函数
|
/re-run all-failed |
| import paddle | ||
| import paddle.nn as nn | ||
|
|
||
| from .activation import ACT2FN |
| from ...nn.attention.interface import ALL_ATTENTION_FUNCTIONS | ||
| from ...nn.criterion.interface import CriterionLayer | ||
| from ...nn.embedding import Embedding as GeneralEmbedding | ||
| from ...nn.experts import MoeExperts as Qwen3VLMoeTextExperts |
|
/re-run all-failed |
3 participants
PR types
Others
PR changes
Others
Description
fuse moe lora