KeyError: 'base_model.model.model.layers.0.self_attn.q_proj.lora_A.default.weight'

[huiwudiyi]

2. 加载优化器

optimizer_state_path = f"{model_path}/optimizer.pt"
optimizer_state = torch.load(optimizer_state_path, map_location='cpu')

print(optimizer_state['state'])

{0: {'step': tensor(375.),
'exp_avg': tensor([[-3.3680e-07, -2.6961e-07, 3.3326e-08, ..., 6.7300e-08,
2.9751e-07, -1.3901e-07],
...]),
'exp_avg_sq': tensor([[7.5530e-13, 2.1739e-13, 4.8634e-15, ..., 8.3630e-14, 1.0447e-13,
4.4873e-14],
...])}

Traceback (most recent call last):
File "", line 198, in _run_module_as_main
File "", line 88, in _run_code
File "/root/autodl-tmp/LESS/less/data_selection/get_info.py", line 156, in
collect_grads(dataloader,
File "/root/autodl-tmp/LESS/less/data_selection/collect_grad_reps.py", line 195, in collect_grads
m, v = prepare_optimizer_state(model, adam_optimizer_state, device)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/autodl-tmp/LESS/less/data_selection/collect_grad_reps.py", line 132, in prepare_optimizer_state
avg = torch.cat([optimizer_state[n]["exp_avg"].view(-1) for n in names])
~~~~~~~~~~~~~~~^^^
KeyError: 'base_model.model.model.layers.0.self_attn.q_proj.lora_A.default.weight'

What is this problem and how can it be solved？

[GenerallyCovetous]

same problem, have you solved it?

[Gumpest]

Me, too... @karthikncode