transformers怎么使用gguf？

[VacantHusky]

我想使用int4,发现只有gguf的。然后我下载了llama3.1_8b_chinese_chat_q4_k_m-shareAI.gguf。
从这里下载的：https://hf-mirror.com/shareAI/llama3.1-8b-instruct-dpo-zh

然后使用transformers。

from transformers import AutoTokenizer, AutoModelForCausalLM

model_id = ".\\models\\llama3.1-8b-instruct-dpo-zh"
filename = "llama3.1_8b_chinese_chat_q4_k_m-shareAI.gguf"

tokenizer = AutoTokenizer.from_pretrained(model_id, gguf_file=filename)
model = AutoModelForCausalLM.from_pretrained(model_id, gguf_file=filename)

运行后报错：

Traceback (most recent call last):
...
  File "chat.py", line 75, in load_model
    model = AutoModelForCausalLM.from_pretrained(
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "E:\work\work_py\ENVS\Qwen\Lib\site-packages\transformers\models\auto\auto_factory.py", line 564, in from_pretrained
    return model_class.from_pretrained(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "E:\work\work_py\ENVS\Qwen\Lib\site-packages\transformers\modeling_utils.py", line 3963, in from_pretrained
    ) = cls._load_pretrained_model(
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "E:\work\work_py\ENVS\Qwen\Lib\site-packages\transformers\modeling_utils.py", line 4092, in _load_pretrained_model
    is_safetensors = archive_file.endswith(".safetensors")
                     ^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'endswith'

[VacantHusky]

我知道原因了，只能使用cpu而不能使用gpu。

# 可以
model = AutoModelForCausalLM.from_pretrained(model_id, gguf_file=filename, device_map="cpu")

# 报错
model = AutoModelForCausalLM.from_pretrained(model_id, gguf_file=filename, device_map="auto")

能否提供gptq格式的？