[RFC]: quant llm from alpindale

[flozi00]

Motivation.

Higher throughput und memory savings are always cool 😎

I think that could be integrated very easily, what do you think about it's design ?

Proposed Change.

aphrodite-engine/aphrodite-engine@7317765

Feedback Period.

No response

CC List.

No response

Any Other Things.

No response

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[flozi00]

#8751