Hermes: Efficient Serving of LLM Applications with Probabilistic Demand Modeling

This repository contains the artifact of our paper "Hermes: Efficient Serving of LLM Applications with Probabilistic Demand Modeling".

Get started

$ conda create --name Hermes python=3.10 && conda activate Hermes
$ pip install -e ./Hermes
$ pip install -e ./vllm
$ python3 -m Hermes.run --policy Hermes