smoke test in standup.sh step 10 is issued too soon

[effi-ofer]

Component

I don't know

Describe the bug

The smoke test can fail because the prefill/decode pods are not ready yet. The pods are Running but not in the Ready state.

Steps to reproduce

Issue standup.sh to create llm-d.

Additional context or screenshots

No response

[effi-ofer]

If after the smoke test fails, I wait until the pods are truly ready and then issue the smoke test again via "standup.sh ... -s 10" then the smoke test succeeds.

[effi-ofer]

This is what the decode pod looks like when the smoke test already runs:

meta-lla-01e4ca79-instruct-decode-65fcb9b75-6lrpq 1/2 Running 0 17s 10.130.2.95 fmaas-vllm-d-wv25b-worker-h100-3-4l5t5

[maugustosilva]

@effi-ofer We have code (function wait_for_pods_created_running_ready) that deliberately waits until all pods are in Ready state in step 9 before proceeding. If you see a scenario where this is failing, then we have a bug. Can you provide more data? I will try to reproduce.

[effi-ofer]

Hey @maugustosilva I do see it occasionally in my runs when I run the following on the fmaas-vllm-d cluster:

./setup/standup.sh -c $PWD/myenv-llmd-base-2d0p.sh --non-admin

Where myenv-llmd-base-2d0p.sh is:

export LLMDBENCH_HARNESS_NAME="inference-perf"
export LLMDBENCH_HF_TOKEN="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"

export LLMDBENCH_VLLM_COMMON_NAMESPACE="effi2"
export LLMDBENCH_HARNESS_NAMESPACE="effi2"

export LLMDBENCH_DEPLOY_METHODS="modelservice"
export LLMDBENCH_VLLM_MODELSERVICE_DECODE_EXTRA_ARGS="[--disable-hybrid-kv-cache-manager --gpu-memory-utilization 0.80 --enable-chunked-prefill --max_num_batched_tokens 512 --max_model_len 4096]"
export LLMDBENCH_VLLM_MODELSERVICE_PREFILL_EXTRA_ARGS="[--disable-hybrid-kv-cache-manager --gpu-memory-utilization 0.80 --enable-chunked-prefill --max_num_batched_tokens 512 --max_model_len 4096]"

export LLMDBENCH_VLLM_MODELSERVICE_DECODE_CPU_MEM=350G
export LLMDBENCH_VLLM_MODELSERVICE_PREFILL_CPU_MEM=350G
export LLMDBENCH_VLLM_MODELSERVICE_DECODE_REPLICAS=2
export LLMDBENCH_VLLM_MODELSERVICE_PREFILL_REPLICAS=0
export LLMDBENCH_VLLM_COMMON_AFFINITY="nvidia.com/gpu.product:NVIDIA-H100-80GB-HBM3"

export LLMDBENCH_LLMD_IMAGE_REGISTRY=quay.io
export LLMDBENCH_LLMD_IMAGE_REPO=oroz
export LLMDBENCH_LLMD_IMAGE_NAME=llm-d
export LLMDBENCH_LLMD_IMAGE_TAG=cpu-offloading-11

export LLMDBENCH_VLLM_COMMON_MAX_MODEL_LEN=4096
export LLMDBENCH_DEPLOY_CURRENT_MODEL="meta-llama/Llama-3.1-8B-Instruct"
export LLMDBENCH_DEPLOY_MODEL_LIST="meta-llama/Llama-3.1-8B-Instruct"

export LLMDBENCH_CONTROL_WORK_DIR="/home/effi/llm-d-benchmark/runs"

export LLMDBENCH_HARNESS_WAIT_TIMEOUT=3600

export LLMDBENCH_VLLM_COMMON_PVC_STORAGE_CLASS="ocs-storagecluster-cephfs"
export LLMDBENCH_VLLM_GAIE_CHART_NAME="oci://us-central1-docker.pkg.dev/k8s-staging-images/gateway-api-inference-extension/charts/inferencepool"

export LLMDBENCH_VLLM_MODELSERVICE_GATEWAY_CLASS_NAME="istio"
export LLMDBENCH_VLLM_GAIE_CHART_VERSION="v0"
export LLMDBENCH_GATEWAY_API_INFERENCE_EXTENSION_CRD_REVISION="v1.2.0"

export LLMDBENCH_VLLM_MODELSERVICE_INFERENCE_MODEL=true # (default is "false")