Move ztwim-test-generator inside oape plugin #12
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,112 @@ | ||
| --- | ||
| description: Generate all ZTWIM PR test artifacts in one run (test scenarios, execution steps, e2e Go code) | ||
| argument-hint: "<pr-url> [--output <path>] [--env <cluster-type>]" | ||
| --- | ||
|
|
||
| ## Name | ||
| oape:ztwim-generate-all | ||
|
|
||
| ## Synopsis | ||
| ``` | ||
| /oape:ztwim-generate-all <pr-url> [--output <path>] [--env <cluster-type>] | ||
| ``` | ||
|
|
||
| ## Description | ||
|
|
||
| **Single command** that analyzes a ZTWIM operator Pull Request and generates **all three outputs** in one run: | ||
|
|
||
| 1. **test-cases.md** — Test scenarios (what to test, PR-specific focus, verification, cleanup). | ||
| 2. **execution-steps.md** — Step-by-step procedure with executable `oc` commands (prerequisites, install, stack, verify, PR-specific steps, cleanup). | ||
| 3. **\<prno\>_test_e2e.go** — Go e2e test code (Ginkgo Describe/Context/It) for the ZTWIM repo `test/e2e/`, with operator/operand scenarios and PR-specific tests. | ||
|
|
||
| All files are written into **one output directory**: `<output-dir>/ztwim_pr_<number>/`. Default `<output-dir>` is `output` (create if missing). Use `--output <path>` to set a different base. | ||
|
|
||
| - **Repository**: openshift/zero-trust-workload-identity-manager only. | ||
| - **Install/Stack**: Use plugin fixtures (`plugins/oape/ztwim-test-generator/fixtures/operator-install.yaml`, `plugins/oape/ztwim-test-generator/fixtures/ztwim-stack.yaml`). Do not discover from repo. | ||
| - **E2E**: Follow upstream test/e2e structure; see plugin [docs/e2e-structure.md](../ztwim-test-generator/docs/e2e-structure.md) and [fixtures/e2e-important-scenarios.md](../ztwim-test-generator/fixtures/e2e-important-scenarios.md). | ||
|
|
||
| ## Implementation | ||
|
|
||
| ### Step 1: Validate PR and Analyze Changes (Once) | ||
|
|
||
| 1. **Validate PR URL** is for ZTWIM: `https://github.com/openshift/zero-trust-workload-identity-manager/pull/<number>`. If not, inform the user this command is for ZTWIM PRs only. | ||
| 2. **Use browser tools**: Navigate to the PR URL, then to "Files changed" (append `/files`). Use **browser_snapshot** to read PR description and changed files. Do **not** use `gh` CLI. | ||
| 3. Extract PR number and (optionally) a short description from the title. Map changed files to: | ||
| - Test focus (API types, CRD, controller, RBAC, samples, e2e). | ||
| - E2E focus (operator/operand scenarios, PR-specific It blocks). | ||
|
|
||
| ### Step 2: Output Directory | ||
|
|
||
| - **Path**: `<output-dir>/ztwim_pr_<number>/` (e.g. `output/ztwim_pr_72/`). | ||
| - **Default `<output-dir>`**: `output` (relative to workspace root). Create the directory if it does not exist. | ||
| - **With `--output <path>`**: Use `<path>` as the base; write into `<path>/ztwim_pr_<number>/`. | ||
| - **With `--env`**: Use only when generating execution-steps content (e.g. env-specific notes); optional. | ||
|
|
||
| ### Step 3: Generate test-cases.md | ||
|
|
||
| Write **test-cases.md** into the output directory. Content must include: | ||
|
|
||
| - Operator info (ZTWIM), repository. | ||
| - Prerequisites (cluster, env vars: APP_DOMAIN, JWT_ISSUER_ENDPOINT, CLUSTER_NAME). | ||
| - Install: fixture path, `oc apply -f .../operator-install.yaml`, wait for CSV and deployment. | ||
| - Stack: fixture path, envsubst, `envsubst < .../ztwim-stack.yaml | oc apply -f -`. | ||
| - PR-specific test cases derived from Files changed (field tests, controller tests, validation). | ||
| - Verification (oc get CRs, oc wait, oc logs in zero-trust-workload-identity-manager). | ||
| - Cleanup order: SpireOIDCDiscoveryProvider → SpiffeCSIDriver → SpireAgent → SpireServer → ZeroTrustWorkloadIdentityManager → subscription → CSV → OperatorGroup → namespace. | ||
|
|
||
| ### Step 4: Generate execution-steps.md | ||
|
|
||
| Write **execution-steps.md** into the output directory. Content must include: | ||
|
|
||
| - Prerequisites: `which oc`, `oc version`, `oc whoami`, `oc get nodes`, `oc get clusterversion`, packagemanifests check, then APP_DOMAIN/JWT_ISSUER_ENDPOINT/CLUSTER_NAME and echo. | ||
| - Install: `oc apply -f <path-to>/operator-install.yaml`, then oc wait for CSV and deployment, oc get pods. | ||
| - Stack: envsubst and oc apply for ztwim-stack.yaml. | ||
| - CR verification: oc get all ZTWIM CRs, oc wait for Ready. | ||
| - PR-specific execution steps (from Files changed). | ||
| - Cleanup: full oc delete sequence in fixed order. | ||
|
|
||
| Use plugin fixture paths; document `<path-to>` as `plugins/oape/ztwim-test-generator/fixtures`. | ||
|
|
||
| ### Step 5: Generate \<prno\>_test_e2e.go and e2e-suggestions.md | ||
|
|
||
| **e2e-suggestions.md** (in the same output directory): | ||
|
|
||
| - Short list of which operator/operand e2e scenarios apply. | ||
| - Which are highly recommended for this PR. | ||
| - PR-specific It block suggestions. | ||
|
|
||
| **\<prno\>_test_e2e.go** (in the same output directory): | ||
|
|
||
| - Package `e2e`; same imports and style as upstream (see [docs/e2e-structure.md](../ztwim-test-generator/docs/e2e-structure.md), [fixtures/e2e-sample_test.go.example](../ztwim-test-generator/fixtures/e2e-sample_test.go.example)). | ||
| - Use `k8sClient`, `clientset`, `testCtx`, `utils.*`; no BeforeSuite/TestE2E. | ||
| - Describe/Context/It blocks with `By("…")`; comment each It (e.g. `// PR-suggested: ...`) for pick-and-choose. | ||
| - Include important scenarios (install, recovery, operand conditions, ZTWIM aggregation, OperatorCondition Upgradeable, CR-driven config) and PR-specific tests from Files changed. | ||
|
|
||
| ### Step 6: Confirm Output | ||
|
|
||
| Tell the user the output directory path and list the three (or four) generated files: test-cases.md, execution-steps.md, \<prno\>_test_e2e.go, and optionally e2e-suggestions.md. | ||
|
|
||
| ## Arguments | ||
|
|
||
| - **$1 (pr-url)**: ZTWIM operator GitHub PR URL — `https://github.com/openshift/zero-trust-workload-identity-manager/pull/<number>`. | ||
| - **--output**: Output base directory (optional). Default: `output`. All files go in `<output>/ztwim_pr_<number>/`. | ||
| - **--env**: Target environment for execution-steps (optional): `aws`, `gcp`, `azure`, `vsphere`, `baremetal`. | ||
|
|
||
| ## Examples | ||
|
|
||
| ``` | ||
| /oape:ztwim-generate-all https://github.com/openshift/zero-trust-workload-identity-manager/pull/72 | ||
| # Writes: output/ztwim_pr_72/test-cases.md | ||
| # output/ztwim_pr_72/execution-steps.md | ||
| # output/ztwim_pr_72/72_test_e2e.go | ||
| # output/ztwim_pr_72/e2e-suggestions.md | ||
|
|
||
| /oape:ztwim-generate-all https://github.com/openshift/zero-trust-workload-identity-manager/pull/72 --output .work | ||
| # Writes: .work/ztwim_pr_72/test-cases.md, .work/ztwim_pr_72/execution-steps.md, .work/ztwim_pr_72/72_test_e2e.go, .work/ztwim_pr_72/e2e-suggestions.md | ||
| ``` | ||
|
|
||
| ## Notes | ||
|
|
||
| - **ZTWIM only**: For openshift/zero-trust-workload-identity-manager PRs only. | ||
| - **Single PR analysis**: Browser is used once for the PR and Files changed; the same analysis drives all three outputs. | ||
| - **Fixtures**: Install and stack content come from `plugins/oape/ztwim-test-generator/fixtures`; do not discover from repo. | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,115 @@ | ||
| --- | ||
| description: Suggest e2e tests and generate Go code for ZTWIM operator PRs from upstream e2e structure | ||
| argument-hint: "<pr-url> [--output <path>]" | ||
| --- | ||
|
|
||
| ## Name | ||
| oape:ztwim-generate-e2e-from-pr | ||
|
|
||
| ## Synopsis | ||
| ``` | ||
| /oape:ztwim-generate-e2e-from-pr <pr-url> [--output <path>] | ||
| ``` | ||
|
Comment on lines
+10
to
+12
Contributor
There was a problem hiding this comment. Add language tags to fenced code blocks. The synopsis and examples fences don’t specify a language, which triggers MD040. Consider tagging them as 🔧 Suggested tweak-```
+```text
/oape:ztwim-generate-e2e-from-pr <pr-url> [--output <path>]@@ Writes: output/ztwim_pr_123/123_test_e2e.go, output/ztwim_pr_123/e2e-suggestions.md/oape:ztwim-generate-e2e-from-pr https://github.com/openshift/zero-trust-workload-identity-manager/pull/123 --output .work Writes: .work/ztwim_pr_123/123_test_e2e.go, .work/ztwim_pr_123/e2e-suggestions.mdIn |
||
|
|
||
| ## Description | ||
|
|
||
| Analyzes a **ZTWIM** operator Pull Request and (1) suggests which e2e tests to add based on the upstream [test/e2e](https://github.com/openshift/zero-trust-workload-identity-manager/tree/main/test/e2e) structure, and (2) generates **Go e2e test code** in a file **`<prno>_test_e2e.go`** (e.g. `123_test_e2e.go`) that you can copy from into the ZTWIM repo. Tests are written with **operator and operand context** and include **important scenarios** that should be highly checked. | ||
|
|
||
| - **Repository**: openshift/zero-trust-workload-identity-manager only. | ||
| - **Upstream e2e**: test/e2e (e2e_suite_test.go, e2e_test.go, utils). See plugin [docs/e2e-structure.md](../ztwim-test-generator/docs/e2e-structure.md). | ||
|
|
||
| ## Implementation | ||
|
|
||
| ### Step 1: Validate PR and Analyze Changes | ||
|
|
||
| 1. **Validate PR URL** is for ZTWIM: `https://github.com/openshift/zero-trust-workload-identity-manager/pull/<number>`. If not, inform the user this command is for ZTWIM PRs only. | ||
| 2. **Use browser tools**: Navigate to the PR, then to "Files changed" (append `/files` to PR URL). Use **browser_snapshot** to read changed files and paths. | ||
|
|
||
| Map file patterns to e2e focus: | ||
|
|
||
| | File Pattern | E2E Focus | | ||
| |--------------|-----------| | ||
| | `api/**/*_types.go` | New/updated CR fields; consider new or updated It specs for CR create/update and condition checks. | | ||
| | `*controller*.go`, `*reconcile*.go` | Reconciliation and operand lifecycle; OperatorCondition; recovery tests. | | ||
| | `config/crd/**` | Schema/validation; optional negative It (invalid CR). | | ||
| | `config/samples/*.yaml` | Example CR usage; align generated test CRs with samples. | | ||
| | `test/e2e/**` | Follow existing Describe/Context/It style; suggest similar or extended scenarios. | | ||
|
|
||
| ### Step 2: Use Upstream E2E Structure | ||
|
|
||
| Read this plugin's [docs/e2e-structure.md](../ztwim-test-generator/docs/e2e-structure.md). Generated code must: | ||
|
|
||
| - **Package**: `e2e`. | ||
| - **Imports**: Same as upstream (e.g. `context`, `fmt`, Ginkgo, `operatorv1alpha1`, `test/e2e/utils`, `corev1`, `metav1`, `client`, etc.). Do not add unused imports. | ||
| - **Clients**: Use suite-level `k8sClient`, `clientset`, `apiextClient`, `configClient` and per-test `testCtx` (from `BeforeEach` with timeout). Assume these exist; do not redefine. | ||
| - **Helpers**: Use `utils.*` (e.g. `utils.OperatorNamespace`, `utils.WaitForSpireServerConditions`, `utils.WaitForDeploymentAvailable`). Match names from upstream constants.go and utils.go. | ||
| - **Style**: `Describe` / `Context` / `It`; `By("…")` for steps; `DeferCleanup` for teardown; `Eventually` with `WithTimeout`/`WithPolling` where appropriate. For a code-style reference, see plugin fixture [fixtures/e2e-sample_test.go.example](../ztwim-test-generator/fixtures/e2e-sample_test.go.example). For a scenario checklist, see [fixtures/e2e-important-scenarios.md](../ztwim-test-generator/fixtures/e2e-important-scenarios.md). | ||
|
|
||
| ### Step 3: Suggest E2E Tests (Summary) | ||
|
|
||
| Produce a short **suggestion list** for the PR: | ||
|
|
||
| - Which **operator** scenarios apply (install, CRDs, pod recovery, log level via Subscription). | ||
| - Which **operand** scenarios apply (SpireServer, SpireAgent, SpiffeCSIDriver, SpireOIDCDiscoveryProvider: create, conditions, Ready). | ||
| - **ZTWIM** aggregation (operand status, Ready). | ||
| - **OperatorCondition** Upgradeable (True when healthy; False when operand down; recovery). | ||
| - **CR-driven config** (resources, nodeSelector, tolerations, affinity, log level) for any CR touched by the PR. | ||
| - Any **PR-specific** It blocks (new fields, new validation, changed behavior). | ||
|
|
||
| Mark which scenarios are **highly recommended** for this PR given the Files changed. | ||
|
|
||
| ### Step 4: Generate `<prno>_test_e2e.go`** | ||
|
|
||
| Generate a single Go file named **`<prno>_test_e2e.go`** (e.g. `123_test_e2e.go`). The file must be **self-contained** for the package `e2e` and assume the suite (e2e_suite_test.go) and utils are present. | ||
|
|
||
| - **Header**: Standard Apache-2.0 copyright and package `e2e`. | ||
| - **Imports**: Only what is needed; match upstream style. | ||
| - **Structure**: One or more `Describe` or `Context` blocks. Each `It` should be **commented** with a short line (e.g. `// PR-suggested: operator recovery`) so the user can easily pick and choose which tests to copy into `e2e_test.go` or keep in this file. | ||
| - **Content**: Include both (a) **important scenarios** from [docs/e2e-structure.md](../ztwim-test-generator/docs/e2e-structure.md) that are relevant to the PR, and (b) **PR-specific** tests derived from Files changed. Prefer reusing existing utils and condition names; if the PR adds new API fields or conditions, generate plausible code and add a comment that the condition/field name may need to match the actual API. | ||
| - **No duplicate suite logic**: Do not define `BeforeSuite`, `TestE2E`, or client setup; only test blocks that run inside the existing suite. | ||
|
|
||
| ### Step 5: Important Scenarios to Include (when relevant) | ||
|
|
||
| Always consider including or suggesting tests for: | ||
|
|
||
| 1. Operator installed; all managed CRDs Established; operator Deployment Available. | ||
| 2. Operator recovers from force pod deletion (new pod Running, deployment Available again). | ||
| 3. ZeroTrustWorkloadIdentityManager created with trust domain, cluster name, bundle ConfigMap. | ||
| 4. SpireServer / SpireAgent / SpiffeCSIDriver / SpireOIDCDiscoveryProvider created and respective conditions (e.g. StatefulSetAvailable, DaemonSetAvailable, DeploymentAvailable, Ready) True. | ||
| 5. ZeroTrustWorkloadIdentityManager aggregates 4 operands; each operand Ready in status. | ||
| 6. OperatorCondition Upgradeable: True when healthy; False when an operand pod is deleted; True again after recovery. | ||
| 7. CR-driven configuration: SpireServer (or other operand) resources, nodeSelector, tolerations, affinity, log level; operator log level via Subscription env. | ||
|
|
||
| Generate at least 2–3 **highly recommended** It blocks that directly exercise code or CRs touched by the PR. | ||
|
|
||
| ### Step 6: Output | ||
|
|
||
| **All generated files go inside a single output directory.** | ||
|
|
||
| - **Output directory**: `<output-dir>/ztwim_pr_<number>/` (e.g. `output/ztwim_pr_123/`). | ||
| - **Default `<output-dir>`**: `output` (relative to workspace root). Create it if it does not exist. | ||
| - **With `--output <path>`**: Use `<path>` as the output base; write into `<path>/ztwim_pr_<number>/`. | ||
|
|
||
| - **Suggestion list**: Write **e2e-suggestions.md** inside that directory (and optionally show in reply). Content: which operator/operand scenarios apply, highly recommended tests, PR-specific It blocks. | ||
| - **Go file**: Write **`<prno>_test_e2e.go`** inside that directory (e.g. `output/ztwim_pr_123/123_test_e2e.go`). Tell the user they can copy this file into the ZTWIM repo under `test/e2e/` or copy individual It blocks into `e2e_test.go`. | ||
|
|
||
| ## Arguments | ||
|
|
||
| - **$1 (pr-url)**: ZTWIM operator GitHub PR URL — `https://github.com/openshift/zero-trust-workload-identity-manager/pull/<number>`. | ||
| - **--output**: Output base directory (optional). Default: `output`. Generated files go in `<output>/ztwim_pr_<number>/`. | ||
|
|
||
| ## Examples | ||
|
|
||
| ``` | ||
| /oape:ztwim-generate-e2e-from-pr https://github.com/openshift/zero-trust-workload-identity-manager/pull/123 | ||
| # Writes: output/ztwim_pr_123/123_test_e2e.go, output/ztwim_pr_123/e2e-suggestions.md | ||
|
|
||
| /oape:ztwim-generate-e2e-from-pr https://github.com/openshift/zero-trust-workload-identity-manager/pull/123 --output .work | ||
| # Writes: .work/ztwim_pr_123/123_test_e2e.go, .work/ztwim_pr_123/e2e-suggestions.md | ||
| ``` | ||
|
|
||
| ## Notes | ||
|
|
||
| - **ZTWIM only**: For openshift/zero-trust-workload-identity-manager PRs only. | ||
| - **Browser**: Use browser tools for PR and Files changed; do not rely on gh CLI for file list. | ||
| - **Pick-and-choose**: The generated Go file is meant for choosing which tests to add; comments in the file should make it easy to copy only the needed blocks. | ||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Add a language identifier to the synopsis code fence (MD040).
This keeps markdownlint clean and consistent with other docs.
✅ Suggested fix
In
@plugins/oape/commands/ztwim-generate-all.mdaround lines 10 - 12, The codefence in the synopsis for the /oape:ztwim-generate-all command lacks a language
identifier; update the triple-backtick block that contains
"/oape:ztwim-generate-all [--output ] [--env ]" to
include a language token (e.g., shell) after the opening backticks so the fence
reads ```shell and matches the rest of the docs and satisfies markdownlint
MD040.