Skip to content
This repository was archived by the owner on Aug 17, 2025. It is now read-only.

Commit 435f625

Browse files
NewJerseyStyleNewJerseyStyle
authored
Update from Gemini
1 parent 711cd01 commit 435f625

File tree

4 files changed

+252
-160
lines changed

4 files changed

+252
-160
lines changed

‎README.md‎

Lines changed: 123 additions & 110 deletions
Original file line numberDiff line numberDiff line change
@@ -1,110 +1,123 @@
1-
# AI Website Navigation Testing
2-
Proof of concept for an AI-powered system that uses a screen reader to assess the user experience (UX) of a website for users with visual impairments. The AI simulates human interaction, providing a report on achieved and failed goals, focusing on ease of use and accessibility.
3-
4-
## Project Overview
5-
6-
The project aims to automate the process of verifying website accessibility and UX by using a screen reader to interact with the website and extract information about the completion status of specific tasks or goals. The AI component analyzes the screen reader output to identify whether each goal was successfully achieved or not, and provides insights into potential usability issues.
7-
8-
## Goals
9-
10-
The specific goals to be achieved by the system are defined in a configuration file (e.g., `goals.json`). An example might include:
11-
12-
* Navigate to the homepage.
13-
* View the homepage and guess what the site is about.
14-
* Submit the contact form.
15-
* Access the FAQ section and find info about payment method.
16-
17-
18-
## Technology Stack
19-
20-
* **Programming Language:** Javascript
21-
* **Screen Reader:** [NVDA](https://www.nvaccess.org/download/), [Guidepup](https://www.guidepup.dev/)
22-
* **AI Libraries:** Langchain, Zod
23-
* **Webdriver:** [Playwright](https://playwright.dev/)
24-
25-
26-
## Setup and Usage
27-
28-
The tests are run automatically on every push or pull request to the `main` branch using GitHub Actions.
29-
30-
### Configuration
31-
32-
The test configuration is managed through GitHub Actions secrets and variables.
33-
34-
1. **Configure Goals:**
35-
* The test goals are defined in a JSON file. You can modify the `example-goals.json` file to define your own goals.
36-
* Each goal should be an object in an array with a `"goal"` property. Optionally, you can include an `"expect"` property for expected outcomes for interpretation task.
37-
* Example `example-goals.json`:
38-
```json
39-
[
40-
{
41-
"goal": "Navigate to the homepage."
42-
},
43-
{
44-
"goal": "View the homepage and guess what the site is about.",
45-
"expect": "It is the source code repository of software."
46-
}
47-
]
48-
```
49-
50-
2. **Configure Environment Variables:**
51-
* The environment variables for the test are defined in the `.github/workflows/node.js.yml` file and should be configured in your repository's "Settings" > "Secrets and variables" > "Actions".
52-
* **Secrets:**
53-
* `GROQ_API_KEY`: Your Groq API key. This should be stored as a secret.
54-
* **Variables:**
55-
* `GROQ_MODEL`: The Groq model to use (e.g., `llama-3.3-70b-versatile`).
56-
* `GROQ_MODEL_TEMP`: The temperature for the model (e.g., `0`).
57-
* `TEST_ENTRY_URL`: The URL of the website to test.
58-
* `TEST_CASE_JSON_PATH`: The path to the JSON file containing the test goals (e.g., `example-goals.json`).
59-
60-
### Running the Tests
61-
62-
The tests will run automatically when you push a commit to the `main` branch or open a pull request. You can view the test results in the "Actions" tab of your GitHub repository.
63-
64-
For local development, you can still run the tests using Playwright. Make sure you have Node.js and npm installed.
65-
66-
1. **Install Dependencies:**
67-
```bash
68-
npm install
69-
```
70-
2. **Install Playwright Browsers:**
71-
```bash
72-
npx playwright install --with-deps webkit
73-
```
74-
3. **Set Environment Variables:**
75-
Create a `.env` file and add the necessary variables (see above).
76-
4. **Run Tests:**
77-
```bash
78-
npx playwright test
79-
```
80-
81-
82-
## Results
83-
84-
The output will be a simple report detailing:
85-
86-
* **Successfully Achieved Goals:** A list of goals that were successfully completed.
87-
* **Failed Goals:** A list of goals that were not successfully completed, along with potential reasons for failure (if identified). This section should provide actionable feedback for improving the website's accessibility and UX.
88-
* **Accessibility Issues:** A list of identified accessibility issues, such as missing alt text or poor semantic HTML.
89-
90-
## User Experience Considerations
91-
92-
This proof-of-concept prioritizes ease of use for both the AI and human users. The AI's actions are designed to mimic the natural flow of a human user interacting with a website using a screen reader. The system's design aims to be intuitive and straightforward, minimizing any complexities that might hinder accessibility for users with disabilities. Future development will focus on user testing to further refine the UX and ensure accessibility for a wide range of users.
93-
94-
## Testing Methodology
95-
96-
The AI testing methodology is designed to simulate real-world user interactions. This includes:
97-
98-
* **Realistic Navigation:** The AI navigates the website using screen reader commands that a human user would typically employ.
99-
* **Error Handling:** The system incorporates robust error handling to gracefully manage unexpected situations, such as broken links or unexpected website changes.
100-
* **Accessibility Checks:** Beyond goal completion, the system also performs basic accessibility checks, such as verifying the presence of appropriate ARIA attributes and semantic HTML.
101-
102-
## Limitations
103-
104-
1. The AI testing system only consider keyboard navigation. Did not provide insights in touch screen.
105-
2. The AI may need more instruction in order to control the behavior to mimic specific operation styles (e.g. reading each element on website vs jumping to next header).
106-
3. The screen reader support only added for NVDA, VoiceOver also supported by not implemented.
107-
108-
## Contributing
109-
110-
Contributions are welcome! Please feel free to submit pull requests.
1+
# AI Website Navigation Testing
2+
Proof of concept for an AI-powered system that uses a screen reader to assess the user experience (UX) of a website for users with visual impairments. The AI simulates human interaction, providing a report on achieved and failed goals, focusing on ease of use and accessibility.
3+
4+
## Project Overview
5+
6+
The project aims to automate the process of verifying website accessibility and UX by using a screen reader to interact with the website and extract information about the completion status of specific tasks or goals. The AI component analyzes the screen reader output to identify whether each goal was successfully achieved or not, and provides insights into potential usability issues.
7+
8+
## Goals
9+
10+
The specific goals to be achieved by the system are defined in a configuration file (e.g., `goals.json`). An example might include:
11+
12+
* Navigate to the homepage.
13+
* View the homepage and guess what the site is about.
14+
* Submit the contact form.
15+
* Access the FAQ section and find info about payment method.
16+
17+
18+
## Technology Stack
19+
20+
* **Programming Language:** Javascript
21+
* **Screen Reader:** [NVDA](https://www.nvaccess.org/download/), [Guidepup](https://www.guidepup.dev/)
22+
* **AI Libraries:** Langchain, Zod
23+
* **Webdriver:** [Playwright](https://playwright.dev/)
24+
25+
26+
## Setup and Usage
27+
28+
The tests are run automatically on every push or pull request to the `main` branch using GitHub Actions.
29+
30+
### Configuration
31+
32+
The test configuration is managed through GitHub Actions secrets and variables.
33+
34+
1. **Configure Goals:**
35+
* The test goals are defined in a JSON file. You can modify the `example-goals.json` file to define your own goals.
36+
* Each goal should be an object in an array with a `"goal"` property. Optionally, you can include an `"expect"` property for expected outcomes for interpretation task.
37+
* Example `example-goals.json`:
38+
```json
39+
[
40+
{
41+
"goal": "Navigate to the homepage."
42+
},
43+
{
44+
"goal": "View the homepage and guess what the site is about.",
45+
"expect": "It is the source code repository of software."
46+
}
47+
]
48+
```
49+
50+
2. **Configure Environment Variables:**
51+
* The environment variables for the test are defined in the `.github/workflows/node.js.yml` file and should be configured in your repository's "Settings" > "Secrets and variables" > "Actions".
52+
* The test script can use either the Groq API or any other OpenAI-compatible API endpoint.
53+
54+
* **Using Groq (Default):**
55+
* **Secrets:**
56+
* `GROQ_API_KEY`: Your Groq API key.
57+
* **Variables:**
58+
* `GROQ_MODEL`: The Groq model to use (e.g., `llama-3.3-70b-versatile`).
59+
* `GROQ_MODEL_TEMP`: The temperature for the model (e.g., `0`).
60+
61+
* **Using a custom OpenAI-compatible endpoint:**
62+
* **Secrets:**
63+
* `OPENAI_API_KEY`: Your API key for the custom endpoint.
64+
* **Variables:**
65+
* `OPENAI_API_BASE`: The base URL of your custom OpenAI-compatible endpoint.
66+
* `OPENAI_MODEL`: The model to use (e.g., `gpt-4`).
67+
* `OPENAI_MODEL_TEMP`: The temperature for the model (e.g., `0`).
68+
69+
* **Common Variables:**
70+
* `TEST_ENTRY_URL`: The URL of the website to test.
71+
* `TEST_CASE_JSON_PATH`: The path to the JSON file containing the test goals (e.g., `example-goals.json`).
72+
73+
### Running the Tests
74+
75+
The tests will run automatically when you push a commit to the `main` branch or open a pull request. You can view the test results in the "Actions" tab of your GitHub repository.
76+
77+
For local development, you can still run the tests using Playwright. Make sure you have Node.js and npm installed.
78+
79+
1. **Install Dependencies:**
80+
```bash
81+
npm install
82+
```
83+
2. **Install Playwright Browsers:**
84+
```bash
85+
npx playwright install --with-deps webkit
86+
```
87+
3. **Set Environment Variables:**
88+
Create a `.env` file and add the necessary variables depending on the service you want to use (Groq or a custom OpenAI-compatible endpoint).
89+
4. **Run Tests:**
90+
```bash
91+
npx playwright test
92+
```
93+
94+
95+
## Results
96+
97+
The output will be a simple report detailing:
98+
99+
* **Successfully Achieved Goals:** A list of goals that were successfully completed.
100+
* **Failed Goals:** A list of goals that were not successfully completed, along with potential reasons for failure (if identified). This section should provide actionable feedback for improving the website's accessibility and UX.
101+
* **Accessibility Issues:** A list of identified accessibility issues, such as missing alt text or poor semantic HTML.
102+
103+
## User Experience Considerations
104+
105+
This proof-of-concept prioritizes ease of use for both the AI and human users. The AI's actions are designed to mimic the natural flow of a human user interacting with a website using a screen reader. The system's design aims to be intuitive and straightforward, minimizing any complexities that might hinder accessibility for users with disabilities. Future development will focus on user testing to further refine the UX and ensure accessibility for a wide range of users.
106+
107+
## Testing Methodology
108+
109+
The AI testing methodology is designed to simulate real-world user interactions. This includes:
110+
111+
* **Realistic Navigation:** The AI navigates the website using screen reader commands that a human user would typically employ.
112+
* **Error Handling:** The system incorporates robust error handling to gracefully manage unexpected situations, such as broken links or unexpected website changes.
113+
* **Accessibility Checks:** Beyond goal completion, the system also performs basic accessibility checks, such as verifying the presence of appropriate ARIA attributes and semantic HTML.
114+
115+
## Limitations
116+
117+
1. The AI testing system only consider keyboard navigation. Did not provide insights in touch screen.
118+
2. The AI may need more instruction in order to control the behavior to mimic specific operation styles (e.g. reading each element on website vs jumping to next header).
119+
3. The screen reader support only added for NVDA, VoiceOver also supported by not implemented.
120+
121+
## Contributing
122+
123+
Contributions are welcome! Please feel free to submit pull requests.

‎package.json‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -32,6 +32,7 @@
3232
"dependencies": {
3333
"@langchain/core": "^0.3.42",
3434
"@langchain/groq": "^0.2.0",
35+
"@langchain/openai": "^0.2.1",
3536
"commander": "^13.1.0",
3637
"langchain": "^0.3.19",
3738
"zod": "^3.24.2"

‎playwright.config.js‎

Lines changed: 16 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -1,17 +1,17 @@
1-
import { screenReaderConfig } from "@guidepup/playwright";
2-
import { devices } from "@playwright/test";
3-
4-
const config = {
5-
...screenReaderConfig,
6-
reportSlowTests: null,
7-
timeout: 3 * 60 * 1000,
8-
retries: 2,
9-
projects: [
10-
{
11-
name: "webkit",
12-
use: { ...devices["Desktop Safari"], headless: false },
13-
},
14-
],
15-
};
16-
1+
import { screenReaderConfig } from "@guidepup/playwright";
2+
import { devices } from "@playwright/test";
3+
4+
const config = {
5+
...screenReaderConfig,
6+
reportSlowTests: null,
7+
timeout: 3 * 60 * 1000,
8+
retries: 2,
9+
projects: [
10+
{
11+
name: "webkit",
12+
use: { ...devices["Desktop Safari"], headless: false },
13+
},
14+
],
15+
};
16+
1717
export default config;

0 commit comments

Comments
 (0)