Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/copilot-instructions.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ A web scraping application that collects daily frozen custard flavors from multi
app/
scrapers/ # Individual scraper implementations
scraper_base.py # Base class with common functionality
leons.py # Facebook scraper example
bigdeal.py # Facebook scraper example
oscars.py # Selenium scraper example
kopps.py # BeautifulSoup scraper example
locations.yaml # Location configuration registry
Expand Down
10 changes: 5 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -39,15 +39,15 @@ A web scraping application that collects daily flavor information from frozen cu
- **Culver's**
- **Gilles**
- **Kopp's Frozen Custard**
- **Leon's Frozen Custard**
- **Murf's Frozen Custard**
- **Oscar's Frozen Custard**
- **Big Deal Burgers**

## Features
- **Robust scrapers** for each shop using multiple strategies:
- BeautifulSoup for static sites (Kopp's, Bubba's, Murf's)
- Selenium for dynamic sites (Oscar's, Culver's)
- Playwright for Facebook pages (Leon's)
- Playwright for Facebook pages (Big Deal Burgers)
- **Base scraper architecture** with common functionality and error handling
- **Location registry** (locations.yaml) for easy configuration
- **Modern UI** with date-anchored cards and interactive map
Expand Down Expand Up @@ -102,7 +102,7 @@ python -m http.server --directory static 8080

- **Run tests for a specific scraper:**
```bash
pytest tests/test_leons_scraper.py -v
pytest tests/test_bigdeal_scraper.py -v
```

- **Run with coverage:**
Expand Down Expand Up @@ -280,7 +280,7 @@ docker compose up
- Enable debug logging to inspect HTML structure
- Update selectors in the relevant scraper function

5. **Facebook scraper issues (Leon's)**
5. **Facebook scraper issues (Big Deal Burgers)**
- Posts may be truncated - scraper automatically clicks "See more" buttons
- Flavor announcements must match specific patterns (e.g., "is our flavor of the day")
- Check up to 10 recent posts for flavor announcements
Expand Down Expand Up @@ -326,7 +326,7 @@ See the [Scraper Architecture](#scraper-architecture) section above for detailed

4. **Commit with descriptive messages**:
```bash
git commit -m "Add Leon's scraper with Facebook integration"
git commit -m "Add Big Deal Burgers scraper with Facebook integration"
```

5. **Push and create PR**:
Expand Down
11 changes: 0 additions & 11 deletions app/locations.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -87,17 +87,6 @@ gilles:
url: "https://gillesfrozencustard.com/flavor-of-the-day"
enabled: true

leons:
- id: leons-oshkosh
name: "Leon's Frozen Custard"
brand: Leons
address: "121 W Murdock Ave, Oshkosh, WI, United States, 54901"
lat: 44.080552821673784
lng: -88.53003838993757
url: "https://www.facebook.com/LeonsCustard/"
facebook: "https://www.facebook.com/LeonsCustard/"
enabled: true

bigdeal:
- id: bigdeal-main
name: "Big Deal Burgers"
Expand Down
2 changes: 1 addition & 1 deletion app/scrapers/bigdeal.py
Original file line number Diff line number Diff line change
Expand Up @@ -270,7 +270,7 @@ def _scrape_facebook_page_attempt(self, url, attempt):
# Use a more precise heuristic to detect flavor announcements.
# We require a flavor-related word and either a time-related word
# or an explicit announcement pattern. This mirrors the stricter
# approach used in other scrapers (e.g., Leon's) to avoid
# approach used in other Facebook-based scrapers to avoid
# matching generic posts that just mention "today" or "custard".
has_flavor_word = "flavor" in text_lower or "custard" in text_lower
time_keywords = ["today", "daily", "of the day", "tonight"]
Expand Down
Loading