Eliminate duplicate inner_text() calls in Facebook scrapers#42
Closed
Eliminate duplicate inner_text() calls in Facebook scrapers#42
Conversation
…32) * Initial plan * fix: Cache Playwright inner_text() to avoid duplicate cross-browser calls Co-authored-by: jjlauterbach <1447549+jjlauterbach@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: jjlauterbach <1447549+jjlauterbach@users.noreply.github.com>
* Initial plan * feat: Add retry logic with exponential backoff to Big Deal scraper Co-authored-by: jjlauterbach <1447549+jjlauterbach@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: jjlauterbach <1447549+jjlauterbach@users.noreply.github.com>
* Initial plan * Add See more expansion and scroll logic to BigDeal scraper Co-authored-by: jjlauterbach <1447549+jjlauterbach@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: jjlauterbach <1447549+jjlauterbach@users.noreply.github.com>
* Initial plan * Add HTML entity decoding to Big Deal Burgers scraper Co-authored-by: jjlauterbach <1447549+jjlauterbach@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: jjlauterbach <1447549+jjlauterbach@users.noreply.github.com>
…aper (#33) * Initial plan * fix: Add page scroll and See more expansion to Big Deal scraper Co-authored-by: jjlauterbach <1447549+jjlauterbach@users.noreply.github.com> * fix: Merge bigdeal branch changes - add _sanitize_flavor_name and fix formatting Co-authored-by: jjlauterbach <1447549+jjlauterbach@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: jjlauterbach <1447549+jjlauterbach@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
…al scraper (#39) * Initial plan * fix: Guard inner_text() against stale element errors; fix test mocks Co-authored-by: jjlauterbach <1447549+jjlauterbach@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: jjlauterbach <1447549+jjlauterbach@users.noreply.github.com>
…er-article (#38) * Initial plan * Mirror Leon's pattern in bigdeal scraper: collect articles, filter top-level, expand See more per-article Co-authored-by: jjlauterbach <1447549+jjlauterbach@users.noreply.github.com> * Integrate new test from base branch to resolve merge conflict Co-authored-by: jjlauterbach <1447549+jjlauterbach@users.noreply.github.com> * Fix test conflicts: keep bigdeal inline mocks, only add helper and update See more test Co-authored-by: jjlauterbach <1447549+jjlauterbach@users.noreply.github.com> * Move query_selector mock setup to avoid hunk conflict with bigdeal's evaluate addition Co-authored-by: jjlauterbach <1447549+jjlauterbach@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: jjlauterbach <1447549+jjlauterbach@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Initial plan * fix: apply black formatting to fix CI build Co-authored-by: jjlauterbach <1447549+jjlauterbach@users.noreply.github.com> * fix: reformat with black 24.4.2 to match pre-commit config Co-authored-by: jjlauterbach <1447549+jjlauterbach@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: jjlauterbach <1447549+jjlauterbach@users.noreply.github.com>
…rapers (#41) * Initial plan * Reduce duplicate inner_text() calls in Facebook scrapers Co-authored-by: jjlauterbach <1447549+jjlauterbach@users.noreply.github.com> * Fix black formatting in leons.py and bigdeal.py to pass CI Co-authored-by: jjlauterbach <1447549+jjlauterbach@users.noreply.github.com> * Fix formatting: use black 24.4.2 (pinned CI version) to reformat files Co-authored-by: jjlauterbach <1447549+jjlauterbach@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: jjlauterbach <1447549+jjlauterbach@users.noreply.github.com> Co-authored-by: Jeff Lauterbach <jjlauterbach@gmail.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Contributor
Author
|
Just as a heads up, I was blocked by some firewall rules while working on your feedback. Expand below for details. |
Copilot
AI
changed the title
[WIP] WIP Address feedback on Big Deal Burgers scraper implementation
Eliminate duplicate inner_text() calls in Facebook scrapers
Feb 22, 2026
Deploying daily-custard with
|
| Latest commit: |
2ed3029
|
| Status: | ✅ Deploy successful! |
| Preview URL: | https://7d4e29d5.daily-custard.pages.dev |
| Branch Preview URL: | https://copilot-sub-pr-30.daily-custard.pages.dev |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
is_facebook_post_from_today()was callingarticle.inner_text()internally, then the scraper called it again immediately after — two cross-process Playwright DOM calls per post when only one is needed.Changes
utils.py: Added optionalarticle_textparameter tois_facebook_post_from_today(); when supplied, skips the internalinner_text()callleons.py/bigdeal.py: Fetcharticle.inner_text()once per post intotext_content, validate it, then pass it viaarticle_text=text_contentto the date-check helpertests/test_utils.py: Addedtest_pre_fetched_text_is_used_without_calling_inner_textto assertinner_text()is never invoked on the article mock whenarticle_textis provided💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.