How to stay on a single dynamic page and continuously retrieve data?

[themoneyevo]

I want to use PlaywrightCrawler to continuously scrape new incoming data from a single dynamic page. However, the page automatically closes. I want to know how to keep the page open continuously without being closed. Additionally, the webpage requires login, and I want to use cookies so that each time I start the crawler, it automatically logs in using the stored cookies.

[janbuchar]

Hello, do I understand it right that you want to load some kind of a web app and periodically check it for new data without ever closing the window? The typical use case for Crawlee is processing many pages with a requestHandler function that is expected to terminate quickly (there is even a timeout for that).

What advantages does PlaywrightCrawler have over using Playwright on its own in your particular use case?

[themoneyevo]

The automatic session management of PlaywrightCrawler is quite appealing to me, but I am now switching to using Playwright to complete my work.

[janbuchar]

That is probably the most practical approach, yes. I guess you could instantiate your own SessionPool and use it to generate a session and then inject it into your Playwright browser, but that would be pretty involved.

If you're staying on the same site for the whole duration of the crawl, you probably couldn't benefit from session rotation anyway.