How to read og:image from a playwright crawler?

[Sir-hennihau]

Somehow I didn't figure out how to do that with this library yet :D. Some help would be appreciated. Would just get the HTML and use a different parsing library or can you do it with the tools provided by crawlee?

const crawler = new PlaywrightCrawler({
    // Use the requestHandler to process each of the crawled pages.
    async requestHandler({ request, page, enqueueLinks, log, pushData }) {
        const title = await page.title(); // <--- How to og image instead of just the title??
        log.info(`Title of ${request.loadedUrl} is '${title}'`);
        // Save results as JSON to ./storage/datasets/default
        await pushData({ title, url: request.loadedUrl });

        // Extract links from the current page
        // and add them to the crawling queue.
        await enqueueLinks();
    },
    // Comment this option to scrape the full website.
    maxRequestsPerCrawl: 20,
    // Uncomment this option to see the browser window.
    // headless: false,
});

// Add first URL to the queue and star

```

[janbuchar]

Hello, and thanks for your interest in crawlee! The og:image is just a meta tag, so you should be able to do something like await page.locator('meta[property=og:image]').first() and see where that gets you.

[Sir-hennihau]

hey @janbuchar , thanks for your answer. somehow, i didnt figure out how to do it the way you described. spent 2-3 hours and didnt manage it. what i ended up doing is to parse with cheerio and use the cheerio selector :D.

const ogImage = $('meta[property="og:image"]').attr("content"); rang the bells for me

[janbuchar]

Well, I'm glad you found a way eventually 😄 If you'd like to, you could supply a complete executable reproduction script and we could try troubleshooting it here.