v2.43.0

What's New

Token Usage Tracking for RemoteMultimodalEngine

The remote multimodal automation engine now tracks and returns token usage conforming to the OpenAI API format:

• AutomationUsage struct with prompt_tokens, completion_tokens, total_tokens
• Usage is accumulated across all inference rounds
• Stored on Page.remote_multimodal_usage

Extraction Support

New extraction capabilities for RemoteMultimodalEngine, similar to the OpenAI integration:

• extra_ai_data - Enable extraction mode
• extraction_prompt - Custom extraction instructions
• screenshot - Capture final screenshot

Extracted data is automatically stored on Page.extra_remote_multimodal_data as AutomationResults.

Example Usage

use spider::features::automation::RemoteMultimodalConfigs;

let mm = RemoteMultimodalConfigs::new(
    "http://localhost:11434/v1/chat/completions",
    "qwen2.5-vl",
)
.with_extra_ai_data(true)
.with_extraction_prompt(Some("Extract all product names and prices"))
.with_screenshot(true);

website.configuration.remote_multimodal = Some(Box::new(mm));

// After crawling, access on page:
for page in website.get_pages().await {
    if let Some(usage) = &page.remote_multimodal_usage {
        println!("Tokens: {:?}", usage);
    }
    if let Some(data) = &page.extra_remote_multimodal_data {
        for result in data {
            println!("Extracted: {:?}", result.content_output);
        }
    }
}

Full Changelog: v2.42.0...v2.43.0

v2.42.0

WebDriver Support

Full W3C WebDriver protocol support via thirtyfour crate for Selenium Grid, remote browsers, and cross-browser testing.

use spider::website::Website;
use spider::features::webdriver_common::{WebDriverConfig, WebDriverBrowser};

let mut website = Website::new("https://example.com")
    .with_webdriver(
        WebDriverConfig::new()
            .with_server_url("http://localhost:4444")
            .with_browser(WebDriverBrowser::Chrome)
            .with_headless(true)
    )
    .build()
    .unwrap();

website.crawl().await;

Features

• Browser Support: Chrome, Firefox, Edge via WebDriver protocol
• Remote Connections: Selenium Grid, BrowserStack, SauceLabs compatible
• Stealth Mode: Automation marker cleanup when webdriver_stealth enabled
  • Hides navigator.webdriver
  • Spoofs navigator.deviceMemory
  • Cleans CDP markers (cdc_, $cdc_)
  • Removes Selenium markers
• Automation Scripts: Reuses existing chrome automation configuration

Feature Flags

[dependencies]
spider = { version = "2.42", features = ["webdriver"] }
# With stealth:
spider = { version = "2.42", features = ["webdriver", "webdriver_stealth"] }

New Files

• spider/src/features/webdriver.rs
• spider/src/features/webdriver_common.rs
• spider/src/features/webdriver_args.rs
• examples/webdriver.rs
• examples/webdriver_remote.rs

v2.41.1

v2.41.0 - WebDriver Support

This release adds WebDriver support via the thirtyfour crate, enabling browser
automation using the W3C WebDriver protocol. Connect to remote Selenium Grid,
chromedriver, geckodriver, and more.

New Features

WebDriver Integration

• Automatic crawl integration: When webdriver_config is set, crawl()
  automatically uses WebDriver instead of raw HTTP
• Multiple browser support: Chrome, Firefox, and Edge
• Remote server support: Connect to Selenium Grid, BrowserStack, or any W3C
  WebDriver-compatible server
• Stealth mode: Navigator property overrides to reduce automation detection

New Feature Flags

Flag	Description
webdriver	Core WebDriver support via thirtyfour
webdriver_headed	Run browser in headed (visible) mode
webdriver_stealth	Enable stealth mode to hide automation
webdriver_screenshot	Screenshot capture support
webdriver_chrome	Chrome-specific features
webdriver_firefox	Firefox-specific features
webdriver_edge	Edge-specific features

Usage

use spider::website::Website;                                                   
use spider::features::webdriver_common::{WebDriverConfig, WebDriverBrowser};    
                                                                                
let mut website = Website::new("https://example.com")                           
    .with_webdriver(                                                            
        WebDriverConfig::new()                                                  
            .with_server_url("http://localhost:4444")                           
            .with_browser(WebDriverBrowser::Chrome)                             
            .with_headless(true)                                                
    )                                                                           
    .build()                                                                    
    .unwrap();                                                                  
                                                                                
// crawl() automatically uses WebDriver                                         
website.crawl().await;                                                          
                                                                                
Examples                                                                        
                                                                                
Three new examples added:                                                       
- webdriver - Basic WebDriver crawling                                          
- webdriver_remote - Remote Selenium Grid connection                            
- webdriver_screenshot - Screenshot capture                                     
                                                                                
# Start chromedriver                                                            
chromedriver --port=4444                                                        
                                                                                
# Or use Selenium Grid with Docker                                              
docker run -d -p 4444:4444 --shm-size="2g" selenium/standalone-chrome:latest    
                                                                                
# Run example                                                                   
cargo run --example webdriver --features="webdriver webdriver_stealth"          
                                                                                
Full Changelog                                                                  
                                                                                
https://github.com/spider-rs/spider/compare/v2.40.5...v2.41.1

v2.40.2

Whats Changed

Solve web challenges, perform actions, and more with remote multimodal iterative automation.

• Remote Multimodal Engine for Chrome automation using vision + LLM
• Iterative automation loop:
  capture → infer plan → execute → re-capture → repeat
• Unified RemoteMultimodalConfigs to configure:
  • API endpoint
  • Model selection
  • Prompts
  • Retry behavior
  • Capture strategies
• Strict JSON automation plans:
  { "label": "...", "done": true|false, "steps": [...] }

website.with_remote_multimodal(Some(
    RemoteMultimodalConfigs::new("http://localhost:11434/v1/chat/completions", "GLM-4.7-Flash")
        .with_api_key(Some(API_KEY))
));

Full Changelog: v2.39.14...v2.40.2

v2.39.14

What's Changed

This release brings built in Chrome gemini nano support and remote vision support.

• Add with_on_should_crawl_callback_closure by @WilliamVenner in #346
• feat(solver): add built in gemini nano support

New Contributors

• @WilliamVenner made their first contribution in #346

Full Changelog: v2.38.122...v2.39.14

v2.38.122

What's Changed

• fix(chrome): add automatic chrome executable detection by @yebei199 in #343
• feat(gemini): add Gemini AI support for dynamic browser scripting by @swistaczek in #344
• chore(smart): add mismatch cypher retry

New Contributors

• @yebei199 made their first contribution in #343
• @swistaczek made their first contribution in #344

Full Changelog: v2.38.109...v2.38.122

v2.38.109

Whats Changed

Fix smart mode lifecycles loading.

Full Changelog: v2.38.68...v2.38.109

v2.38.70

Whats Changed

Fix smart mode re-rendering document content.

Full Changelog: v2.38.44...v2.38.70

v2.38.46

What's Changed

• fix "real_browser" disabled by @rumpl in #336
• fix builder methods wait for
• fix headless http -> https upgrade cf
• fix smart mode re-render tracking and content forwarding

New Contributors

• @rumpl made their first contribution in #336

Full Changelog: v2.37.180...v2.38.46

v2.37.180

What's Changed

• spider_cli: fix duplicated argument -r by @zazolabs in #324
• chore(chrome): fix compile [#328]
• spider_cli: fix download files url empty parse
• feat(spider): add with_max_bytes_allowed to track global browser context bytes for session
• chore(cli): add proxy_url [#330]

New Contributors

• @zazolabs made their first contribution in #324

Full Changelog: v2.37.159...v2.37.180