Javascript generated content is not rendering with Spider

[todd-bush]

Hi all,

The site I'm trying to scrape is https://clinicaltrials.gov/study/NCT01310673?cond=Gout&rank=3.

The main content I'm specifically looking for is with the <app-root> elements, but with Spider I'm receiving

<app-root></app-root>

I am using the Python Client.

crawl_params = {
    'request':'chrome', 
    'limit': 1, 
    'proxy_enabled': True,
    'wait_for':
    {
        'idle_network':{'timeout':{'secs':5,'nanos':0}},
        'dom': {'selector':'main', 'timeout':{'secs':5,'nanos':0}},
        'selector':
            {'selector':'#studyDetailsInfo'},
        }
    }

spider = Spider(api_key=os.environ.get('SPIDER_API_KEY'))

protocol_site = 'https://clinicaltrials.gov/study/NCT01310673?cond=Gout&rank=3'

docs = spider.crawl_url(protocol_site, crawl_params)

print(docs)

I'm attempting to wait on the <main> or an element with the studyDetailsInfo id, both of which reside within the elements.

Any assistance would be greatly appreciated.

[j-mendez]

Hi all,

The site I'm trying to scrape is https://clinicaltrials.gov/study/NCT01310673?cond=Gout&rank=3.

The main content I'm specifically looking for is with the <app-root> elements, but with Spider I'm receiving

<app-root></app-root>

I am using the Python Client.

crawl_params = {
    'request':'chrome', 
    'limit': 1, 
    'proxy_enabled': True,
    'wait_for':
    {
        'idle_network':{'timeout':{'secs':5,'nanos':0}},
        'dom': {'selector':'main', 'timeout':{'secs':5,'nanos':0}},
        'selector':
            {'selector':'#studyDetailsInfo'},
        }
    }

spider = Spider(api_key=os.environ.get('SPIDER_API_KEY'))

protocol_site = 'https://clinicaltrials.gov/study/NCT01310673?cond=Gout&rank=3'

docs = spider.crawl_url(protocol_site, crawl_params)

print(docs)

I'm attempting to wait on the <main> or an element with the studyDetailsInfo id, both of which reside within the elements.

Any assistance would be greatly appreciated.

Hi, thank you for the issue. This is now resolved for this instance. If you see issues like this happen try using the param disable_intercept to allow all js scripts to render.