-
Notifications
You must be signed in to change notification settings - Fork 877
Description
I need to use a socks5 proxy with custom parameters for crawler requests, I am passing it to the crawler via "agent" got setting:
const agent = new SocksProxyAgent(
'socks5://proxyurl', {
...
}
);
const transport = new Crawler({
...
agent: {
https: agent
},
...
});
This works with got, but with crawler request returns an error:
Crawler WARN RequestError: The "msecs" argument must be of type number. Received undefined occurred on "url"
After some debugging, I have found that error occures inside SocksProxyAgent object when it tries to set connection timeout, https://github.com/TooTallNate/proxy-agents/blob/b923f5363875d8fd56477094d5ffc562840a4b76/packages/socks-proxy-agent/src/index.ts#L183 which by default is set to null in constructor, so in code above agent.timeout is null. When preparing got options for request crawler's function cleanObject from utils
Line 76 in 5f6219c
| export const cleanObject = (obj: Record<string, unknown>): Record<string, unknown> => { |
I am not sure why Crawler needs to clean up nulls from its options, but in this case it breaks custom https agent.
Setting timeout option for SocksProxyAgent to integer can be used as workaround, but Crawler should probably not clean up custom request agents.