AWS creds provider while running spark app on Kubernetes #117
Description
I'm running spark streaming application on Kubernetes, where-in service-account has the aws IAM role to connect to aws services.
But while running the app is not able to connect to aws with any of the configs below:
"spark.sql.kinesis.aws.credentials.provider": "com.amazonaws.auth.WebIdentityTokenCredentialsProvider"
"spark.sql.kinesis.aws.credentials.provider": "com.amazonaws.auth.DefaultAWSCredentialsProviderChain"
"spark.sql.kinesis.aws.credentials.provider": "com.amazonaws.auth.AWSSessionCredentialsProvider"
Tried both configs awsUseInstanceProfile true and false, with no luck.
The spark app is running fine my local system with DefaultAWSCredentialsProviderChain, but having issues while running on Kubernetes. Could anyone please help on this?