Ta8 forever loop Team Assignment #8 - Data Engineering

[sganes21]

This is a simple step function state machine setup. The step function is composed of extract, transform, and load functions. The extract function extracts data from a publicly available API. For this assignment we used the marketstackAPI. The API requires authentication through a public access key, which we pass into our function as an environmental variable. For this assignment, we queried data on the following companies: Apple (AAPL), Microsoft (MSFT), Amazon (AMZN), Tesla (TSLA), and Nvidia (NVDA). Our transform lambda function transforms the data into a pandas dataframe and then converts the data to a .csv format. The load lambda function contains a lambda function handler to load data onto an S3 object as a csv file. Deployment is handled through a docker container instead of coding directly into an AWS lambda.

A presentation tutorial of our step machine can be found here:

https://drive.google.com/file/d/1D-QBFwL8GaDg1MoPd4yLN8ZmMPabVS4a/view