EMR + Hadoop to Redshift ELT workflow using spark steps API and orchestrated by Apache-Airflow, which ingests disparate datasets focused around 7Gb of I94 arrivals information to produce a simple star schema in Redshift
Daily Incremental load ETL pipeline for Ecommerce company using AWS Lambda and AWS EMR cluster, Deployed using Apache airflow in a docker container.
ETL Data pipeline using aws services
Lambda to start EMR and run a map reduce job
Load data from the Million Song Dataset into a final dimensional model stored in S3.
Analysis performed on data from the Steam platform using Apache Spark and Cloud services such as Amazon Web Services.
A Cloud based Reddit stock sentiment analyzer that analyzes overall sentiment from a configurable selection of stock subreddits for each stock. The architecture utilizes AWS MSK (Kafka), AWS EMR (PySpark) and AWS Lambda (Python 3) for maximum scalability and the OpenAI API for sentiment analysis through prompt engineering.
Data Pipeline Analytics Platform is an end-to-end generic Big Data pipeline. Involves following tech stack: AWS S3, AWS Redshift, AWS EMR Cluster, Apache Spark, Apache Airflow.
With this app, you can see what programming skills are most in-demand in the current job market.
PySpark RDD and DataFrame Examples
ETL Pipeline extracts JSON files from AWS S3 bucket and transforms these using an AWS EMR Spark Cluster and stores the data into an AWS S3 bucket in parquet file format.
Implemented random forest machine learning algorithm using pyspark on AWS EMR to classify the wines. The model is then deployed in docker container.
Built a data model, data warehouse and pipeline for extracting transforming and loading data into a star schema-based data model in a redshift database
In this repo, I build a LogisticRegression prediction model with Dask and PySpark and initialize an AWS EMR cluster to run the entire pipeline.
Developing a Flow with EMR and Airflow
Udacity project: implementing an ETL to process data with Apache Spark and store them in AWS S3 storage
Add a description, image, and links to the aws-emr-clusters topic page so that developers can more easily learn about it.
To associate your repository with the aws-emr-clusters topic, visit your repo's landing page and select "manage topics."