A portable Datamart and Business Intelligence suite built with Docker, Mage, dbt, DuckDB and Superset

The Zoomcamp MLOps Course covers tools like MLflow, Mage, Flask, Prometheus, Evidently, Grafana, Prefect, Terraform, and GitHub Actions. It emphasizes experiment tracking, model deployment, monitoring, CI/CD, and orchestration, culminating in an end-to-end project integrating best practices in MLOps.

The CNPJ Data ETL Pipeline is designed to automate the download, processing, and storage of public CNPJ data from the Brazilian Federal Revenue. The pipeline is built with Mage.ai and AWS S3 to ensure efficient data management and scalability.

An end-to-end data engineering pipeline that processes and analyzes Maintenance Work Orders using Mage, Docker, Google BigQuery, MariaDB, and Looker Studio. It features a seamless integration of cloud and open-source tools for scalable data storage, transformation, and visualization.

End-to-end data engineering project

Data modeling and ETL pipeline for data analytics on Uber dataset using Google cloud storage, BigQuery, and Looker Studio

Python Lib to build and process data in Datalake - Cloud Function, Cloud Run, Lambda

A data engineering project built around Smogon's Stats API.

Solutions for @DataTalksClub's Data Engineering Zoomcamp 2024.

Docker image for Mage AI deployment using Docker

mlops zoomcamp capstone project

A full data pipeline project. from the ETL to the Dashboard

This repository contains the research, code, and examples related to the orchestration of data pipelines using a microservices architecture. The project explores the challenges of constructing modern data-centric pipelines and evaluates the role of orchestration tools in Data Science workflows.

In this project, I built a data pipeline using Mage.ai for ETL, GCP for storage, BigQuery for querying, and Looker Studio for analytics. This project helped me learn how to process, store, and visualize data effectively using modern tools.

Performed data analytics on Uber data using various tools and technologies, including GCP Storage, Python, Compute Instance, Mage Data Pipeline Tool, BigQuery, and Looker Studio.

Engineered a robust data pipeline using Google Cloud Platform (GCP) services like Cloud Storage, Compute Engine, BigQuery, resulting in a 30% decrease in data processing and analysis time of the San Francisco Crime dataset.Designed an interactive dashboard using Looker Studio, empowering stakeholders to explore crime data visually.

A collection of codes and mini projects that bridge the gap between machine learning and operational excellence.

An end-to-end data engineering project using Amazon S3, EC2, mage.io, Google BigQuery and Looker.