🎨 UI for the Free Data Engineering Zoomcamp Course provided by DataTalksClub
-
Updated
Aug 2, 2024 - Python
#
mageai
Here are 17 public repositories matching this topic...
Contains a Python script designed to validate data replication tasks in AWS Database Migration Service (DMS). The script checks for potential issues such as missing or duplicate data in the tables being replicated.
-
Updated
Apr 1, 2025 - Python
The Data Engineering Zoomcamp covers essential skills in containerization, workflow orchestration, data warehousing, analytics engineering, batch, and streaming processing. It includes tools like Docker, Terraform, BigQuery, dbt, Spark, Kafka, Kestra, Postgres, Google Data Studio, and Metabase.
docker bigquery streaming kafka spark terraform postgresql plotly orchestration metabase data-engineering dbt workflow-automation warehousing google-data-studio streamlit kestra duckdb azurite-engine mageai
-
Updated
Apr 1, 2025 - Python
The AWS DMS Task Restart and Status Checker is a Python script designed to restart various AWS Database Migration Service (DMS) tasks and check their status. This script leverages AWS SDK (Boto3) and Mage.ai for DMS task integration and management, ensuring efficient and reliable task handling.
python data-engineering boto3 data-migration task-management script-automation aws-dms aws-dms-task-status mageai dms-tasks-restart
-
Updated
Apr 1, 2025 - Python
An ETL data pipeline that extracts data from source and loads it to destination, automated using mage.ai
-
Updated
Nov 21, 2023 - Python
Automated script developed in Mage.ai to monitor and validate the status of AWS DMS CDC tasks, ensuring data integrity and synchronization. Sends notifications for any detected validation issues.
notifications python validation data-engineering monitoring-scripts aws-dms aws-dms-task-status cdc-data mageai
-
Updated
Apr 1, 2025 - Python
This script automates cleaning up a specific folder in an S3 bucket, deleting all objects within it. It uses secure AWS credentials and is built on the Mage.ai platform. Additionally, the roadmap includes approach handling and logging for greater robustness and monitoring.
python s3-bucket data-engineering boto3 cleanup-script automation-script s3-aws monitoring-automation mageai cloud-storage-management s3-cleanup
-
Updated
Apr 3, 2025 - Python
This project is a Data Engineering solution implemented to validate the data types of columns in PostgreSQL tables in a DataMart. It aims to validate whether the data stored in tables conforms to the expected data types, improving data integrity and reliability.
-
Updated
Jul 28, 2024 - Python
pokéFastApi: A backend API with Docker Compose integration for retrieving and searching Pokémon data using FastAPI and DuckDB.
-
Updated
Jan 11, 2024 - Python
Using Mage.ai orchestration tool for ETL process
-
Updated
Mar 28, 2024 - Python
-
Updated
Jun 20, 2024 - Python
Data Engineering Swiss Air Quality
-
Updated
Mar 31, 2024 - Python
Real-time data pipeline for collecting, processing, and visualizing telemetry data from Assetto Corsa Competizione using Mage, Kafka, PostgreSQL, and Metabase for interactive insights into vehicle performance and driver statistics.
-
Updated
Sep 29, 2024 - Python
DE Zoomcamp 2024 Final Project 🧙
python bigquery google-cloud data-warehouse data-engineering data-lake dbt etl-pipeline dbt-cloud looker-studio mageai
-
Updated
Jul 24, 2024 - Python
Improve this page
Add a description, image, and links to the mageai topic page so that developers can more easily learn about it.
Add this topic to your repo
To associate your repository with the mageai topic, visit your repo's landing page and select "manage topics."