Skip to content

Joseph-T-Gordon

A complete log of all recent projects and skills

Menu
  • About me
  • Posts/Projects
  • Resume/Certifications
Menu

Post #175 (AWS) – Big Data Ingestion Pipeline

Posted on December 13, 2023December 13, 2023 by admin

The Big Data Ingestion Pipeline is the architecture for your big data analytics. For instance it could be fully serverless, data is transformed and queries can be ran on data via SQL, the reports of the queries are stored in S3, and the data is then sent to a data warehouse where dashboards can be created.

An ingestion pipeline could be a group of IoT devices that have data stream through kinesis DataStreams, which then passes that data through Kinesis FireHose, that data is then sent to an S3 bucket, utilizing SQS you can then trigger a lambda function that will trigger a Athena SQL query where the output is stored in an S3 bucket, and you can use QuickSight for a dashboard and even put the data in Redshift for further analytics.

Leave a Reply Cancel reply

Your email address will not be published. Required fields are marked *

© 2025 Joseph-T-Gordon | Powered by Minimalist Blog WordPress Theme