Data Lake Accelerator Platform
Solo QA on a cloud data engineering platform. Built API test suites, automated AWS S3/Athena/Glue pipeline validation, and created a dataset comparison framework across 4 data zones.
Context
Solo QA alongside 5 developers on an AWS cloud data-engineering platform, for 1 year.
Problem
The platform moved data through ingestion, transformation, and output across 4 data zones with no test coverage: the Django APIs had no automated tests, and nothing verified that data survived S3 transfers, Athena queries, and Glue ETL jobs intact.
Approach
Started with a comprehensive manual test strategy for the whole platform, then automated layer by layer. Built API test suites for the Django backend using Python's requests library, covering all CRUD operations, edge cases, and error handling. Automated S3 data retrieval across same-region and cross-region configurations with Boto3 to validate integrity during transfer and storage, and automated the execution and validation of Athena queries and Glue ETL jobs — checking transformations, schema consistency, and job completion. The key build was a dataset comparison framework using Pandas and DeepDiff that divided and validated data across all 4 zones. Reported results and tracked pipeline bugs through structured MS Excel reports.
Results
- 100% API coverage on the Django backend
- 4 data zones validated end to end
- Cross-region S3 data integrity verified