About the role
Senior Quality Engineer Req ID: 1452887
Big Data QA Engineer (Data Quality & Analytics)
Job Summary
We are seeking an experienced Big Data QA Engineer with strong expertise in data quality assurance, ETL/ELT validation, cloud data platforms, and analytics testing. The ideal candidate will have hands-on experience validating large-scale data pipelines, ensuring data integrity across cloud environments, and leveraging Generative AI to automate and enhance data quality processes.
This role requires a strong understanding of Google BigQuery, SQL, ETL validation, Big Data technologies, and cloud-based analytics platforms. Experience with Claude, Agentic AI, RAG, and Docker is essential to support AI-driven data validation and intelligent quality assurance.
Key Responsibilities
Data Quality Assurance
Required Qualifications
Required Technical Skills
Big Data
Preferred Skills
Domain Experience
Soft Skills
Primary Technologies
Google BigQuery SQL ETL ELT GCP AWS Apache Spark Apache Beam Google Dataflow Python Shell Scripting Tableau Looker Data Quality Data Validation Claude Generative AI Agentic AI RAG Docker JIRA Confluence Agile
Big Data QA Engineer (Data Quality & Analytics)
Job Summary
We are seeking an experienced Big Data QA Engineer with strong expertise in data quality assurance, ETL/ELT validation, cloud data platforms, and analytics testing. The ideal candidate will have hands-on experience validating large-scale data pipelines, ensuring data integrity across cloud environments, and leveraging Generative AI to automate and enhance data quality processes.
This role requires a strong understanding of Google BigQuery, SQL, ETL validation, Big Data technologies, and cloud-based analytics platforms. Experience with Claude, Agentic AI, RAG, and Docker is essential to support AI-driven data validation and intelligent quality assurance.
Data Quality Assurance
- Design, develop, and execute comprehensive test plans for Big Data and analytics platforms.
- Validate data quality, completeness, consistency, and accuracy across large-scale datasets.
- Perform data reconciliation, schema validation, and data integrity checks throughout ETL/ELT pipelines.
- Identify, investigate, and resolve data discrepancies across multiple data sources.
- Validate business rules and ensure accurate reporting for downstream analytics and business intelligence.
- Validate ETL/ELT processes to ensure accurate extraction, transformation, and loading of enterprise data.
- Perform end-to-end testing of data ingestion, transformation, and reporting workflows.
- Validate high-volume data pipelines and monitor data quality across cloud platforms.
- Execute data reconciliation and schema validation between source and target systems.
- Support testing of distributed data processing frameworks.
- Validate data pipelines hosted on Google Cloud Platform (GCP) and Amazon Web Services (AWS).
- Perform data validation using Google BigQuery and SQL.
- Support testing of cloud-native data ingestion and transformation processes.
- Ensure reliable and scalable data quality across enterprise cloud environments.
- Develop automated data validation scripts using Python, Shell Scripting, or similar technologies.
- Build reusable automation utilities for regression and reconciliation testing.
- Enhance automation coverage to reduce manual validation efforts.
- Utilize Claude (mandatory) to create, optimize, and maintain SQL queries, reconciliation logic, validation rules, and analytical summaries.
- Implement AI-driven automation to replace manual data validation processes.
- Apply Generative AI for anomaly detection, trend analysis, intelligent reconciliation, and root cause analysis.
- Develop RAG (Retrieval-Augmented Generation) solutions using data dictionaries, metadata, pipeline documentation, and historical defects to improve AI accuracy.
- Build Agentic AI workflows that automate metadata validation, pipeline analysis, issue summarization, and recommended next actions.
- Containerize validation utilities and AI solutions using Docker for consistent execution across environments.
- Work closely with Data Engineers, Analytics Engineers, BI Developers, QA teams, and Product Owners.
- Participate in Agile ceremonies, sprint planning, backlog grooming, and release activities.
- Communicate data quality findings to both technical and business stakeholders.
- Minimum 5 years of experience in Software Quality Assurance with a strong focus on Big Data Testing and Analytics Quality Assurance.
- Strong hands-on experience with Google BigQuery.
- Advanced SQL skills for data validation and reconciliation.
- Experience validating ETL/ELT processes.
- Strong understanding of data quality principles and data integrity validation.
- Experience working with cloud platforms:
- Google Cloud Platform (GCP)
- Amazon Web Services (AWS)
- Excellent analytical, debugging, and problem-solving skills.
- Strong communication skills with the ability to explain technical findings to business stakeholders.
Big Data
- Google BigQuery
- SQL
- Big Data Testing
- Data Validation
- Data Reconciliation
- Data Integrity
- Data Quality
- ETL Testing
- ELT Validation
- Data Pipelines
- Schema Validation
- Source-to-Target Validation
- Apache Beam
- Google Dataflow
- Apache Spark
- Google Cloud Platform (GCP)
- Amazon Web Services (AWS)
- Python
- Shell Scripting
- Automation Frameworks
- Data Quality Automation
- Claude (Mandatory)
- Generative AI
- Agentic AI
- RAG (Retrieval-Augmented Generation)
- AI-Assisted Data Validation
- Intelligent Anomaly Detection
- Root Cause Analysis (RCA)
- Docker
- CI/CD Pipelines
- Tableau
- Looker
- JIRA
- Confluence
- Agile/Scrum
- Experience testing enterprise data warehouses and analytics platforms.
- Experience validating streaming analytics and media data.
- Knowledge of enterprise metadata management and data governance.
- Familiarity with business intelligence reporting validation.
- Experience working with large-scale cloud-native data platforms.
- Media & Entertainment (Preferred)
- Streaming Analytics
- Digital Analytics
- Business Intelligence
- Enterprise Data Platforms
- Strong analytical and critical thinking abilities.
- Excellent troubleshooting and problem-solving skills.
- Strong written and verbal communication.
- Ability to collaborate with cross-functional teams.
- Self-driven with a continuous improvement mindset.
Google BigQuery SQL ETL ELT GCP AWS Apache Spark Apache Beam Google Dataflow Python Shell Scripting Tableau Looker Data Quality Data Validation Claude Generative AI Agentic AI RAG Docker JIRA Confluence Agile