Skills & Tech Stack Matrix
Comprehensive mapping of technologies, experience levels, and real-world project applications
Data Lakehouse & Processing
3 skills
Apache Hudi
Expert
Projects:
- •Real-time data ingestion
- •Spark Streaming Hudi Jobs on EMR Serverless
- •Data Lake optimization
Apache Iceberg
Expert
Projects:
- •Data Lake architecture
- •Query optimization
- •Multi-tenant data ingestion
Apache Spark
Expert
Projects:
- •Batch Framework (1TB/month)
- •Streaming pipelines
- •EMR Serverless jobs
AWS Services
8 skills
AWS Glue
Expert
Projects:
- •ETL jobs
- •Data cataloging
- •Serverless data processing
Amazon EMR
Expert
Projects:
- •Spark clusters
- •EMR Serverless 7.5.0
- •Big data processing
S3 Tables
Expert
Projects:
- •Multi-tenant data ingestion (Zeta Global)
- •Data Lake storage
- •Cost optimization
AWS Lambda
Advanced
Projects:
- •Event-driven processing
- •Real-time data pipelines
- •Serverless architecture
S3
Expert
Projects:
- •Data Lake storage
- •Multi-tenant architecture
- •Cost optimization
AWS Compute
Expert
Projects:
- •EC2
- •EMR clusters
- •Serverless compute
EventBridge
Advanced
Projects:
- •Real-time event processing
- •MongoDB streams integration
SQS
Advanced
Projects:
- •Message queuing
- •Event-driven architecture
Languages & Tools
4 skills
PySpark
Expert
Projects:
- •Data processing pipelines
- •ETL workflows
- •Real-time analytics
Python
Expert
Projects:
- •Data engineering
- •ETL pipelines
- •API development
Java
Advanced
Projects:
- •Backend services
- •Enterprise applications
SQL
Expert
Projects:
- •Data querying
- •Database optimization
- •Analytics
Query Engine
7 skills
Trino
Advanced
Projects:
- •Distributed query engine
- •Data Lake queries
Apache Spark
Expert
Projects:
- •SQL queries
- •Data processing
- •Analytics
Apache Flink
Advanced
Projects:
- •Stream processing
- •Real-time analytics
Amazon Athena
Expert
Projects:
- •50X query optimization
- •Data Lake querying
- •Cost reduction (80% savings)
Amazon Redshift
Advanced
Projects:
- •Data warehousing
- •Analytics queries
StarRocks
Proficient
Projects:
- •OLAP queries
- •Real-time analytics
Snowflake
Advanced
Projects:
- •Cloud data warehouse
- •Analytics and reporting
Orchestration
3 skills
Apache Airflow
Advanced
Projects:
- •Workflow orchestration
- •Data pipeline scheduling
AWS Step Functions
Advanced
Projects:
- •Workflow orchestration
- •State machine design
Custom Scheduler (AWS)
Expert
Projects:
- •Internal scheduling framework
- •1000+ jobs management
- •Scalable architecture
Streaming
1 skill
Apache Kafka
Advanced
Projects:
- •Real-time data streaming
- •Event-driven architecture
Search & Analytics
1 skill
Elasticsearch
Proficient
Projects:
- •Search and analytics
- •Log aggregation
Additional Technologies & Tools
Docker
Terraform
Kubernetes
CI/CD
Git
Jenkins
MongoDB
DynamoDB
PostgreSQL
MySQL
Redis
REST APIs
GraphQL
Microservices
Event-Driven Architecture
Data Modeling
Data Warehousing
ETL/ELT
Data Governance
Cost Optimization
Performance Tuning
Scalable Architecture
Real-time Processing
Batch Processing
Stream Processing
AI/ML Integration
Machine Learning
Data Science
Cloud Architecture
Distributed Systems
Software Engineering
API Development
System Design
Infrastructure as Code