Data Egineer

На данный момент Data Egineer.

Мои компетенции и опыт

• Designed and developed a custom data ingestion framework supporting databases, APIs, Kafka, and Google Sheets, enabling unified data collection across heterogeneous sources.

• Built and maintained lakehouse architecture using Apache Iceberg, Spark, and dbt on AWS (S3 + Glue Catalog), implementing Bronze/Silver/Gold data layers for structured analytical processing.

• Migrated multiple ETL pipelines from ClickHouse DWH to Snowflake DWH, ensuring zero data loss and minimal downtime during transition.

• Refactored and optimized SQL scripts, reducing memory usage from нужен доступ к резюме GB to 2-21 GB (up to 95% reduction).

• Improved data processing workflows, increasing ETL pipeline performance by 20.3%-42.8%.

• Owned and orchestrated 50+ Airflow DAGs processing 20+ TB daily across multi-terabyte ClickHouse clusters (ReplicatedReplacingMergeTree), including partition-based MERGE operations and deduplication strategies.

• Implemented SCD Type 2 patterns and incremental data models in dbt for accurate historical tracking across financial and user-behavior datasets.

• Developed data quality checks and monitoring for data marts: retention analytics, spend distribution, margin reports, and security withdrawal analysis.

• Managed data pipelines for MySQL, PostgreSQL, and ClickHouse sources using Docker, Git, and CI/CD workflows on Linux.