Adarsh Raj

About

I'm a Senior Data Engineer based in Bengaluru, India with 6+ years spent designing and owning scalable data platforms, pipelines, and cloud infrastructure. I build production-grade systems end to end, from ingestion and orchestration to warehousing, modeling, APIs, and the operational workflows a business runs on. What I care about is building platforms that stay reliable and easy for teams to extend as products, headcount, and data volumes grow.

My work covers the full data engineering lifecycle, including ETL and ELT pipeline development, warehouse architecture and data modeling, orchestration with Airflow and Dagster, real-time processing with Kafka and Spark, analytics engineering with dbt, and backend and API work in Python and FastAPI. I move comfortably between batch and event-driven systems, and I have shipped on Snowflake, BigQuery, Redshift, and Databricks across AWS, GCP, and Azure.

I started out working directly with founders and CTOs at YC and Techstars backed startups, making architecture calls from day one and delivering systems other engineers could trust, operate, and extend. I am comfortable owning ambiguous problems, aligning technical decisions with what the product actually needs, and delivering with a high degree of autonomy. Fully remote since 2020, I work well across time zones and communicate proactively from design through delivery.

Metrics

• 6x. Faster Data Operations.

  0x

  Faster Data Operations

• 40%. Cloud Cost Reduced.

  0%

  Cloud Cost Reduced

• 50 TB. Largest Data Migration Delivered.

  0 TB

  Largest Data Migration Delivered

• 50M+. Payment Events Processed Per Day.

  0M+

  Payment Events Processed Per Day

• 30+. Analytics Stakeholders Served With Self-Service.

  0+

  Analytics Stakeholders Served With Self-Service

• 4 hours to 50 seconds. Warehouse Query Time Cut.

  4 hours to 50 seconds

  Warehouse Query Time Cut

• 15,000. Transactions Per Second Handled.

  0

  Transactions Per Second Handled

• 12B. Rows Processed In A Single Project.

  0B

  Rows Processed In A Single Project

• sub-50ms. API Response At Cache Hit.

  sub-0ms

  API Response At Cache Hit

• 6 hours to 10 minutes. Reporting Runtime Cut.

  6 hours to 10 minutes

  Reporting Runtime Cut

Skills

Featured work

• 

  5 diagrams

  Real-Time Transaction Monitoring and Payment Intelligence Platform

  50,000+ events per day, sub-second end-to-end latency, exactly-once delivery

  I architected and developed an event-driven platform that watches payments as they happen and flags fraud within sub-second latency. Card and gateway events from Stripe, payment gateways, and internal services land in Apache Kafka, and Apache Flink runs the fraud and enrichment logic with sliding-window pattern detection, idempotency keys, and a customer-metadata join against PostgreSQL. Results fan out to a live WebSocket dashboard, a Snowflake warehouse for compliance history, and banking webhook callbacks, with failed events routed to a dead-letter queue. It runs multi-AZ with exactly-once delivery, S3 checkpointing for recovery, and Prometheus and Grafana for observability.

  • Apache Kafka
  • Apache Flink
  • Snowflake
  • PostgreSQL
  • FastAPI
  • WebSocket
  • Prometheus
• 

  5 diagrams

  Customer Analytics Data Warehouse with Medallion Architecture on Snowflake

  4 hours to 50 seconds query time, 30+ self-service stakeholders, daily refresh

  I created a medallion-architecture warehouse that turns raw MySQL, Shopify, and Google Analytics data into business-ready facts. Airflow orchestrates ingestion, transformation, and quality DAGs, and dbt handles standardization with SCD Type 2 dimensions in the silver layer and orders, returns, and customer-lifetime-value facts in gold. dbt tests run on every pull request, lineage is auto-documented, and Snowflake zero-copy cloning keeps development and production clean. Query times dropped from 4 hours to 50 seconds for more than thirty self-service stakeholders.

  • Snowflake
  • dbt
  • Apache Airflow
  • Medallion architecture
  • SCD Type 2
  • Tableau
  • Looker
  • Power BI
• 

  5 diagrams

  High-Performance Shipment Data API and Logistics Serving Platform

  10,000+ requests per minute, sub-50ms cache-hit latency, 100% request audit coverage

  I developed a FastAPI serving layer that delivers shipment predictions and route optimizations to mobile apps, partner APIs, and internal dispatch, holding sub-50ms responses under load. A Redis caching layer with TTL expiry sits in front of PostgreSQL, the data warehouse, and an ML route model, while JWT auth with per-consumer RBAC and token-bucket rate limiting protect every endpoint. It runs on a multi-AZ Kubernetes cluster with horizontal pod autoscaling, rolling deploys, and full request auditing through Prometheus, Grafana, and structured logs.

  • FastAPI
  • Redis
  • Kubernetes
  • PostgreSQL
  • JWT and RBAC
  • Prometheus
  • Grafana
• 

  5 diagrams

  Enterprise Data Lakehouse for Manufacturing IoT and ERP Consolidation

  2M+ sensor events per day, streaming and batch unified, one governed source of truth

  I designed and implemented a lakehouse that consolidates high-volume manufacturing IoT sensor streams and ERP records into one governed analytics platform. Streaming sensor data and batch ERP extracts are unified on Databricks and Microsoft Fabric with a medallion layout, so operations and finance work from the same source of truth instead of siloed exports. It is built for both real-time monitoring and historical reporting at manufacturing scale.

  • Databricks
  • Microsoft Fabric
  • Apache Spark
  • Apache Kafka
  • Delta Lake
  • ERP integration
• 

  5 diagrams

  Data Governance and Quality Framework for Healthcare Compliance and Observability

  200+ automated dbt tests, column-level lineage, HIPAA-compliant audit trail

  I worked on a governance and data-quality framework for a HIPAA-sensitive healthcare environment. It layers automated dbt tests, schema and freshness checks, column-level lineage, and access controls over the warehouse, with alerting and observability so data issues surface before they reach a report. The result is auditable, compliant data that teams can trust for clinical and operational decisions.

  • dbt
  • Data quality
  • Column-level lineage
  • HIPAA
  • Observability
  • Access control
• 

  5 diagrams

  Real-Time Telematics and Underwriting Platform

  10M+ telematics events per day, data contracts and SLAs enforced, live underwriting decisions

  I architected real-time telematics and driver-behavior pipelines that feed live underwriting decisions at a commercial trucking insurance company. Dagster orchestrates the ELT, FastAPI services surface underwriting and operational data to internal tools, and clear data contracts, schema standards, and SLAs keep the numbers trustworthy across data science and product.

  • Dagster
  • FastAPI
  • Snowflake
  • Telematics
  • ELT
• 

  5 diagrams

  Unified Operational Reporting and Automation Layer

  3 sources consolidated daily, dbt-tested reporting layer, 90% less manual handling

  For a lean remote team in East African FMCG and supply chain, I owned the full data and automation stack. Airflow pipelines consolidated PostgreSQL, MongoDB, and Airtable into a single reporting layer on AWS with dbt transformations, Python services and automations cut manual handling, and Looker dashboards gave operations real financial and inventory visibility.

  • Apache Airflow
  • dbt
  • AWS
  • Python
  • Looker
• 

  5 diagrams

  AI-Powered Compliance Platform, Founding Build

  10K+ compliance checks automated, OpenAI and Gemini routing, confidence-gated review

  As founding engineer at a Techstars-backed compliance startup, I built the backend from scratch in Python and FastAPI on GCP, owned the data layer across Supabase and BigQuery with Metabase reporting, and integrated OpenAI and Gemini to give the product real AI-driven compliance and automation. I set the GCP architecture and wrote the specs that let the system scale past the founding build.

  • Python
  • FastAPI
  • GCP
  • BigQuery
  • OpenAI
  • Gemini

Experience

1. February 2025 to Present

   Cover Whale

   Senior Data Engineer

   New York, United States (Remote)

   Architected real-time telematics and driver-behavior pipelines feeding underwriting workflows, owned Dagster-orchestrated ELT that kept warehouse movement reliable, built FastAPI services surfacing underwriting and operational data, and defined data contracts and pipeline SLAs with data science and product.

2. January 2024 to January 2025

   Ramani.io

   Senior Data and Automation Engineer

   Dar es Salaam, Tanzania (Remote)

   Owned the full data and automation stack, designed Airflow pipelines that consolidated PostgreSQL, MongoDB, and Airtable into a unified reporting layer on AWS with dbt, and built Python services and Looker dashboards for operations, finance, and inventory.

3. June 2023 to December 2023

   Complya (Techstars '24)

   Founding Senior Software Engineer

   Minnesota (Remote)

   Built the backend from scratch in Python and FastAPI on GCP, owned the data layer across Supabase and BigQuery with Metabase reporting, integrated OpenAI and Gemini into compliance features, and led GCP architecture and deployment.

4. September 2020 to June 2023

   Alopa Infotech

   Software Engineer to Senior Software Engineering Consultant

   Bengaluru

   Built and owned a HIPAA-compliant healthcare product across the full lifecycle in Node.js and Python, managed MySQL and Snowflake operations and pipelines on AWS, and grew from junior work to senior backend and data-infrastructure ownership.

5. January 2020 to July 2020

   Everlytics Data Science

   Data Engineer

   Bengaluru

   Designed ETL and ELT pipelines using AWS, Apache Airflow, and Snowflake, wrote Python data-processing scripts, and built Tableau dashboards for business stakeholders.

Certifications

Let's talk

• adarsh18raj@gmail.com
• +91 9206543619
• LinkedIn
• Download resume
• Bengaluru, India. Available remote worldwide.