
Data engineering can feel like a moving target. Every year brings a new orchestrator, a new table format, and a new modern data stack diagram with forty logos on it. It's easy to feel like you're always one tool behind.
Here's the good news: beneath the churn, the core skill set is surprisingly stable. Tools change, but the problems don't: moving data reliably, shaping it so people can use it, and keeping it trustworthy at scale. This post walks through eleven technical skills every data engineer should have, why each one matters, and how to start building it.
1. SQL, Well Beyond the Basics
SQL is still the lingua franca of data, and it's the skill you'll use most. Knowing SELECT and GROUP BY isn't enough. A strong data engineer is fluent in:
Joins and their pitfalls, especially accidental row fan-out and duplicate keys
Window functions for running totals, ranking, deduplication, and sessionization
CTEs and subqueries for readable, maintainable transformations
Query plans, so you can explain why a query is slow instead of guessing
Partitioning, clustering, and indexing, and when each one helps
Here's the kind of query you should be able to write without looking anything up:
SELECT
customer_id,
order_id,
order_total,
SUM(order_total) OVER (
PARTITION BY customer_id
ORDER BY order_date, order_id
ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW
) AS running_total,
ROW_NUMBER() OVER (
PARTITION BY customer_id
ORDER BY order_date DESC, order_id DESC
) AS recency_rank
FROM orders;Notice the explicit ROWS frame and the order_id tie-breaker. Without them, orders placed on the same date can end up with identical running totals or an unstable ranking, a classic source of silent errors.
Skill check: Can you open a query plan and explain where the time is going?
2. Python (and Comfort with a Second Language)
Python is the default glue language of data engineering. You'll use it for ingestion scripts, API integrations, orchestration code, and transformations. Focus on writing real software, not just notebooks:
Clean functions, modules, and packages
Working with files, APIs, and databases (requests, SQLAlchemy, pandas or Polars)
Error handling, retries, and logging
Type hints and unit tests
It also pays to be comfortable with a second language. Many engineers pick up Scala or Java for the JVM data ecosystem, or Go and Rust for performance-sensitive tooling. You don't need to be an expert, but you should be able to read the code behind the tools you depend on.
3. Data Modeling and Warehousing
Pipelines move data; models make it usable. Poor modeling is the most common reason a technically correct pipeline still fails the business.
Learn the classics: normalization, star and snowflake schemas, fact and dimension tables, and slowly changing dimensions. Then learn the modern layers on top, such as the staging, intermediate, and marts layering popularized by analytics engineering, and when a wide denormalized table beats a perfectly normalized one.
You should also understand how analytical databases differ from transactional ones. Columnar storage, compression, and massively parallel processing change which design choices are good ones.
4. Ingestion, ETL/ELT, and Workflow Orchestration
A pipeline that runs once on your laptop is a script. A pipeline that runs every day, recovers from failures, and tells you when something's wrong is engineering.
Understand the difference between ETL and ELT, and why most modern teams load raw data first and transform inside the warehouse. Know the common ingestion patterns too: full loads, incremental loads, API pulls, file drops, and change data capture (CDC), which streams row-level changes out of operational databases instead of repeatedly re-reading whole tables.
Then learn at least one orchestrator well. Apache Airflow remains the most established choice. Airflow 3, released in April 2025, added DAG versioning, event-driven scheduling, and a rebuilt UI, and Airflow 2 reached end of life in April 2026, so that's the version to learn. Dagster and Prefect take different approaches to defining and observing workflows.
The concepts matter more than the tool:
Idempotency: rerunning a job should never corrupt or duplicate data
Backfills: reprocessing history safely and predictably
Dependencies and scheduling: knowing what must finish before what
Retries, alerts, and SLAs: failing loudly instead of silently
Add a SQL transformation framework like dbt and you have the backbone of most analytics stacks. dbt Labs has since merged with Fivetran, and the next major version, dbt Core 2.0, is in alpha, so check the current docs before settling on a version.
5. Batch and Streaming Processing
Sooner or later, a dataset will be too large for one machine, or a stakeholder will need data in seconds instead of hours. That's where distributed processing comes in.
For batch, learn Apache Spark: how it partitions data, what causes shuffles, why skew hurts, and how to read the execution UI. For streaming, learn the fundamentals of event-driven systems with Apache Kafka (or a managed equivalent), and understand how engines like Flink or Spark Structured Streaming handle state.
The ideas to internalize are event time versus processing time, windowing, late-arriving data, and delivery guarantees (at-most-once, at-least-once, and exactly-once, which in practice usually means at-least-once delivery combined with idempotent or transactional writes). Not every problem needs streaming, and knowing when it doesn't is part of the skill.
6. Cloud Platforms and Storage Formats
Most data engineering happens in the cloud, so get hands-on with at least one major provider (AWS, Google Cloud, or Azure). Focus on the building blocks:
Object storage (S3, GCS, Azure Blob) as the foundation of the data lake
Managed warehouses and lakehouse engines (BigQuery, Snowflake, Redshift, Databricks)
Identity and access management, networking basics, and cost controls
Equally important is knowing your file and table formats. Understand why columnar formats like Parquet beat CSV and JSON for analytics. Then learn how open table formats add transactions, schema evolution, and time travel on top of Parquet files in object storage. Apache Iceberg has become the default choice for many new open lakehouses, while Delta Lake remains widely used, especially on Databricks, and the two are increasingly interoperable. Learn the catalog layer too, since it's what lets multiple engines safely share the same tables.
7. Data Quality, Testing, and Observability
A pipeline that runs successfully but produces wrong numbers is worse than one that fails. Trust is the real product of a data team, and it's built with testing and monitoring:
Validation tests: not-null, uniqueness, accepted values, and referential integrity
Freshness and volume checks: did the data arrive, and is the row count plausible?
Schema change detection: catching upstream changes before they break downstream consumers
Data contracts: explicit agreements between producers and consumers
Lineage and alerting: knowing what's affected when something breaks
Tools like dbt tests, Great Expectations, and dedicated observability platforms help, but the habit of asking "how will I know if this breaks?" is what really counts.
8. Software Engineering Practices
Data engineering is software engineering with data as the main input. The practices that keep application code healthy keep pipelines healthy too:
Linux and the command line: shell fluency for debugging, scripting, and working on remote machines
Git and code review: every change tracked, reviewed, and reversible
CI/CD: automated tests and deployments instead of manual copying
Docker: reproducible environments that behave the same locally and in production
Infrastructure as code: Terraform or similar tools to define infrastructure in version control
Testing: unit tests for transformation logic and integration tests for pipelines
Teams that treat pipelines as production software spend far less time firefighting.
9. Distributed Systems and Performance Thinking
You don't need a computer science PhD, but you do need a mental model for how distributed systems behave. Understand partitioning, replication, consistency trade-offs, and why network and disk I/O often matter more than raw compute.
Pair that with practical performance and cost awareness. Learn to spot small-file problems, unnecessary full-table scans, data skew, and over-provisioned clusters. In the cloud, an inefficient pipeline isn't just slow, it's expensive, and being able to cut a bill in half is a very visible skill.
10. Security, Privacy, and Governance
Data engineers handle some of an organization's most sensitive information. Know the essentials:
Least-privilege access and role-based permissions
Encryption at rest and in transit
Secrets management (never hard-code credentials)
Handling personally identifiable information through masking, tokenization, and retention policies
Regulatory basics such as GDPR and similar privacy laws
Governance isn't glamorous, but a single leaked table can undo years of goodwill.
11. Data for AI Workloads
More data teams are now asked to feed AI systems, not just dashboards. You don't need to become a machine learning researcher, but you should understand how data flows into these systems:
Unstructured data: extracting and cleaning text from documents, tickets, and web pages
Chunking, embeddings, and freshness: keeping a vector index up to date as source data changes
Permissions: making sure retrieval respects who is allowed to see what
Evaluation and lineage: tracing a bad answer back to the data that produced it
Reproducible datasets: versioned training and evaluation data you can rebuild
The underlying skills are the same ones as before: reliable pipelines, quality checks, and governance. The difference is that mistakes now surface in an AI system's answers, where they're harder to spot.
Where to Start
If that list feels overwhelming, remember that nobody masters all of it at once. A sensible order looks like this:
Foundation: SQL, Python, Git, and the command line
Core craft: data modeling, a warehouse, and one orchestrator
Scale and reliability: Spark, streaming basics, testing, and observability
Depth: cloud architecture, table formats and catalogs, cost optimization, governance, and data for AI
The best way to learn is to build. Create an end-to-end project: ingest data from a real API, land it in object storage, transform it into a modeled warehouse, schedule it with an orchestrator, and add tests and alerts. Put it on GitHub and write about what broke along the way, because that's where the real learning happens.
Final Thoughts
The most valuable data engineers aren't the ones who know the most tools. They're the ones who understand the fundamentals deeply enough to pick up any tool quickly: strong SQL, solid software habits, sound data modeling, and a relentless focus on reliability and trust.
Learn those well, stay curious about the new stuff, and the next shiny logo on the diagram won't worry you at all.
Comments(0)
Join the discussion by logging into your account.
No comments yet. Be the first to comment!