Submit your salary here

I transitioned from a mainframe development position early/mid career (late 20s) into my now System Analyst role. Problem is, the job had been heavily business/third party app support focused as opposed to how it was advertised. I was expecting something more technical focused mixed with reporting and data integration work (which I do some of this but most of my current job is just supporting our operations team and writing some sql).

I really want to transition back into a dev type role and have been eyeing DE since my current role is DA adjacent and it just seems so interesting and fun. Is the transition this late in my career (32 going on 33) worth it? I’m worried I’ll try to study and learn DE topics just for employers to not give me a second glance.

I’m also concerned on keeping a strong salary. Currently make about $78k in a MCOL area and can’t afford to take less due to having children. Any thoughts or advice would be appreciated!

Hello everyone, I have just been placed. Informatica is an entirely new domain for me. While I am proficient in SQL and Python, IDMC presents a completely different set of challenges. Could anyone please recommend any guides or channels that would be beneficial for beginners?

Hey everyone,

What are some good resources to learn data engineering considerations for autonomous driving systems?

If anyone works on this field, please let me know if you’d be up for a chat.

Thanks in advance!

Working on a pipeline using spark/flink where i have to do some transformation work with content in a zip file.

I believe spark and flink don’t natively support regular zip files, and was wondering if anyone has any suggestions on how to handle this.

I am forced by requirements to use zip files, spark and flink, but i can consider adding additional technologies into the workflow if its better to just decompress outside of spark and flink.

Appreciate any suggestions and help!

Both companies I've been at in the past 3-4 years have fought about end user validation, who does it, how do they know what's correct, then mix in deadlines and suddenly the finger pointing begins. I've seen this story before, whenever there's a conflict like this surrounding a handoff it eventually becomes the developers/team responsibility.

I'm sure AI will be the answer and likely speed it up but do you agree about the pattern I'm talking about?

Hi everyone! Here is a brief video on YouTube about modern data systems that I created.

Hi guys, I’m looking for some outside input into my career. I started about 7 years ago as ‘data person’, for a company that has a SaaS application. Over the years I’ve grown and have more of a focus on data engineering. From the start of this year I’ve switched to a function combing data engineering and infrastructure engineering (Azure).

I want to become a Data Architect, but I’m not sure what I should focus on to really have enough knowledge (deep and wide) to call myself Architect.

I did a self-assessment with chatgpt and claude, and now I also want some input from real people :)

Some of the stuff i’ve worked on the last year:
- investigate replacement for Azure Synapse and implement -> Dagster, has been operational since december
- learn and understand the infrastructure in Azure of our company, Terraform, AKS, github, CI/CD, and more

Project I’m currently working on:
- Finalize user action logging in the application, build pipelines and documentation. Find vendor and implement monitoring and alerting (customers are pushing for this, so high priority/impact project, running it solo)
- Work on Azure Security (also alerting and monitoring)
- Azure cost reduction, see where we can save money
- Improve dataset we provide to customers, rebuild to dimensional model and adjust pipelines to fit

Also I’m trying to document as much of my choices and considerations to reflect my thinking process.

One of the things I really miss is pushback on my decisions, other that ‘can you explain why’, since we only have a cloud architect who understands the stuff I do.

So what should I work on? I was thinking maybe DAMA cert to start, then look for a good Security cert, but not sure yet..

I am modeling a relationship between 3 sources. Basically this is the content of each table:

Table 1: requestor table (100s of entities)

Table 2: engine database (10000s of entities)

Table 3: oil sample database (10000s of entities)

What is the best practice for the relationships? I have fields on each table that I would like to use in the PowerBI dashboard.

i work for a consulting company and i'm working in a client that has a lot of their data in old on-prem environments, being more specific, oracle, sql server, postgres.

their workflow is pretty basic: run ADF to collect data from these sources and send into ADLS, then use Databricks to process it. we're planning to modernize their environment (using unity catalog and other new stuff) and one of the things we're thinking is to retire the ADF

the reason is simple: ADF is a pain in the ass (we're having a hard time working with it specially because it was another company that build all that shit, the connection with Azure DevOps/Repos is always horrible to manage and the workflow itself needs an upgrade); and because Microsoft is pushing hard the Fabric Data Factory

i know that Databricks with Lakeflow Conn can connect to on-prem using express route, VPN, but in scenarios where this could not be possible, what tool could be used to send data from on prem to azure data lake?

i like to code so my first suggestion was to use local airflow and simply read from db and upload to adls. it's free, it's versionable and has tons of documentation; but i'd like to test other options before suggesting anything

i was reading about airbyte, the pros and cons of the tool, and looks interesting

how do you guys handle this kind of workload?

Hi there!
I work as a data engineer in a SaaS company. We use Salesforce in our CS and sales departments and one of my obligations is to make data available for reporting, forecasts and also AI projects.
Now we are trying to move Salesforce data into external data warehouse, but the process became much more difficult. The initial load was ok, but later it became hard to keep everything accurate. We need to do a lot of manual work like capture updates and deleted records, keep historical field values. Also sometimes relationships between objects break and we need to fix this. 
We already tried to use Bulk API for the initial load and Change Data Capture for later updates. But this didn’t help much as we were still responsible for API limits, some failed events, backfills and checking whether the warehouse actually matches Salesforce. 
Did any of u have the same issue? Is custom pipeline realistic? Or we better use a special SF tool for it?

As a self-taught developer, I built a platform-agnostic system designed to deterministically detect cumulative drift and anomalies in data streams. It breaks data silos and resolves pipeline bottlenecks directly at the source, all without triggering false alarms.Here are a few screenshots of the architecture and the dashboard. Just wanted to share my project with the community.

Full disclosure: I work at Astronomer and I'm one of the judges.

Beyond the Dag is a data engineering hackathon with an Apache Airflow focus. Take note: this is purely about open-source Airflow!

It's all about showing modern Airflow has evolved beyond just moving data.

Treat Airflow as a platform, extend it with plugins, add agents with the new Common AI provider, and keep a human in the loop where it matters. Use the tracks as inspiration, not limits, and aim for something that shows what Airflow is really capable of.

Why am I sharing this with the community?

I love exploring creative ways to learn frameworks and technologies. In the past, I've built some unusual projects, including gesture-controlled Airflow, native retro game emulation within Airflow featuring Doom, Age of Empires, and others, as well as a Minecraft PaperMC plugin that lets you build Airflow Dags inside Minecraft. I've also participated in many hackathons, and my experience has always been that I learned a lot and met some great people.

Another reason is that hackathons with a clear DE focus are rare, making them valuable additions to your profile. For example, participating in those helped me during my U.S. visa application process.

Because of that, I think this is a great project for learning about Airflow, having some fun, strengthening your data engineering profile, and maybe even winning a Switch 2 😉.

The build window already started and is open until September 24. We kept it relatively relaxed, so it's still time to join. There are also prizes, including a Nintendo Switch 2.

More details and registration at:

https://www.astronomer.io/events/beyond-the-dag-data-engineering-hackathon-2026

Do you understand how Apache Iceberg actually works?

Even after using Apache Iceberg, I did not grok it before doing practical experiments and sitting down to read the official specification thrice. I have written down my understanding in a blog post: https://thingsworthsharing.dev/iceberg/

If you want to dig deeper, but the official specification seems daunting, this is for you!

I recently started my first ever full time role in data engineering. However, it appears that the big data stack for my company has already been established.

Therefore I’m dealing more with bug fixes and new feature implementations but the stack isn’t likely to change unless I make a good case for it. Furthermore, my company heavily depends on AI like Claude and Codex to assist with solving these tasks, which offloads a lot of the coding to these agents.

What kind of recommendation would you senior devs give to a fresh grad like myself? I feel my coding competency going down the more I work. However, since these tasks are offloaded to AI, I do believe that my understanding of data flow in big data architecture has deepened quicker.

Our stack is supports both streaming and batch processing if it matters.

Hi community.

At work, we’re facing a bit of a challenge: we receive a huge volume of JSON-format events from multiple microservices into Kafka topics. We want to build a dashboard to monitor these topics and alert the business if we don't receive the expected events (we’ve already identified specific projects and know the expected TPS for each). Do you know of any monitoring tools that might work for us?

Thanks in advance!

Those who went fully remote, what's your thoughts

I currently work hybrid 2 days a week in the office in London which is fine, I'm principal data engineer and this gives me some face time with direct reports and the regional ones I influence. But it's at least 6 hours a week commute and £6k train tickets a year.

I'm looking at a fully remote role, I wonder whether I'll feel the downsides of that. Are you forgettable as a fully remote? Do you miss office time? It's it hard to have influence?

What's your experience?

I have been thinking of studying for the CDMP exam- I am a data analyst but want to transition data governance and in the future to AI governance. Would the CDMP or some other certifications help?

I’m currently a DE for a major retailer (think Kroger, Walmart, Albertsons, etc.), working on the pricing and product data streaming side. The WLB is amazing, and I’m earning around market rate for a junior DE in a major city. However, I’m starting to gain interest in high-frequency data streaming for large brokerages and financial firms.

For those who have made a similar move, how did you manage to switch industries, and which industry do you believe is the most lucrative for a data engineer?

I’ve been in tech for over 20 years and I have never in my life used python, or any other programming language, to manipulate data.

I’ve manipulated data in every single way you can imagine, including parsing out semi structured files, such as json and xml using SQL alone.

python should be used to handle FILES themselves or make API calls. things of that nature.

why is this field such a horrendous mess? not to mention, Python is a horribly performant language. if you care about performance you should be using C#

I'm a data engineer at an e-commerce org, and I'm fairly happy - we have interesting problems, large volumes of data, and plenty of initiatives on our backlog. I largely get to work on what I want, at my own pace, and choose how things are implemented.

But I am getting tired of doing everything with the end goal being to sell more things and increase profit extraction from consumers.

Who is working in interesting data related fields/careers that provide essential goods or services to the public/planet/humanity. Keen to hear any stories or opinions!

EDIT: thanks for all the interesting responses!