Working with data should be fun! In 2021, after having worked as a data engineer for 5 years, I didn't feel like building yet another data stack. I concluded that I would be better off joining a modern data tool company to work on product and marketing. Instead of building, buying, and integrating a custom data stack that a single company would benefit from, I could contribute to building and promoting a tool used by thousands.
This is how I started an inspiring career as a DevRel and marketing consultant for Airbyte, Datafold, and a dozen other modern data tool companies. However, despite all the product features and content we shipped, I still don't want to get back to being a data engineer, analyst, or scientist. Why?
The Data Market Today
The Modern Data Stack (MDS) provides a developer experience that is scattered across various tools. The data community agrees that consolidation is necessary and already happening through acquisitions and feature expansions. Don't get me wrong, the MDS wave was a good thing. It was the result of ambitious data practitioners who wanted to question the status quo and provide data teams with best-in-class products for each product category. Each of these companies had to start somewhere and obviously could not implement all features of a data platform at once. The main issue I see is that the MDS product categories that emerged are somehow "artificial". I believe we now have enough context to create a more "natural" data landscape.
Good old problems are still present. Most data teams are solving the same problems. Most data tools are building the same features. Every data tool implements some sort of data table view, but they all look and feel different. Is that a good thing? Is data lineage a feature or a product category? If it's a feature, why should you pay multiple times for data lineage when buying a data warehouse, a data quality tool, an IDE, a data transformation framework, and a data orchestrator? Each of these tools spent money building data lineage, and they are charging you. If it’s a product category you buy separately, how do you ensure that it integrates and scales with the rest of your stack? Is buying a data platform the only outcome?
The questions are endless, and so is the content span (tool websites and docs, Gartner reports, influencer blogs, Reddit threads...), making the experience of building or buying a data stack that your data team will love either dreadful or an arbitrary choice.
I believe the data engineer, analyst, and scientist developer experience is about to change, again. If LLMs and AI tools force data people to become 10x more productive to stay relevant, there is no way we, humans, can survive with the current data developer experience. We must learn how to leverage AI to build the next generation of data tools harder, better, faster, stronger, smarter.
Tools for Data Tomorrow
I have decided to put my data engineer career on hold until we have the next generation of data tools that make working with data lovable, efficient, reliable, and evolutive. Migrating your data stack every year is not fun; we need some solid ground as a combination of tools (starting with the developer experience) and open standards.
I started Tools for Data to further understand the current data tools landscape, to contribute to a new product and content quality standard. Today, you will find curated lists of existing data tools by category and guides to navigate each product category. I strive to create the most valuable content to help you build and buy data tools you will fall in love with today and tomorrow.
Exciting times!