Why Data Teams Are Ditching All-in-One Platforms and Rolling Their Own Stack

A few years ago, buying a single vendor's end-to-end data platform felt like the smart, responsible choice. One contract, one support line, one throat to choke. Your VP of Engineering loved it. Your CFO loved it. Your data engineers tolerated it.

Fast forward to today, and a growing number of those same teams are quietly unplugging from the monolith. They're building their own stacks—stitching together open source ingestion tools, transformation frameworks, orchestration engines, and storage layers—and frankly, a lot of them are thriving.

This isn't a fringe movement anymore. It's worth asking why it's happening, and whether it might be the right call for your team.

The Monolith Problem Nobody Talks About in Sales Calls

All-in-one platforms sell a compelling story: everything works together, you get a unified UI, and you don't need a team of specialists to keep the lights on. For certain organizations—particularly those with smaller teams or tighter compliance requirements—that story holds up reasonably well.

But for teams that are actually pushing data at scale, the cracks show up fast.

First, there's the lock-in problem. When your ingestion, transformation, orchestration, and serving layers all live inside one vendor's ecosystem, switching costs become enormous. You're not just migrating a tool—you're migrating a worldview. That leverage doesn't go unnoticed at renewal time.

Second, monolithic platforms tend to be mediocre at everything rather than excellent at anything. The ETL connector that ships with your platform is probably fine. But it's probably not as good as Airbyte. The transformation layer is probably functional. But it's probably not as good as dbt. You're paying for integration convenience, and you're paying for it in capability.

Third—and this is the one that frustrates engineers most—you're often waiting on the vendor's roadmap. Need a feature that isn't there yet? File a ticket and hope. With open source components, you can contribute the fix yourself or find a community member who already has.

What a Modern Composable Stack Actually Looks Like

The modular approach, often called the "modern data stack" or more recently the "composable data stack," isn't about reinventing everything from scratch. It's about choosing the best tool for each layer and connecting them thoughtfully.

A typical setup might look something like this:

• Ingestion: Airbyte (open source) for pulling data from SaaS tools and databases
• Storage: A cloud data warehouse like BigQuery or open source alternatives like Apache Iceberg on S3
• Transformation: dbt (open source core) for SQL-based modeling
• Orchestration: Apache Airflow or Dagster for scheduling and pipeline management
• Serving/BI: Apache Superset for dashboards, or Metabase for something lighter

None of these tools were built by the same company. All of them are either fully open source or have robust open source cores. And together, they cover the full data lifecycle without requiring you to sign a seven-figure enterprise agreement.

The Teams Making the Switch

This isn't purely theoretical. Smaller tech companies and scale-ups have been leading this shift, often because they have the engineering talent to manage it and the budget sensitivity to care about the cost difference.

One pattern that comes up repeatedly: teams that started on a managed platform outgrow it. Their data volumes scale, their use cases diversify, and suddenly the platform that worked fine at Series A is a bottleneck at Series C. Rebuilding on composable open source tools becomes not just appealing but necessary.

Larger organizations are moving more cautiously—and reasonably so. Governance, compliance, and support SLAs matter more when you're a public company or operating in regulated industries. But even there, hybrid approaches are emerging: keep sensitive workloads on managed platforms, run experimental or high-volume pipelines on open source infrastructure.

The Real Challenges (Let's Be Honest)

Anybody selling you on the composable stack without acknowledging the downsides is missing the point.

Operational overhead is real. When you own the stack, you own the outages. Managed platforms handle upgrades, security patches, and infrastructure scaling for you. Open source means someone on your team is doing that work. For smaller teams without dedicated platform engineers, this can be genuinely painful.

Integration glue is your responsibility. These tools weren't designed together. Making them talk to each other cleanly requires engineering effort. Schema mismatches, authentication quirks, and version incompatibilities are your problem to solve.

Support is community-based. Stack Overflow, GitHub issues, and Slack communities are excellent—until you have a production incident at 2 a.m. and need an answer in 20 minutes. Vendor support, for all its flaws, is at least a phone call away.

The composable stack rewards teams with strong engineering culture and penalizes teams that are already stretched thin. That's not a knock on the approach—it's just context you need before making the decision.

A Simple Framework for Deciding What's Right for You

Here's how I'd think about this decision:

Go composable if: You have dedicated data engineers (not just analysts), your data volumes or use cases are outgrowing your current platform, you're sensitive to vendor lock-in on strategic infrastructure, or you have specific capability needs that your current platform doesn't cover well.

Stick with managed platforms if: Your team is small and engineering bandwidth is limited, your use cases are relatively standard, compliance and support SLAs are non-negotiable, or you're early enough that moving fast matters more than optimizing costs.

Consider hybrid if: You have some workloads that fit a managed platform well and others that don't. This is more operationally complex, but it's often the most pragmatic path for mid-sized organizations.

The Broader Shift

What's happening here isn't just about tools. It reflects a broader maturation of the data engineering discipline. Teams that once needed vendors to abstract away complexity are now capable of managing that complexity themselves—and they're recognizing that the abstraction comes at a cost.

The open source ecosystem around data has never been stronger. The tooling is more mature, the communities are larger, and the documentation (ironically) is often better than what you get from enterprise vendors. The composable stack isn't a workaround anymore—for many teams, it's the first choice.

Vendors aren't going away. But the era of buying a platform and hoping it does everything you need is quietly ending. The teams that learn to build their own stacks—deliberately, with clear tradeoffs in mind—are going to have a structural advantage over those that don't.

That's not a prediction. It's already happening.