I am pretty honest: there are a ton of vibecoded projects around, and I made another one... great huh?I'm working as a data platform engineer in Germany. My day job is basically spinning up & running lakehouse platforms on US hyperscalers for SMB and some DAX companies.
I mean, data engineers like to work on these platforms but since some time I really have ethical concerns about it - like:
- vendor lock-in
- high costs (because of cloud costs - and in most cases you are not able to connect your own metal if you have one)
- non-EU vendors...
- other open-source solutions have high license costs for LDAP
etc. pp.That's why is started to build my own opinionated data lakehouse platform in my spare time (tbh I also started learning agentic engineering and what harnesses are..), only curated from open source projects already available.The building blocks are pretty straightforward imo:
- Iceberg tables on whatever S3 storage you already have
- Trino for SQL and Spark Connect for jobs
- Airflow for schedules, plus notebooks, Kafka streams and dashboards
- One web UI and one API over all of it
- stuff around like OIDC, secrets management etc.It's fully pre-alpha. But I want to use it as…
Discussion (0)
No comments yet. Start the conversation!