Bikalpa Dhakal
Bikalpa Dhakal is a software engineer and open-source enthusiast based in Pokhara, Nepal. He currently works in the Analytics and Processing team at Canonical, where he helps maintain the Charmed Apache Spark project and its repositories. Before joining Canonical, he worked for companies such as Logpoint. More recently at Canonical, he was involved in the technical integration between Charmed Apache Spark and Charmed Kubeflow. Outside of his professional work, Bikalpa's hobbies include cinema, traveling and aviation.
Intervention
In modern data science, the "Data Lakehouse" is a powerful way to manage large-scale data. However, for many teams, the "last mile" of integration—connecting Machine Learning platforms like Kubeflow to big data processing engines like Apache Spark—remains a major technical challenge. In a shared Kubernetes cluster, managing security, user permissions, and connection strings across different user profiles often leads to manual errors and security risks.
In this session, we will explore an open-source solution to this problem using Juju and Charmed Operators. We will walk through a "Multi-Integrator" architecture where each Kubeflow profile receives its own secure connection to the data platform. Instead of manual configuration, we use automated integrators to inject Spark Service Accounts, configurations, and security secrets directly into user namespaces.
By using Charmed Spark and Charmed Kubeflow, we can create a modular system where data scientists can move from a simple notebook to a distributed Spark job without worrying about the underlying YAML files or credentials. We will discuss the logic of this design, how it handles multi-tenancy, and how it simplifies the path to a production-ready data lakehouse on Ubuntu.
More info: https://events.canonical.com/event/146/contributions/951/