2024-08-03, 13:55–14:25 (Asia/Taipei), TR514
Recently, there has been a growing trend of running database products on Kubernetes. In this article, the author shares their experience of deploying and managing MySQL on Kubernetes, highlighting some of the challenges they faced as a beginner, including the pitfalls they encountered and how they overcame them. This presentation will describe the technical challenges we have encountered with MySQL, MySQL Operator, and MySQL on Kubernetes, and the approaches we have taken to address them. There are various possibilities for individuals who are not specialized in DB engineering to collaborate with MySQL. This presentation is intended to serve as a reference for those who are beginner to databases.
This presentation begins with an introduction to Kubernetes, the concept of Operators, and specifically, the MySQL Operator for Kubernetes. It then explores three primary challenges encountered during the deployment and management of MySQL on Kubernetes
The first issue encountered was the unexpected termination of MySQL connections. Through numerous connection attempts from varied locations, it was discerned that the Kubernetes Service could be causing this problem. Further examination of Kubernetes Services' workings revealed IPVS's role in the issue. Adjusting the MySQL timeout settings to match the IPVS timeout settings resolved the disconnection problem by aligning their timeout periods.
The second challenge was modifing MySQL settings, hindered by a lack of detailed guidance in the official documentation for MySQL instances deployed with the MySQL Operator. Attempts to modify configurations via Helm's values file and performing a helm upgrade were unsuccessful. However, manually editing the ConfigMap directly led to the successful update of MySQL settings.
The third challenge revolved around a gradual increase in MySQL's memory usage without an identifiable cause. Troubleshooting suggested the issue might be related to the memory allocator rather than a memory leak within MySQL itself. Enhancing network bandwidth and storage I/O speeds helped mitigate this issue. As a provisional measure, monitoring memory usage and restarting the MySQL server upon noticing high usage were effective in preventing Out-Of-Memory (OOM) occurrences.
Finally, I have some advice for those who are going to try MySQL on K8s. For those considering deploying MySQL on Kubernetes, using the MySQL Operator is advisable for high-availability setups. However, it's important to note that while the Operator simplifies the initial setup, it might not cover detailed recovery mechanisms for all issues. Troubleshooting complex configurations like the InnoDB Cluster can be challenging. For a simpler MySQL setup, deploying directly from a container image without using an Operator might be more straightforward and easier to manage.
Infrastructure Engineer. Enjoying building Kubernetes and deploying applications on it since last year. At COSCUP2023, she presented "Digital twin workbench with Jupyter hub/lab and Hadoop/Spark/Kafka for geospatial-temporal applications.