When running stateful services like MySQL on a distributed resource management platform called Mesos, you will face the core challenges of data persistence and resource binding, and this situation is directly related to the reliability of the service and the flexibility of the cluster.
Core requirements for data persistence
For a database like MySQL, the data must be stored securely and be accessible again after the service is restarted or migrated. On traditional physical machines or virtual machines, data is generally stored on local disks. However, in a Mesos cluster, tasks may be scheduled to run on any available slave node, which creates the problem of separating data and computing. If the task is terminated and restarted on another node, the data will be lost.
Original solution for binding specific nodes
There is a direct solution, which is to bind the database service to a specific slave node, or to several specific slave nodes. For example, you can label these nodes with the role of "MySQL" and configure the task scheduling policy to start only on these nodes. At the same time, mount the local directory on the node, such as /data , to the data directory of the task container. In this way, the data is persisted on the local disk of the specific node.
Significant drawbacks of local binding scheme
Although the above solution achieves persistence, it has caused serious problems. It damages the dynamic scheduling advantage of the Mesos cluster. The bound node resources cannot be used by other services, leaving the resources idle. In addition, once a hardware failure occurs on the node, the service will be interrupted and there is a risk of data loss. This is in fact a regression to the static allocation of resources.
Introducing a distributed file system
To solve the problem of data portability, a common approach is to configure a distributed file system, or DFS, for the entire cluster, such as HDFS, Ceph or NFS. The tasks performed by the database will write data to the mounted DFS volume. In this way, no matter which slave node a task is scheduled to, as long as it can access DFS, it can read and write previous data, achieving decoupling of data and computing nodes.
New tradeoffs brought by DFS
Adopting a distributed file system is not a flawless solution. It will bring additional network IO overhead, which may have an impact on the read and write performance of the database, especially for latency-sensitive businesses. At the same time, DFS itself also needs to be deployed and maintained, which increases the architectural complexity and operation and maintenance costs of the cluster. This is equivalent to trading performance and complexity for flexibility and reliability.
Modern solutions for persistent volumes
Contemporary container orchestration platforms and updated Mesos versions provide a more elegant solution: Persistent Volume. Mesos can abstract the local disk space in the slave node into a "volume" and claim that its life cycle is decoupled from the task itself. When a database task is scheduled, it can request to mount a specific persistent volume and data is written to it. Even after the task ends, the volume and data are still retained.
The working mechanism and resource bundling of persistent volumes
The next task of the same type that can apply to mount the same persistent volume is to read historical data. To achieve this situation, the scheduler must allocate the persistent volume and computing resources (CPU, memory) as a whole resource supply (offer). This ensures that the task of obtaining a volume must obtain the right to run on the corresponding node, eliminating the problem of volumes and computing resources being scheduled separately.
Thinking about future architecture
{“id” : { “value” : “offerid-123456789”},
“framework_id” : “MYID”,
“slave_id” : “slaveid-123456789”,
“hostname” : “hostname1.prod.twttr.net”
“resources” : [
// Regular resources.
{ “name” : “cpu”, “type” : SCALAR, “scalar” : 10 }
{ “name” : “mem”, “type” : SCALAR, “scalar” : 12GB }
{ “name” : “disk”, “type” : SCALAR, “scalar” : 90GB }
// Persistent resources.
{ “name” : “cpu”, “type” : SCALAR, “scalar” : 2,
“persistence” : { “framework_id” : “MYID”, “handle” : “uuid-123456789” } }
{ “name” : “mem”, “type” : SCALAR, “scalar” : 2GB,
“persistence” : { “framework_id” : “MYID”, “handle” : “uuid-123456789” } }
{ “name” : “disk”, “type” : SCALAR, “scalar” : 10GB,
“persistence” : { “framework_id” : “MYID”, “handle” : “uuid-123456789” } }
]
...
}To bring stateful services to the cloud-native platform, persistent volumes have become the standard answer. It achieves a balance in performance, flexibility and data reliability. When making selections, teams must consider their own data volume, performance requirements and operation and maintenance capabilities for DFS. In the future, as advanced functions such as storage snapshots and cloning are integrated, the migration and backup of stateful services will become more convenient.
During your actual work, are you more inclined to use a distributed file system or a persistent volume solution to manage database status? Feel free to share your experiences and opinions in the comment area. If you find this article helpful, please give it a like and support.
