Clustering provider in Orleans

(Jin Qing’s Column, Nov. 3, 2021)

When deployed to a cluster of nodes,
Orleans internally implements a protocol to manage it’s silos,
including discovery, failure and reconfigure,
which is called cluster membership management.

Orleans has clustering membership providers for: Azure, SQL server, Zookeeper.

Clustering provider is one of key aspects of silo configuration.

Orleans.Clustering.Kubernetes

OrleansContrib/Orleans.Clustering.Kubernetes
is a clustering provider for running Orleans cluster on Kubernetes.

Tell silo to use Kubernetes as the Cluster Membership Provider:

var silo = new SiloBuilder()....UseKubeMembership()....Build();

Interface

UseKubeMembership() instantiates a KubeMembershipTable which implements
IMembershipTable.

    public interface IMembershipTable{Task InitializeMembershipTable(bool tryInitTableVersion);Task DeleteMembershipTableEntries(string clusterId);Task CleanupDefunctSiloEntries(DateTimeOffset beforeDate);Task<MembershipTableData> ReadRow(SiloAddress key);Task<MembershipTableData> ReadAll();Task<bool> InsertRow(MembershipEntry entry, TableVersion tableVersion);Task<bool> UpdateRow(MembershipEntry entry, string etag, TableVersion tableVersion);Task UpdateIAmAlive(MembershipEntry entry);}

Implement

KubeMembershipTable
access Kubernetes API server to read and write silo entry CRD.

InitializeMembershipTable()
- TryInitClusterVersion()
  - _kubeClient.GetNamespacedCustomObjectAsync
  - _kubeClient.CreateNamespacedCustomObjectAsync
DeleteMembershipTableEntries(string clusterId)
- _kubeClient.DeleteNamespacedCustomObjectAsync
InsertRow(…)
- _kubeClient.GetNamespacedCustomObjectAsync
- _kubeClient.ReplaceNamespacedCustomObjectAsync
- _kubeClient.CreateNamespacedCustomObjectAsync
ReadAll()
- GetClusterVersion()
  - _kubeClient.ListNamespacedCustomObjectAsync
- GetSilos()
  - _kubeClient.ListNamespacedCustomObjectAsync
ReadRow(SiloAddress key)
- _kubeClient.GetNamespacedCustomObjectAsync
UpdateIAmAlive(MembershipEntry entry)
- _kubeClient.GetNamespacedCustomObjectAsync
- _kubeClient.ReplaceNamespacedCustomObjectAsync
UpdateRow(…)
- _kubeClient.ReplaceNamespacedCustomObjectAsync
CleanupDefunctSiloEntries(DateTimeOffset beforeDate)
- _kubeClient.DeleteNamespacedCustomObjectAsync

The operators to NamespacedCustomObject are:

Delete
Get
Create
Replace
List

CRD

Two CRDs, ClusterVersion and Silo are defined in files:

ClusterVersionCRD.yaml
SiloEntryCRD.yaml

Custom resource objects are stored in etcd of Kubernetes.