dapr的consistent hash

序

本文主要研究一下dapr的consistent hash

consistent_hash

dapr/pkg/placement/hashing/consistent_hash.go

var replicationFactor int// ErrNoHosts is an error for no hosts
var ErrNoHosts = errors.New("no hosts added")// ConsistentHashTables is a table holding a map of consistent hashes with a given version
type ConsistentHashTables struct {Version stringEntries map[string]*Consistent
}// Host represents a host of stateful entities with a given name, id, port and load
type Host struct {Name  stringPort  int64Load  int64AppID string
}// Consistent represents a data structure for consistent hashing
type Consistent struct {hosts     map[uint64]stringsortedSet []uint64loadMap   map[string]*HosttotalLoad int64sync.RWMutex
}

ConsistentHashTables定义了Version、Entries属性，Entries是个map，value为Consistent；Consistent定义了hosts、sortedSet、loadMap、totalLoad、sync.RWMutex属性

GetInternals

dapr/pkg/placement/hashing/consistent_hash.go

// GetInternals returns the internal data structure of the consistent hash
func (c *Consistent) GetInternals() (map[uint64]string, []uint64, map[string]*Host, int64) {c.RLock()defer c.RUnlock()return c.hosts, c.sortedSet, c.loadMap, c.totalLoad
}

GetInternals方法返回hosts、sortedSet、loadMap、totalLoad属性

Add

dapr/pkg/placement/hashing/consistent_hash.go

// Add adds a host with port to the table
func (c *Consistent) Add(host, id string, port int64) bool {c.Lock()defer c.Unlock()if _, ok := c.loadMap[host]; ok {return true}c.loadMap[host] = &Host{Name: host, AppID: id, Load: 0, Port: port}for i := 0; i < replicationFactor; i++ {h := c.hash(fmt.Sprintf("%s%d", host, i))c.hosts[h] = hostc.sortedSet = append(c.sortedSet, h)}// sort hashes ascendinglysort.Slice(c.sortedSet, func(i int, j int) bool {return c.sortedSet[i] < c.sortedSet[j]})return false
}

Add方法创建Host并添加到loadMap中，之后根据replicationFactor次数对host进行hash并添加到hosts及sortedSet中，最后对sortedSet进行排序

Get

dapr/pkg/placement/hashing/consistent_hash.go

// Get returns the host that owns `key`.
//
// As described in https://en.wikipedia.org/wiki/Consistent_hashing
//
// It returns ErrNoHosts if the ring has no hosts in it.
func (c *Consistent) Get(key string) (string, error) {c.RLock()defer c.RUnlock()if len(c.hosts) == 0 {return "", ErrNoHosts}h := c.hash(key)idx := c.search(h)return c.hosts[c.sortedSet[idx]], nil
}

Get方法先对key进行hash，然后通过search查找idx，最后找到idx在sortedSet中对应的host，最后从hosts中返回对应host

GetHost

dapr/pkg/placement/hashing/consistent_hash.go

// GetHost gets a host
func (c *Consistent) GetHost(key string) (*Host, error) {h, err := c.Get(key)if err != nil {return nil, err}return c.loadMap[h], nil
}

GetHost方法先get key，之后再去loadMap获取host

GetLeast

dapr/pkg/placement/hashing/consistent_hash.go

// GetLeast uses Consistent Hashing With Bounded loads
//
// https://research.googleblog.com/2017/04/consistent-hashing-with-bounded-loads.html
//
// to pick the least loaded host that can serve the key
//
// It returns ErrNoHosts if the ring has no hosts in it.
//
func (c *Consistent) GetLeast(key string) (string, error) {c.RLock()defer c.RUnlock()if len(c.hosts) == 0 {return "", ErrNoHosts}h := c.hash(key)idx := c.search(h)i := idxfor {host := c.hosts[c.sortedSet[i]]if c.loadOK(host) {return host, nil}i++if i >= len(c.hosts) {i = 0}}
}

GetLeast方法先对key进行hash，然后通过search获取idx，之后通过loadOK来获取least loaded host

search

dapr/pkg/placement/hashing/consistent_hash.go

func (c *Consistent) search(key uint64) int {idx := sort.Search(len(c.sortedSet), func(i int) bool {return c.sortedSet[i] >= key})if idx >= len(c.sortedSet) {idx = 0}return idx
}

search方法通过sort.Search根据key获取idx

小结

dapr的Consistent定义了hosts、sortedSet、loadMap、totalLoad、sync.RWMutex属性；它定义了GetInternals、Add、Get、GetHost、GetLeast、search等方法。

doc

dapr

dapr的consistent hash相关推荐

关于consistent hash的思考及改进方案
这里默认读者已经知道了一致性hash算法的原理. 1. 为什么在某台机器宕机之后consistent hash算法能够避免所有或者大部分key重新hash? 首先需要弄清的是,如果某一台机器宕机之后, ...
聊聊jump consistent hash
序本文主要简介一下jump Consistent hash. jump consistent hash jump consistent hash是一致性哈希的一种实现,论文见A Fast, Mini ...
主从mysql replication 集群的sharding memcache集群使用consistent hash
sharding 实现跨越DB的分区与扩展功能 consistent hash 一致哈希实现memcache的扩展 http://cache.baidu.com/c?m=9f65cb4a8c8507e ...
haproxy Consistent Hash浅析
http://blog.sina.com.cn/s/blog_502c8cc40100kfz2.html Haproxy实现了Map-based 和consistent hash算法,来完成通过哈希值 ...
Google Jump Consistent Hash 一致性哈希算法
接触到这个一致性哈希算法是在腾讯音乐的讲座中,用于在线扩容如图中的例子,本来只有group0和group1,现在要增加一个group2用于推送新的数据,如果使用不满足单调性要求的hash方法,首先向 ...
Upstream Consistent Hash
介绍 https://www.nginx.com/resources/wiki/modules/consistent_hash/地址 ngx_http_upstream_consistent_hash ...
2016 -Nginx的负载均衡 - 一致性哈希 (Consistent Hash)
Nginx版本:1.9.1 算法介绍当后端是缓存服务器时,经常使用一致性哈希算法来进行负载均衡. 使用一致性哈希的好处在于,增减集群的缓存服务器时,只有少量的缓存会失效,回源量较小. 在nginx+ ...
consistent hash
https://www.youtube.com/watch?v=ffE1mQWxyKM https://www.youtube.com/watch?v=zaRkONvyGr8
一文搞懂一致性hash的原理和实现
在 go-zero 的分布式缓存系统分享里,Kevin 重点讲到过一致性hash的原理和分布式缓存中的实践.本文来详细讲讲一致性hash的原理和在 go-zero 中的实现. 以存储为例,在整个微服务 ...

dapr的consistent hash

序

consistent_hash

GetInternals

Add

Get

GetHost

GetLeast

search

小结

doc

dapr的consistent hash相关推荐

最新文章

热门文章