← Back to list

high disk watermark 错误

Published on: | Views: 147

问题

遇到写es写不进去的错误,看一下日志:

{"@timestamp":"2022-03-10T06:44:00.075Z", "log.level": "WARN", 
"message":"high disk watermark [90%] exceeded on 
[oJkmx0opT_6ykkDNDPF-eQ][mo][/usr/share/elasticsearch/data] 
free: 21.5gb[9.4%], shards will be relocated away from this node; 
currently relocating away shards totalling [0] bytes; 
the node is expected to continue to exceed the high disk watermark when these relocations are complete", 
"ecs.version":"1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server",
"process.thread.name":"elasticsearch[mo][generic][T#3]","log.logger":"org.elasticsearch.cluster.routing.allocation.DiskThresholdMonitor",
"elasticsearch.cluster.uuid":"zyAlW3zbT-6qRk5b3QTi5Q",
"elasticsearch.node.id":"oJkmx0opT_6ykkDNDPF-eQ",
"elasticsearch.node.name":"mo","elasticsearch.cluster.name":"docker-cluster"}

这个意思是说磁盘已经达到90%水位线了,不能再写入数据了(虽然还有21.5G的空间),emm这个设计其实不太合理。

配置参数

不过官方文档也提供了配置参数可以修改: https://www.elastic.co/guide/en/elasticsearch/reference/8.6/modules-cluster.html#cluster-routing-flood-stage

cluster.routing.allocation.disk.threshold_enabled

默认为true,设置为false会禁用磁盘分配决策器;

cluster.routing.allocation.disk.watermark.low

默认为85%,意思是es不会将分片分给磁盘使用超过85%的节点(主分片不受影响);

cluster.routing.allocation.disk.watermark.high

默认为90%,超过这个值时,es会尝试把分片迁出此节点。

cluster.routing.allocation.disk.watermark.flood_stage

默认为95%,达到这个值时,节点上的索引会变成只读和可删除。

总结

es使用了一些参数来控制磁盘的分配; 这些参数可以使用百分比,也可以直接使用具体的大小,比如1GB,但不能同时使用比例和大小; 实验环境只需要配置cluster.routing.allocation.disk.threshold_enabled=false就可以解决问题;