博客
关于我
强烈建议你试试无所不能的chatGPT,快点击我
记一次ES5 EsAbortPolicy.rejectedExecution 简单处理
阅读量:6683 次
发布时间:2019-06-25

本文共 7445 字,大约阅读时间需要 24 分钟。

  hot3.png

[2017-01-02T19:24:03,553][DEBUG][o.e.a.s.TransportSearchAction] [sa-11.bs.ppcredit.com] [logstash-open_log-2016.12.22][4], node[W_58WjYqTLqaNyQEoRwuzA], [R], s[STARTED], a[id=lAbIQmgtQO2SMyG_bJ7EqQ]: Failed to execute [SearchRequest{searchType=QUERY_THEN_FETCH, indices=[.monitoring-data-2, ..., logstash-tomcat--2016.12.21], indicesOptions=IndicesOptions[id=38, ignore_unavailable=false, allow_no_indices=true, expand_wildcards_open=true, expand_wildcards_closed=false, allow_alisases_to_multiple_indices=true, forbid_closed_indices=true], types=[], routing='null', preference='null', requestCache=null, scroll=null, source={  "size" : 0,  "query" : {    "bool" : {      "must" : [        {          "match" : {            "type" : {              "query" : "api_log",              "operator" : "OR",              "prefix_length" : 0,              "max_expansions" : 50,              "fuzzy_transpositions" : true,              "lenient" : false,              "zero_terms_query" : "NONE",              "boost" : 1.0            }          }        },        {          "match" : {            "message" : {              "query" : "mj.*.etinfo",              "type" : "phrase",              "operator" : "OR",              "prefix_length" : 0,              "max_expansions" : 50,              "fuzzy_transpositions" : true,              "lenient" : false,              "zero_terms_query" : "NONE",              "boost" : 1.0            }          }        },        {          "match" : {            "message" : {              "query" : "调用失败",              "type" : "phrase",              "operator" : "OR",              "prefix_length" : 0,              "max_expansions" : 50,              "fuzzy_transpositions" : true,              "lenient" : false,              "zero_terms_query" : "NONE",              "boost" : 1.0            }          }        }      ],      "filter" : [        {          "range" : {            "@timestamp" : {              "from" : "now-1m",              "to" : "now",              "include_lower" : true,              "include_upper" : true,              "boost" : 1.0            }          }        }      ],      "disable_coord" : false,      "adjust_pure_negative" : true,      "boost" : 1.0    }  },  "ext" : { }}}] lastShard [true]  org.elasticsearch.transport.RemoteTransportException: [elk122.bs.ppcredit.com][172.19.99.122:29300][indices:data/read/search[phase/query]]Caused by: org.elasticsearch.common.util.concurrent.EsRejectedExecutionException: rejected execution of org.elasticsearch.transport.TcpTransport$RequestHandler@3f26bea7 on EsThreadPoolExecutor[search, queue capacity = 1000, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@2a7c7fa6[Running, pool size = 10, active threads = 10, queued tasks = 1000, completed tasks = 4352940]]	at org.elasticsearch.common.util.concurrent.EsAbortPolicy.rejectedExecution(EsAbortPolicy.java:50) ~[elasticsearch-5.1.1.jar:5.1.1]	at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:823) ~[?:1.8.0_74]	at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1369) ~[?:1.8.0_74]	at org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor.doExecute(EsThreadPoolExecutor.java:94) ~[elasticsearch-5.1.1.jar:5.1.1]	at org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor.execute(EsThreadPoolExecutor.java:89) ~[elasticsearch-5.1.1.jar:5.1.1]	at org.elasticsearch.transport.TcpTransport.handleRequest(TcpTransport.java:1343) [elasticsearch-5.1.1.jar:5.1.1]	at org.elasticsearch.transport.TcpTransport.messageReceived(TcpTransport.java:1242) [elasticsearch-5.1.1.jar:5.1.1]	at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:74) [transport-netty4-5.1.1.jar:5.1.1]	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:373) [netty-transport-4.1.6.Final.jar:4.1.6.Final]	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359) [netty-transport-4.1.6.Final.jar:4.1.6.Final]	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:351) [netty-transport-4.1.6.Final.jar:4.1.6.Final]	at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:293) [netty-codec-4.1.6.Final.jar:4.1.6.Final]	at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:280) [netty-codec-4.1.6.Final.jar:4.1.6.Final]	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:396) [netty-codec-4.1.6.Final.jar:4.1.6.Final]	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:248) [netty-codec-4.1.6.Final.jar:4.1.6.Final]	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:373) ~[netty-transport-4.1.6.Final.jar:4.1.6.Final]	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359) ~[netty-transport-4.1.6.Final.jar:4.1.6.Final]	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:351) ~[netty-transport-4.1.6.Final.jar:4.1.6.Final]	at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86) ~[netty-transport-4.1.6.Final.jar:4.1.6.Final]	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:373) [netty-transport-4.1.6.Final.jar:4.1.6.Final]	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359) [netty-transport-4.1.6.Final.jar:4.1.6.Final]	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:351) [netty-transport-4.1.6.Final.jar:4.1.6.Final]	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334) [netty-transport-4.1.6.Final.jar:4.1.6.Final]	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:373) [netty-transport-4.1.6.Final.jar:4.1.6.Final]	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359) [netty-transport-4.1.6.Final.jar:4.1.6.Final]	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926) [netty-transport-4.1.6.Final.jar:4.1.6.Final]	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:129) [netty-transport-4.1.6.Final.jar:4.1.6.Final]	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:651) [netty-transport-4.1.6.Final.jar:4.1.6.Final]	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:536) [netty-transport-4.1.6.Final.jar:4.1.6.Final]	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:490) [netty-transport-4.1.6.Final.jar:4.1.6.Final]	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:450) [netty-transport-4.1.6.Final.jar:4.1.6.Final]	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:873) [netty-common-4.1.6.Final.jar:4.1.6.Final]	at java.lang.Thread.run(Thread.java:745) [?:1.8.0_74]

有一天发现es的每天的log高达12G,排查下来发现,有台机器crontab中每次同时运行了16个api查询,导致ES节点reject excute。

把16个API查询的python执行命令写一个shell里,每隔5个执行sleep2秒,随后再写到crontab里,就不报错了。 ^_^

转载于:https://my.oschina.net/fy666/blog/817440

你可能感兴趣的文章
[学习笔记]矩阵求逆
查看>>
phpMyAdmin 登陆需要密码
查看>>
zookeeper实现队列_Queue
查看>>
转 delete 和 delete []的真正区别
查看>>
outline
查看>>
爬取json Swaggerui界面
查看>>
Windows 8 应用开发 - 异步调用
查看>>
javaScript引入方式
查看>>
[摘录]验证视图MAC失败 Validation of ViewState MAC Failed
查看>>
Cocos2D-X屏幕适配新解
查看>>
asp.net mvc生命周期学习
查看>>
C++ explicit关键字避免隐式转换
查看>>
JS判断IE,FF,Opera,Safari等浏览器类型
查看>>
C++读取文件,将文件内容读取到struct中
查看>>
构建之法阅读笔记02
查看>>
HDU 1212 Big Number 【大数求余模拟】
查看>>
静态类(static)与java值传递、引用传递小测
查看>>
小怪受击身体变红特效代码
查看>>
怎样删除数据库表中所有的数据
查看>>
问题 G: 【一本通提高同余问题】计算器
查看>>