Spring Cloud Config Server遷移節(jié)點(diǎn)或容器化帶來(lái)的問(wèn)題
如果您跟我一樣,目前正在使用Spring Cloud Config做為配置中心的話,本篇將來(lái)要描述的問(wèn)題,強(qiáng)烈推薦了解和關(guān)注!因?yàn)檫@個(gè)問(wèn)題目前存在于所有的版本中,還沒(méi)有完全修復(fù)。
問(wèn)題現(xiàn)象
為了說(shuō)明下面的內(nèi)容,我們可以先嘗試重現(xiàn)一下問(wèn)題:在一個(gè)測(cè)試環(huán)境中,將Spring Cloud Config的配置中心遷移到另外一個(gè)節(jié)點(diǎn)上,即配置中心的IP地址發(fā)生了變化。在完成遷移之后,我們會(huì)發(fā)現(xiàn)該環(huán)境下各個(gè)微服務(wù)應(yīng)用的健康狀態(tài)會(huì)變得時(shí)好時(shí)壞,并且在日志中會(huì)出現(xiàn)類似下面的報(bào)錯(cuò):
- 2018-05-13 17:01:28,569 WARN [http-nio-9920-exec-1] org.springframework.cloud.config.client.ConfigServerHealthIndicator - Health check failed
- java.lang.IllegalStateException: Could not locate PropertySource and the fail fast property is set, failing
- at org.springframework.cloud.config.client.ConfigServicePropertySourceLocator.locate(ConfigServicePropertySourceLocator.java:132)
- at org.springframework.cloud.config.client.ConfigServicePropertySourceLocator$$FastClassBySpringCGLIB$$fa44b2a.invoke(<generated>)
- at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
- at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:738)
- at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
- at org.springframework.retry.interceptor.RetryOperationsInterceptor$1.doWithRetry(RetryOperationsInterceptor.java:91)
- at org.springframework.retry.support.RetryTemplate.doExecute(RetryTemplate.java:287)
- at org.springframework.retry.support.RetryTemplate.execute(RetryTemplate.java:164)
- at org.springframework.retry.interceptor.RetryOperationsInterceptor.invoke(RetryOperationsInterceptor.java:118)
- at org.springframework.retry.annotation.AnnotationAwareRetryOperationsInterceptor.invoke(AnnotationAwareRetryOperationsInterceptor.java:153)
- at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
- at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:673)
- at org.springframework.cloud.config.client.ConfigServicePropertySourceLocator$$EnhancerBySpringCGLIB$$3a43a1f4.locate(<generated>)
- at org.springframework.cloud.config.client.ConfigServerHealthIndicator.getPropertySource(ConfigServerHealthIndicator.java:54)
- at org.springframework.cloud.config.client.ConfigServerHealthIndicator.doHealthCheck(ConfigServerHealthIndicator.java:35)
- at org.springframework.boot.actuate.health.AbstractHealthIndicator.health(AbstractHealthIndicator.java:43)
- at org.springframework.boot.actuate.health.CompositeHealthIndicator.health(CompositeHealthIndicator.java:68)
- at org.springframework.boot.actuate.endpoint.HealthEndpoint.invoke(HealthEndpoint.java:85)
- at org.springframework.boot.actuate.endpoint.mvc.HealthMvcEndpoint.getCurrentHealth(HealthMvcEndpoint.java:177)
- at org.springframework.boot.actuate.endpoint.mvc.HealthMvcEndpoint.getHealth(HealthMvcEndpoint.java:166)
- at org.springframework.boot.actuate.endpoint.mvc.HealthMvcEndpoint.invoke(HealthMvcEndpoint.java:143)
- at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
- at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
- at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
- at java.lang.reflect.Method.invoke(Method.java:498)
- at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205)
- at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133)
- at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:97)
- at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827)
- at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738)
- at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)
- at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:967)
- at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:901)
- at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970)
- at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:861)
- at javax.servlet.http.HttpServlet.service(HttpServlet.java:687)
- at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846)
- at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
- at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
- at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
- at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
- at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
- at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
- at org.springframework.boot.web.filter.ApplicationContextHeaderFilter.doFilterInternal(ApplicationContextHeaderFilter.java:55)
- at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
- at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
- at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
- at com.yonghui.feign.filter.RequestOriginFilter.doFilter(RequestOriginFilter.java:41)
- at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
- at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
- at org.springframework.boot.actuate.trace.WebRequestTraceFilter.doFilterInternal(WebRequestTraceFilter.java:110)
- at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
- at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
- at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
- at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99)
- at com.yonghui.rpc.feature.web.boot.RpcHolder4BootFilter.doFilterInternal(RpcHolder4BootFilter.java:29)
- at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
- at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
- at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
- at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99)
- at com.yonghui.rpc.feature.web.boot.FeatureSupport4BootFilter.doFilterInternal(FeatureSupport4BootFilter.java:24)
- at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
- at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
- at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
- at org.springframework.web.filter.HttpPutFormContentFilter.doFilterInternal(HttpPutFormContentFilter.java:108)
- at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
- at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
- at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
- at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:81)
- at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
- at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
- at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
- at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:197)
- at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
- at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
- at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
- at org.springframework.boot.actuate.autoconfigure.MetricsFilter.doFilterInternal(MetricsFilter.java:106)
- at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
- at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
- at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
- at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:199)
- at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
- at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:504)
- at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140)
- at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81)
- at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87)
- at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342)
- at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:803)
- at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
- at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:790)
- at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1459)
- at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
- at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
- at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
- at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
- at java.lang.Thread.run(Thread.java:748)
- Caused by: org.springframework.web.client.ResourceAccessException: I/O error on GET request for "http://192.168.5.103:9010/config-server/test": Connection refused (Connection refused); nested exception is java.net.ConnectException: Connection refused (Connection refused)
- at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:674)
- at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:621)
- at org.springframework.web.client.RestTemplate.exchange(RestTemplate.java:539)
- at org.springframework.cloud.config.client.ConfigServicePropertySourceLocator.getRemoteEnvironment(ConfigServicePropertySourceLocator.java:172)
- at org.springframework.cloud.config.client.ConfigServicePropertySourceLocator.locate(ConfigServicePropertySourceLocator.java:93)
- ... 95 more
- Caused by: java.net.ConnectException: Connection refused (Connection refused)
- at java.net.PlainSocketImpl.socketConnect(Native Method)
- at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
- at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
- at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
- at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
- at java.net.Socket.connect(Socket.java:589)
- at java.net.Socket.connect(Socket.java:538)
- at sun.net.NetworkClient.doConnect(NetworkClient.java:180)
- at sun.net.www.http.HttpClient.openServer(HttpClient.java:463)
- at sun.net.www.http.HttpClient.openServer(HttpClient.java:558)
- at sun.net.www.http.HttpClient.<init>(HttpClient.java:242)
- at sun.net.www.http.HttpClient.New(HttpClient.java:339)
- at sun.net.www.http.HttpClient.New(HttpClient.java:357)
- at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1220)
- at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1156)
- at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1050)
- at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:984)
- at org.springframework.http.client.SimpleBufferingClientHttpRequest.executeInternal(SimpleBufferingClientHttpRequest.java:78)
- at org.springframework.http.client.AbstractBufferingClientHttpRequest.executeInternal(AbstractBufferingClientHttpRequest.java:48)
- at org.springframework.http.client.AbstractClientHttpRequest.execute(AbstractClientHttpRequest.java:53)
- at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:660)
- ... 99 more
可以看到類似上面的健康檢查失敗錯(cuò)誤,但是并不是一直這樣,這個(gè)環(huán)境下的微服務(wù)會(huì)出現(xiàn)時(shí)好時(shí)壞的情況,那么為什么會(huì)出現(xiàn)這種現(xiàn)象呢?
原因分析
從錯(cuò)誤日志中我們可以發(fā)現(xiàn)一個(gè)非常關(guān)鍵的信息:I/O error on GET request for "http://192.168.5.103:9010/config-server/test"。
報(bào)錯(cuò)說(shuō)明了微服務(wù)檢查配置中心獲取配置的連接是否暢通的時(shí)候出現(xiàn)了連接不上的情況,但是這個(gè)鏈接信息其實(shí)并不是當(dāng)前配置中心的地址,而是我們遷移之前的配置中心的地址。
從健康檢查的實(shí)現(xiàn)源碼ConfigServerHealthIndicator中為入口去分析和調(diào)試,我們可以解答上面現(xiàn)象的兩個(gè)疑問(wèn):
- @Override
- protected void doHealthCheck(Builder builder) throws Exception {
- PropertySource<?> propertySource = getPropertySource();
- builder.up();
- if (propertySource instanceof CompositePropertySource) {
- List<String> sources = new ArrayList<>();
- for (PropertySource<?> ps : ((CompositePropertySource) propertySource).getPropertySources()) {
- sources.add(ps.getName());
- }
- builder.withDetail("propertySources", sources);
- } else if (propertySource!=null) {
- builder.withDetail("propertySources", propertySource.toString());
- } else {
- builder.unknown().withDetail("error", "no property sources located");
- }
- }
- private PropertySource<?> getPropertySource() {
- long accessTime = System.currentTimeMillis();
- if (isCacheStale(accessTime)) {
- this.lastAccess = accessTime;
- this.cached = locator.locate(this.environment);
- }
- return this.cached;
- }
為什么會(huì)健康檢查訪問(wèn)的還是老的配置中心地址?
真正導(dǎo)致健康檢查失敗的語(yǔ)句是getPropertySource中的 this.cached = locator.locate(this.environment);而這里的具體實(shí)現(xiàn)在org.springframework.cloud.config.client.ConfigServicePropertySourceLocator類中,具體實(shí)現(xiàn)如下:
- @Override
- @Retryable(interceptor = "configServerRetryInterceptor")
- public org.springframework.core.env.PropertySource<?> locate(
- org.springframework.core.env.Environment environment) {
- ConfigClientProperties properties = this.defaultProperties.override(environment);
- CompositePropertySource composite = new CompositePropertySource("configService");
- RestTemplate restTemplate = this.restTemplate == null ? getSecureRestTemplate(properties)
- : this.restTemplate;
- Exception error = null;
- String errorBody = null;
- logger.info("Fetching config from server at: " + properties.getRawUri());
- ...
- }
可以看到,真正去訪問(wèn)的地址是直接從properties.getRawUri()獲取的,它已經(jīng)是一個(gè)固化的值,而不是通過(guò)服務(wù)發(fā)現(xiàn)機(jī)制來(lái)動(dòng)態(tài)獲取的。這就導(dǎo)致了當(dāng)我們把配置中心做了遷移,或者直接部署在容器中出現(xiàn)重啟的時(shí)候,IP發(fā)生變化,而所有的微服務(wù)還以為訪問(wèn)的是原來(lái)的配置中心地址,就會(huì)出現(xiàn)健康檢查失敗的問(wèn)題,導(dǎo)致服務(wù)不可用的現(xiàn)象。
為什么健康檢查時(shí)好時(shí)壞?
上面的問(wèn)題會(huì)導(dǎo)致健康檢查失敗,但是這個(gè)服務(wù)并不是一直都不好,而是間斷性的出現(xiàn)不健康。這主要還是健康檢查時(shí)間中的機(jī)制導(dǎo)致,這里可以具體看ConfigServerHealthIndicator的getPropertySource函數(shù),該方法執(zhí)行的時(shí)候中間并不是每一次檢查都會(huì)去訪問(wèn)配置中心(執(zhí)行l(wèi)ocator.locate(this.environment)方法),因此客戶端的健康檢查并不會(huì)每次都健康檢查失敗,從而出現(xiàn)了微服務(wù)健康檢查時(shí)好時(shí)壞的情況。
如何解決
該問(wèn)題目前也在官方的issue中被提出,還處于open狀態(tài),具體可見(jiàn):https://github.com/spring-cloud/spring-cloud-config/issues/514
由于該問(wèn)題目前并沒(méi)有得到解決,雖然提交了一個(gè)PR,但是還有待完善以及提供一些測(cè)試,原本想完全處理好之后再寫一篇文章,但是發(fā)現(xiàn)最近不少問(wèn)過(guò)類似問(wèn)題,所以索性先寫一篇文章,提醒一下用戶以及給一些相關(guān)的建議。
當(dāng)前版本上不太容易通過(guò)擴(kuò)展的方式去解決的這個(gè)問(wèn)題,所以大家可以變通的去避免這個(gè)問(wèn)題:
- 部署在虛擬機(jī)上而不是容器上,避免IP的變動(dòng)
- 可以考慮關(guān)閉微服務(wù)隊(duì)config客戶端的健康檢查,增加參數(shù)management.health.config.enabled=false;但是這個(gè)操作有一個(gè)弊端,雖然遷移不會(huì)引發(fā)服務(wù)時(shí)好時(shí)壞的問(wèn)題了,但是如果有動(dòng)態(tài)配置刷新需求,如果遷移了配置中心,刷新配置操作也是會(huì)失敗的。
【本文為51CTO專欄作者“翟永超”的原創(chuàng)稿件,轉(zhuǎn)載請(qǐng)通過(guò)51CTO聯(lián)系作者獲取授權(quán)】