乱世浮生

Eureka 实例注册状态保持 STARTING 的问题排查

这是真实发生在生产环境的 case，实例启动后正常运行，而在注册中心的状态一直保持STARTING，而本地的状态为UP。导致服务的消费方无法发现可用实例。

这种情况的出现概率非常低，运行一年多未发现两个实例同时出现问题的情况，因此多实例运行可以避免。文末有问题的解决方案，不想花时间看分析过程可直接跳到最后。

环境说明：

eureka-client: 1.7.2 spring-boot: 1.5.12.RELEASE spring-cloud: Edgware.SR3

问题重现

借助Btrace重现, java -noverify -cp .:btrace-boot.jar -javaagent:btrace-agent.jar=script=<pre-compiled-btrace-script> <MainClass> <AppArguments>

思路

主线程更新实例本地状态(STARTING->UP)前, 等待心跳线程完成第一次心跳并尝试注册实例, 获取到当前的状态STARTING. 主线程更新状态后触发

Btrace 脚本

import com.sun.btrace.annotations.BTrace;
import com.sun.btrace.annotations.Kind;
import com.sun.btrace.annotations.Location;
import com.sun.btrace.annotations.OnMethod;

import java.util.concurrent.atomic.AtomicBoolean;

import static com.sun.btrace.BTraceUtils.currentThread;
import static com.sun.btrace.BTraceUtils.println;

/**
 * @author Addo.Zhang
 * @date 2019-07-31
 */
@BTrace(unsafe = true)
public class EurekaRequest {
    public static final AtomicBoolean heartbeatThreadRegistrationStarted = new AtomicBoolean(false);
    public static final AtomicBoolean replicatorThreadRegistrationCompleted = new AtomicBoolean(false);
    public static final AtomicBoolean statusUP = new AtomicBoolean(false);

    @OnMethod(clazz = "org.springframework.cloud.netflix.eureka.serviceregistry.EurekaServiceRegistry", location = @Location(value = Kind.LINE, line = 45))
    public static void waitHeartbeatExecution() {
        println(currentThread() + " is waiting heartbeatThreadRegistrationStarted thread executing first");
        while (!heartbeatThreadRegistrationStarted.get()) ;
    }

    @OnMethod(clazz = "org.springframework.cloud.netflix.eureka.serviceregistry.EurekaServiceRegistry", location = @Location(value = Kind.LINE, line = 46))
    public static void markStatusUp() {
        statusUP.set(true);
        println("Heartbeat thread executed and " + currentThread() + " continues procedure to change status to [UP]");
    }

    @OnMethod(clazz = "com.netflix.discovery.converters.EurekaJacksonCodec$InstanceInfoSerializer", location = @Location(value = Kind.LINE, line = 369))
    public static void continueRegistrationExecution() {
        doExecution();
    }

    @OnMethod(clazz = "com.logancloud.forge.discovery.converters.LoganEurekaJacksonCodec$LoganInstanceInfoSerializer", location = @Location(value = Kind.LINE, line = 117))
    public static void continueRegistrationExecution2() {
        doExecution();
    }

    private static void doExecution() {
        println(currentThread() + " started to proceed registration");
        if (Thread.currentThread().getName().contains("HeartbeatExecutor")) {
            heartbeatThreadRegistrationStarted.set(true);
            while (!statusUP.get() || !replicatorThreadRegistrationCompleted.get()) {
            }
            try {
                Thread.sleep(500); //interval for replicator registration request completed.
            } catch (InterruptedException e) {
                Thread.currentThread().interrupt();
            }
        } else if (Thread.currentThread().getName().contains("InstanceInfoReplicator")) {
            replicatorThreadRegistrationCompleted.set(true);
        }
        println(currentThread() + " thread registration completed");
    }
}