What is common among mighty water dam, humble electric fuse, and i-dunno-what hysterix and hysterix-go? Take a moment to think of an answer.
Yes. All of them are some sort of check that stops bad things from happening by deploying some safeguarding measure whenever a threshold is crossed.
What exactly is circuit breaking?
While researching circuit breakers, I stumbled upon a book called release it. It has a brilliant introduction to the subject which I will reproduce below:
“Not too long ago, when electrical wiring was first being built into houses, many people fell victim to physics. The unfortunates would plug too many appliances into their circuit. Each appliance drew a certain amount of current. When current is resisted, it produces heat proportional to the square of the current times the resistance (I2R).
Because they lacked superconducting home wiring, this hidden coupling between electronic gizmos made the wires in the walls get hot, sometimes hot enough to catch fire. Pfft. No more house.
The fledgling energy industry found a partial solution to the problem of resistive heating, in the form of fuses. The entire purpose of an electrical fuse is to burn up before the house does. It is a component designed to fail first, thereby controlling the overall failure mode. This brilliant device worked well, except for two flaws. First, a fuse is a disposable, one-time use item; therefore, it is possible to run out of them. Second, residential fuses (in the United States) were about the same diameter as copper pennies. Together, these two flaws led many people to conduct experiments with homemade, high-current, low-resistance fuses (that is, a 3/4-inch disk of copper). Pfft. No more house.
Residential fuses have gone the way of the rotary dial telephone. Now, circuit breakers protect overeager gadget hounds from burning their houses down. The principle is the same: detect excess usage, fail first, and open the circuit. More abstractly, the circuit breaker exists to allow one subsystem (an electrical circuit) to fail (excessive current draw, possibly from a short-circuit) without destroying the entire system (the house). Furthermore, once the danger has passed, the circuit breaker can be reset to restore full function to the system.
You can apply the same technique to software by wrapping dangerous operations with a component that can circumvent calls when the system is not healthy. This differs from retries, in that circuit breakers exist to prevent operations rather than reexecute them.”
Where do you break the circuit?
Short answer: Any method where you don’t control unexpected behaviors.
Imagine a call from your service to an external service. This can be HTTP call or gRPC call or tcp call. You don’t always know what the other service will return. It can throw an error that was not part of the contract. The other service might be down for whatever reason resulting in a connection timeout. The other service database might be down which might be resulting in read timeout. Now, just like the fuse sacrifices itself and saves the home from burning, you might want to avoid cascading failure (software engineering equivalent of the burning house).
How do you break the circuit?
Very simple. Enclose your external call (or any call prone to failure) in an object. The object should have metrics that monitor success and failure. When failure goes beyond a threshold, the object should stop doing that external call and instead fall back to some other option (maybe some default return or run in degraded mode). What you do in fallback, or as they say when the circuit is open really depends on the kind of business that your system is in. Say, if your system is inside a social media company that recommends friends intelligently, you can fall back to the default option of showing a randomly selected few profiles as suggestions rather than showing “Something went wrong” or keep showing the loading screen to the user which is a bad experience as a user. On the other hand, if your system accepts payment and your payment gateway is down, you can simply return a static message – “Payment gateway is down. Come back again in few minutes.”
Once a circuit is open, your circuit breaker can wait for some time (which is a wait time), and after that circuit breaker decides to check if the external service is up or not. In this state which is called half close state, circuit breaker makes a trial call. If the trial call succeeds, the circuit breaker returns to the close state else it continues to be in open state.
Can I make my own circuit breaker?
Yes. But there are many circuit breakers frameworks available on github. One popular example is Resilence4J in java and hysterix-go in Golang. I tried to implement it by having an abstract class that has the logic of circuit breaking. The class has an abstract method called start. The external call has to be implemented as an implementation of the “start” method. In the example given below, I will just try to demonstrate opening a circuit based on sliding window of last 10 calls.
Here’s an example of circuit breaker class.
package com.company.my.circuitbreaker;
public abstract class MyCircuitBreakerImpl<T, K> {
private int total;
private int failure;
private double threshold;
private int WINDOW_SIZE = 10;
private boolean isCircuitOpen = false;
MyCircuitBreakerImpl(double threshold){
this.threshold = threshold;
}
K run(T request){
isCircuitOpen();
if(isCircuitOpen){
System.out.println("Circuit Open. So Returning default response");
return null;
}
try {
K response = start(request);
return response;
} catch (Exception e){
System.out.println("Exception occured. Increasing failure count");
failure = failure + 1;
}
return null;
}
private void isCircuitOpen() {
total = total + 1;
if(total == 10){
double failurePercent = failure / total * 100;
if(failurePercent > threshold){
System.out.println("Circuit Open");
isCircuitOpen = true;
}
total = 0;
failure = 0;
}
}
abstract K start(T request);
}
We will now implement the start method.
package com.company.my.circuitbreaker;
public class MyCircuitBreakerFailureImpl extends MyCircuitBreakerImpl<Integer, String> {
MyCircuitBreakerFailureImpl(double threshold) {
super(threshold);
}
@Override
String start(Integer request) {
if(request < 10) {
System.out.println("****** I am success request ******");
return "pong";
}
try {
Thread.sleep(1000);
System.out.println(" ***** I am failed method. I am called here. ********");
} catch (InterruptedException e) {
e.printStackTrace();
}
throw new RuntimeException("Ooops! Something went wrong");
}
}
Running this through the main method.
package com.company.my.circuitbreaker;
public class TestMain {
public static double THRESHOLD = 10;
public static void main(String[] args){
MyCircuitBreakerFailureImpl myCircuitBreakerFailure = new MyCircuitBreakerFailureImpl(THRESHOLD);
for(int i = 1; i < 100; i++){
myCircuitBreakerFailure.run(i);
}
}
}
Output of the code.
****** I am success request ****** ****** I am success request ****** ****** I am success request ****** ****** I am success request ****** ****** I am success request ****** ****** I am success request ****** ****** I am success request ****** ****** I am success request ****** ****** I am success request ****** ***** I am failed method. I am called here. ******** Exception occured. Increasing failure count ***** I am failed method. I am called here. ******** Exception occured. Increasing failure count ***** I am failed method. I am called here. ******** Exception occured. Increasing failure count ***** I am failed method. I am called here. ******** Exception occured. Increasing failure count ***** I am failed method. I am called here. ******** Exception occured. Increasing failure count ***** I am failed method. I am called here. ******** Exception occured. Increasing failure count ***** I am failed method. I am called here. ******** Exception occured. Increasing failure count ***** I am failed method. I am called here. ******** Exception occured. Increasing failure count ***** I am failed method. I am called here. ******** Exception occured. Increasing failure count ***** I am failed method. I am called here. ******** Exception occured. Increasing failure count Circuit Open Circuit Open. So Returning default response Circuit Open. So Returning default response Circuit Open. So Returning default response Circuit Open. So Returning default response Circuit Open. So Returning default response Circuit Open. So Returning default response
How to implement a circuit breaker in Go?
Check out the hystrix-go library here. There is a sample implementation demonstrating the use here.
How to implement a circuit breaker in Java?
Check out the resilence4j library here. Here’s the simple implementation demonstrating the use here.
Further Reading:
If you liked this article and would like one such blog to land in your inbox every week, consider subscribing to our newsletter: https://skillcaptain.substack.com
Leave a Reply