However, ≥2 simultaneous failures and the data is irrecoverable. B1⊕ P1⊕B3⊕B4⊕B5⊕B6 to recover the missing data, B2. And if any 1 data block fails, for example if B2 fails, the remaining blocks are XOR’ed i.e. This increases efficiency significantly as compared to HDFS’ 3-way replication. Meaning - Arrangement of inputs doesn’t change the output.Īs a corollary, we can scale up to our example, with 6 data blocks as input we will consume a total of 7 blocks of storage on the disk, 6 data blocks and 1 parity block. Furthermore, XOR has 2 beautiful mathematical properties: In Exclusive-OR or XOR encoding, denoted as ⊕, the data (INs) are passed through a mathematical operation, called XOR’ing, and results (OUT) in a single parity block, as can be seen above. In case of a failure aka erasure, the data can be reconstructed from this encoding group, known as decoding.īasic mathematical understanding on OR & XOR This process is known as encoding and the (data + parity block/s) is known as an encoding group. Erasure CodingĮrasure Coding (EC) was introduced to Hadoop in version 3.x through a joint effort between the good folks at Intel and Cloudera.Ĭontinuing with the same example, once the data is split into several blocks in HDFS, it is passed on to EC as input, which returns a number of parity blocks. This has led to a new paradigm in securing data with significantly higher efficiency.

While access to cheap storage and excellent network bandwidth is a reality in today’s world, this method of building redundancy for fault-tolerance and resilience is still highly inefficient. Not to mention, the network and I/O usage to create, maintain and perform BAU activities. 200% storage overhead with a (1/n) or (1/3) efficiency or 33% and n-1 = 2 as fault tolerance.

RF = 3), where n = number of replicas, we have 2 replicated data blocks i.e. Consider scaling this to the Petabyte scale, and you will quickly realize the significant under-utilization of available space due to redundancy brought by data replication. With a replication factor of 3 it will consume (6*3) = 18 blocks and (700 MB * 3) = 2100 MB of storage. A 700 MB file is broken down into 6 blocks. The above image is a quick demonstration of the split that we have just discussed.

Duplicacy erasure coding