The evaluation tool can be downloaded here.
The ICDAR 2019 cTDaR evaluates two aspects of table analysis: table detection and recognition. We choose the metric (i) to evaluate the performance of table region detection, and apply the metric (ii) to evaluate that of table recognition. Based on these measures, an overall performance of various algorithms can be compared with each other.
Metric for table region detection task
The task is evaluated by a traditional method. Intersection over Union (IoU)[1] is calculated to estimate whether a table region detected by the participant is correctly located. Let A denote the region detected by a participant and B denote the corresponding region described in the groundtruth file. The IOU is calculated as follows:
$$IoU=\frac{A\bigcap B}{A+B-A\bigcap B}$$
For the task, the precision/recall curve is computed from a method’s ranked output. Recall is defined as the proportion of all positive examples ranked above a given rank. Precision is the proportion of all examples above that rank which are from the positive class. Furthermore, F1 score will be computed based on the recall and precision value. The precision, recall and F1 score will be calculated under circumstances that IoU is equal to 0.6, 0.7, 0.8 and 0.9.
Metric for table recognition task
The task is evaluated by the structure of a table that is defined as a matrix of cells. For each cell, participants are required to return the coordinates of each corner of bounding polygon, textual content (optional) and its start/end column/row positions. We propose the following metric: Cell’s adjacency relation-based table structure evaluation (inspired by Gobel’s method [2]).
For comparing two cell structures, we use the method: for each table region, we align each groundtruth cell to the predicted cell with IoU > σ, identify the valid predicted cells, and then generate a list of adjacency relations between each valid cell and its nearest neighbor in horizontal and vertical directions. Blank cells are not represented in the grid. No adjacency relations are generated between blank cells or a blank cell and a content cell. This 1-D list of adjacency relations can be compared to the groundtruth by using precision and recall measures. If both cells are identical and the direction matches, then it is marked as correctly retrieved; otherwise it is marked as incorrect.
The precision, recall and F1 score will be calculated under circumstances that IoU is equal to 0.6, 0.7, 0.8 and 0.9 as the evaluation for track A.
We will also release a number of tools to enable the participants to automatically compare their result to the groundtruth.
[1] L. Gao, X. Yi, Z. Jiang, L. Hao and Z. Tang, “ICDAR 2017 POD Competition,” in ICDAR, 2017, pp. 1417-1422.
[2] M. C. Gobel, T. Hassan, E. Oro, G. Orsi, ”ICDAR2013 Table Competition,” in Proc. of the 12th ICDAR (IEEE, 2013), pp. 1449-1453.