In this study, the possibility of automatically identifying, extracting, and validating financial Key Performance Indicators (KPIs) in Swedish annual reports was examined. Annual reports often contain information that is semi-structured, with tables, running text, notes, etc., which makes information extraction both time-consuming and difficult to standardise. The study combines doc- ument understanding, adaptive chunking, semantic- and lexical retrieval, and extraction methods to locate and extract KPIs from financial documents.

Before examining any retrieval methods, Docling was used to convert the semi-structured PDF documents into a structured representation. Thereafter, the content was segmented into text-, table- and group-chunks and converted into vector representations by using an embedding model, BGE-M3, which are then stored in the vector database Milvus, for hybrid-based retrieval. For the table extraction, a rule-based method was used, and for the text- and group-chunks, a fine- tuned XLM-RoBERTa model was used. To verify and validate the extracted values, a triangulation method was used, whereby multiple independent sources of evidence within the same annual report are compared. The system was trained on a goldset built from 29 annual reports, covering 62 different KPIs, and then tested on a goldset containing KPIs from 15 unseen annual reports. The retrieval was evaluated by the evaluation measures recall, Mean Reciprocal Rank (MRR), primary first, primary@20 and precision@20.

For the best configuration, querybased text with rerank, the test results with an MRR-score of 0.873 and a recall@20-value of 0.970 for the top 20 candidates, indicate that it was possible to find relevant chunks with the examined retrieval method. The test results of the table-extraction method showed an Exact Match (EM) in 97% of the cases, and for the text-extraction method, an EM in 66% of the cases was achieved. Both have matching F1 scores. Depending on the structure of the table, different results were obtained. For the best configuration, tables with only years in its columnheads showed a result EM in 99% of the cases, while columnheads without years showed EM in 81%. The triangulation method resulted in an EM score of 70% for the testset with the querybased text with rerank configuration. All the configurations resulted in a majority of Clear winners, where the triangulation results shows a large margin between the highest and second highest value.

In summery, the study shows that hybrid retrieval methods in combination with evidencebased triangulation can improve the reliability of information extraction from financial documents, with potential applications within credit risk and anomaly detection within the bank.