GSoC Week 8: Building a Predicate-Linking Gold Set

This is the eighth week of the coding period of GSoC where the main aim was to build a gold set for predicate-to-DBpedia-property linking, fine-tune the normalization model on it, and run the complete pipeline end to end for the first time.

Building the Predicate-Linking Gold Set

When the extraction model produces a triple, its relation comes out as raw Hindi text and needs to be mapped to a formal DBpedia property. Built in three stages: every unique predicate was extracted from the training and validation sentences (8,034 total), F2LLM-8B retrieved the 50 most relevant candidate DBpedia properties for each, and GPT-OSS-120B picked the correct one or NONE if nothing fit.

A Fair, Unbiased Model Comparison

Since the gold set itself was built using F2LLM-8B, comparing two smaller candidate models against it was genuinely unbiased.

F2LLM-1.7B outperformed at every level and was chosen for fine-tuning going forward.

Fine-Tuning, and Checking for Catastrophic Forgetting

585 entries (10% of the gold set) were held out and never touched during training, to test the model afterward on genuinely unseen predicates. F2LLM-1.7B was fine-tuned via QLoRA across two rounds, 3 epochs then 6 more for 9 total.

The second round showed a real, clear improvement — up nearly 5 percentage points at precision@1 and over 9 points at precision@10 — with no sign of catastrophic forgetting.

Evaluating Both Extraction Checkpoints

Using the same fixed set of 150 sentences from the previous week, both trained Gemma 3 4B extraction checkpoints (two learning rates) were scored by GPT-OSS-120B as an LLM judge.

The higher learning rate checkpoint won clearly on Wikipedia and training data, and became the checkpoint used for the rest of the project.

The Full Pipeline, End to End, for the First Time

With both the fine-tuned Gemma 3 4B extraction model and the fine-tuned F2LLM-1.7B normalization model in place, I ran the complete pipeline — extraction followed by normalization — on the same 150-sentence evaluation set.

BenchIE remained noticeably harder than the other two sources, which makes sense given it’s independently authored, out-of-distribution text rather than data generated for this project.

Footnotes