This is the tenth week of the coding period of GSoC where the main aim was transparency and rigor — documenting exactly how every evaluation number is produced, testing whether few-shot prompting could improve results, and organizing the codebase properly.
Being Exact About How F1 Is Calculated
For each sentence, predicted triples from Gemma 3 4B are checked against the gold triples — subject, relation, and object all need to match for a triple to count as correct. Precision is correct triples divided by everything predicted; recall is correct triples divided by everything in the gold answer; F1 combines the two into one balanced number.
“Property”-type triples — around 74% of everything extracted — are excluded from this scoring on both sides. F1 measures how well the pipeline handles genuine facts specifically.
Where the gold answer comes from also matters: for Wikipedia and Train, the correct triples were generated by an LLM during data creation and kept only if GPT-OSS-120B scored them highly as a judge. So F1 on these two sources is really measuring agreement with another model’s judgment. BenchIE is different and genuinely independent — its subject/relation/object spans were human-verified in earlier project years. This makes BenchIE the stricter, more trustworthy of the three measurements.
Testing Few-Shot Prompting
The pipeline runs zero-shot by default. This week tested whether giving the model example sentences beforehand would help.
Random few-shot — three fixed examples picked at random — performed worse than zero-shot on every source tested. This matches a real, published finding (AgentRE, CIKM 2024), which found randomly-selected examples barely help.
Similarity-based few-shot — 5 freshly retrieved examples per test sentence, chosen using F2LLM-1.7B’s embeddings — under-performed zero-shot on Wikipedia and Train, but was the only approach that improved BenchIE, moving it from 0.173 to 0.184.
Organizing the Codebase Properly
The project’s GitHub repository was reorganized this week so every script that produced a real result lives in a clear, documented folder structure, with full project documentation in the README.
Documenting the Pipeline Visually
Two diagrams were built to make the full system understandable at a glance — the end-to-end pipeline from raw Wikipedia text through scoring, coreference resolution, extraction, normalization, human review, and feedback, and a detailed view of the predicate-linking sub-pipeline specifically.
Progress on DBpedia Ontology Mapping
Real progress was made on the DBpedia Mappings Wiki side this week: understanding that Hindi Wikipedia infobox fields aren’t yet connected to DBpedia’s ontology the way English infoboxes are, with “Infobox person” identified as the starting point. A Mappings Wiki account was created and official edit access requested.
Results So Far, at a Glance
| Metric | Value |
|---|---|
| Wikipedia F1 (full pipeline) | 0.493 |
| Train F1 | 0.537 |
| BenchIE F1 | 0.173 |
| Precision@40 (predicate linking) | 89.9% |
| Few-shot (random) | Underperformed zero-shot on every measure |
| Few-shot (similarity-based) | Mixed; the only variant to improve BenchIE |
Footnotes