Classifying cultural items across Wikipedia languages

Task

Each item, from abstract concepts to specific people, gets one of three labels: cultural agnostic (shared and recognised across many cultures), cultural representative (widely known, with a traceable cultural or national origin) or cultural exclusive (known mainly within one culture or country). The training set has 6,251 automatically labelled items, and the validation set 300 items labelled by people. The project compares two methods: one without language models, and one fine-tuning a pre-trained language model.

Without a language model

The idea: a concept that has spread everywhere, like bread, has very different names from one language to another, because each culture adapted it long ago. An item specific to one culture, or recent, like pizza, keeps a name close to its original. So the model looks at how each item lives across Wikipedia:

• Wikidata metadata. Items known everywhere are translated into more languages and described more richly. The features include the number of descriptions, translations and statements, whether a country of origin is set, and the item’s type and category.
• Attention. For each of the 73 language editions, how actively the item’s page is edited, as a proxy for how much attention each culture gives it.
• The name itself. Simple signals, such as all capitals, digits or special characters, which often point to a named entity.
• How the name changes. For every pair of languages, the distance between the item’s names, in spelling (Levenshtein distance) and in pronunciation (IPA transcriptions with Epitran, compared with PanPhon). Each distance is weighted by how related the two languages are (Malaviya et al., 2017), so that Spanish and Italian names don’t look alike just because the languages do. Each 73 × 73 matrix is then summarised as a histogram.

These 170 features feed a feed-forward network (1024 → 768 → 512 → 256 → 128, GELU, batch normalisation, dropout of about 0.4), trained with AdamW, with hyperparameters chosen by grid search.

With a language model

DistilBERT, RoBERTa and DeBERTa were fine-tuned on the first paragraph of each item’s Wikipedia page, with epochs chosen for comparable running times. All three began to overfit early. Early stopping was left out on purpose: the best training accuracy didn’t carry over to validation, likely because the two sets were labelled differently. RoBERTa was chosen as the final model, as the best balance between cost and accuracy.

Results

On the 300 validation items:

Model	Accuracy	Macro F1
Feature-based network	0.77	0.77
DistilBERT (100 epochs, 2 hours)	0.75	0.73
RoBERTa (50 epochs, 2 hours)	0.77	0.76
DeBERTa (3 epochs, 1 hour)	0.75	0.74

Cultural agnostic items were the easiest for every model. The feature-based network found representative items the hardest (F1 0.69), and the language models found exclusive ones the hardest (F1 0.65–0.68). The best scores come close to what we estimated as the accuracy of the training labels themselves.

The feature-based network matches RoBERTa here, and it is easier to interpret. It also needs only structured metadata, while language models depend on rich text that isn’t always available.

Team project with Lorenzo Ciappetta and Marco Mingoli, Sapienza University of Rome.