Transfer Learning (11): Cross-Lingual Transfer

English has abundant labeled data, but there are over 7,000 languages in the world. How can models transfer knowledge learned from English to low-resource languages? Cross-Lingual Transfer enables models trained on English to be directly used on Chinese, Arabic, Swahili — without any target language labeled data.

This article systematically explains methods and implementations of bilingual word embedding alignment, multilingual pre-training, and cross-lingual prompt learning, starting from the mathematical principles of multilingual representation space. We analyze language universals and differences, zero-shot transfer performance, and language selection strategies, and provide complete code (280+ lines) for implementing cross-lingual text classification from scratch.

Problem Definition of Cross-Lingual Transfer

Zero-Shot Cross-Lingual Learning

Scenario: Train on source language (e.g., English), test on target language(e.g., Chinese), with no labeled data in target language.

Formalized as:Goal: Minimize test loss.

Challenges: - Source and target language vocabularies completely different - Syntactic structures and word order may differ greatly - Cultural and pragmatic differences

Few-Shot Cross-Lingual Learning

Scenario: Target language has small amount of labeled data (e.g., 10-100 samples per class).

Formalized as:where (target language samples much fewer than source language).

Multi-Source Language Transfer

Scenario: Transfer from multiple source languagesto target language.

Objective function:

Advantage: Language diversity provides richer linguistic features.

Evaluation Metrics

1. Zero-Shot Accuracy:

2. Cross-Lingual Transfer Gap:Smaller is better, 0 indicates perfect transfer.

3. Average Performance:

Mathematical Principles of Multilingual Representation

Shared Semantic Space

Assumption: Different languages' ways of expressing same concepts share commonalities at deep semantic level.

Formalized as: There exists a language-agnostic semantic spacesuch that:For semantically equivalent sentences (,):

Intuition: 猫 (Chinese) and cat (English) should map to the same region in semantic space.

Theoretical Foundations of Language Universals

Universal Grammar

Chomsky's Universal Grammar theory: All human languages share underlying grammatical structures.

Evidence: - Word orders like SVO, SOV have corresponding relationships at deep level - Parts of speech like nouns, verbs exist across all languages - Recursive structures, question transformations are cross-linguistically universal

Distributional Semantics Hypothesis

"Word meaning is determined by its context" (Distributional Hypothesis):Cross-lingual extension: Similar cross-lingual contexts should produce similar representations.

Bilingual Word Embedding Alignment

Linear Transformation Assumption

Assume linear transformationexists between source language embeddingsand target language embeddings:Goal: Learnto make translation pairsas close as possible.

Procrustes Alignment:Closed-form solution (Orthogonal Procrustes Problem):where.

Adversarial Training Alignment

Conneau et al.¹ proposed unsupervised alignment method:

Discriminator: Distinguish between source and target language word embeddings:

Generator (alignment matrix): Minimize discriminator's ability:

Intuition: If discriminator cannot distinguish aligned source from target language, alignment is successful.

Multilingual Sentence Representations

Parallel Sentence Alignment

Given parallel corpus(,, semantically equivalent), learn encoder:

Translation Language Modeling (TLM)²:

Jointly model parallel sentence pairs:Source and target language tokens can mutually attend, learning alignment.

Contrastive Learning

LASER³ uses contrastive loss:wherecontains negative samples in batch,is cosine similarity,is temperature parameter.

Multilingual Pre-trained Models

Multilingual BERT (mBERT)

Architecture and Pre-training

mBERT⁴ is pre-trained on Wikipedia in 104 languages using:

1. Masked Language Modeling (MLM):

2. Shared vocabulary: 110K WordPiece tokens covering all languages

Key design: - No explicit cross-lingual supervision signal (no parallel corpus) - Sentences from different languages randomly mixed during training - All layers share parameters

Why Does mBERT Work?

Theoretical explanation⁵:

1. Anchor Vocabulary: Numbers, punctuation, English loanwords shared across languages
2. Deep parameter sharing: Forces model to learn language-agnostic features
3. Code-Switching: Naturally occurring multilingual mixing in training data

Empirical findings: - mBERT's hidden layer representations highly aligned across languages - Even without parallel corpus, similar concepts have close representations in different languages

XLM-RoBERTa (XLM-R)

Improved Design

XLM-R⁶ is pre-trained on 2.5TB text in 100 languages, compared to mBERT:

1. Larger model: 550M parameters (mBERT has 110M)
2. More data: 2.5TB vs few GB
3. Better sampling strategy:

Define sampling probability for language:whereis data volume of language,(mitigates high-resource language dominance).

Performance Comparison

On XNLI (cross-lingual natural language inference):

Model	English	Average	Worst Language
mBERT	81.4	65.4	58.3 (Urdu)
XLM-R	88.7	76.2	68.4 (Swahili)

XLM-R significantly outperforms mBERT across all languages.

mT5

Architecture

mT5⁷ is the multilingual version of T5, covering 101 languages, using:

1. Text-to-Text framework: All tasks unified as text generation

2. Denoising Autoencoding:

   • Randomly mask text spans
   • Model reconstructs complete text

Advantages: - Generative architecture suitable for seq2seq tasks (translation, summarization) - Unified framework supports multi-task learning

Comparison with XLM-R

Dimension	XLM-R	mT5
Architecture	Encoder-only	Encoder-decoder
Pre-training task	MLM	Denoising
Applicable tasks	Classification, tagging	Generation, translation
Inference overhead	Low	High

Zero-Shot Cross-Lingual Transfer

Direct Transfer

Simplest strategy: Train on source language, directly test on target language.

Algorithm:

1. Fine-tune multilingual model with source language data$_s_t$ Key: Multilingual model's representations already aligned.

Performance:

On XNLI, English → other languages zero-shot accuracy:

Target Language	mBERT	XLM-R
French	73.5	79.2
Chinese	68.3	76.7
Arabic	64.1	73.8
Swahili	57.2	68.4

High-resource languages perform better.

Translate-Train

Strategy: Translate source language training data to target language, then train on target language.

Algorithm:

1. Use machine translation to translateto$_t_t$3. Evaluate on real target language test data

Advantage: Model directly trained on target language, avoiding language differences.

Disadvantages: - Depends on translation quality (translation errors propagate) - Semantics may be lost or distorted

Translate-Test

Strategy: Translate target language test data to source language, predict with source language model.

Algorithm:

1. Train model on source language$_sx^{(t)}{(s)} = f(^{(s)})$ Advantage: Leverages high-quality source language model.

Disadvantages: Requires translation at inference, increasing latency and cost.

Ensemble Methods

Translate-Train-All (TTA):

Translate training data to all languages, train jointly:whereis training data translated to language.

Advantage: Model sees multiple language expressions, strong generalization.

Disadvantage: High computational cost (requires multiple translations and training).

Cross-Lingual Prompt Learning

Multilingual Prompt Templates

Prompt-Based Learning: Convert task to language model fill-in-the-blank.

English sentiment classification:

The movie was great. It was [MASK]. → wonderful

Cross-lingual extension: Use multilingual templates.

Chinese:

这部电影很好。它[MASK]。 → 很棒

Challenge: Template design varies greatly across languages.

Automatic Template Search

X-FACTR⁸: Automatically discover cross-lingual prompt templates.

Algorithm:

1. Use AutoPrompt⁹ to search optimal template on English
2. Translate template to target language
3. Fine-tune template on target language

Example:

English template:

[X] is located in [Y]. → [X] is in the country of [MASK].

Translated to French:

[X] se trouve en [Y]. → [X] est dans le pays de [MASK].

Language-Agnostic Prompts

XPROMPT¹⁰: Learn language-agnostic continuous prompts.

Model input:whereis learnable continuous vector (language-agnostic).

Training objective:

Advantage: One prompt applicable to all languages, no translation needed.

Code-Switching and Language Mixing

Code-Switching Phenomenon

Code-Switching: Mixing multiple languages within a single sentence.

Example:

I'm feeling 很累，想 sleep 了。
(English+Chinese)

Prevalence: Very common in multilingual communities (e.g., Singapore, India, US Latino communities).

Code-Switching Data Augmentation

Strategy: Artificially create code-switching data during training.

Algorithm¹¹:

1. Parse sentence dependency tree
2. Randomly select words to replace with target language translations
3. Maintain grammatical structure

Example:

Original sentence (English):

I love this movie very much.

Code-switched (English → Chinese):

I 喜欢 this 电影 very much.

Effect: Improves cross-lingual robustness and zero-shot performance.

Language Adaptive Pre-training

MALAPT¹²: Continue pre-training on target language monolingual data.

Algorithm:

1. Initialize with multilingual model (e.g., XLM-R)
2. Continue MLM training on target language monolingual corpus
3. Fine-tune on downstream task

Effect:

Setting	English → Chinese (XNLI)
XLM-R	76.7
+ MALAPT	79.3 (+2.6)

Target language pre-training significantly improves performance.

Complete Code Implementation: Cross-Lingual Text Classification

Below is a complete cross-lingual text classification system including multilingual model loading, zero-shot transfer, few-shot fine-tuning, and evaluation.

"""
Cross-Lingual Text Classification from Scratch
Includes: Multilingual BERT loading, zero-shot transfer, few-shot fine-tuning
"""

import torch
import torch.nn as nn
import torch.optim as optim
from torch.utils.data import Dataset, DataLoader
from transformers import BertTokenizer, BertModel
import numpy as np
import matplotlib.pyplot as plt
from typing import List, Dict, Tuple
from sklearn.metrics import accuracy_score, classification_report

# Set random seed
torch.manual_seed(42)
np.random.seed(42)

# ============================================================================
# Multilingual Text Classifier
# ============================================================================

class MultilingualTextClassifier(nn.Module):
    """
    Text classifier based on multilingual BERT
    """
    def __init__(self, model_name: str = 'bert-base-multilingual-cased', num_classes: int = 3):
        super().__init__()
        
        # Load multilingual BERT
        self.bert = BertModel.from_pretrained(model_name)
        self.hidden_size = self.bert.config.hidden_size
        
        # Classification head
        self.classifier = nn.Sequential(
            nn.Dropout(0.1),
            nn.Linear(self.hidden_size, self.hidden_size),
            nn.Tanh(),
            nn.Dropout(0.1),
            nn.Linear(self.hidden_size, num_classes)
        )
        
    def forward(self, input_ids, attention_mask):
        # BERT encoding
        outputs = self.bert(input_ids=input_ids, attention_mask=attention_mask)
        
        # Use [CLS] token representation
        cls_output = outputs.last_hidden_state[:, 0, :]
        
        # Classification
        logits = self.classifier(cls_output)
        
        return logits

# ============================================================================
# Multilingual Dataset
# ============================================================================

class MultilingualDataset(Dataset):
    """
    Multilingual text classification dataset
    """
    def __init__(self, texts: List[str], labels: List[int], tokenizer, max_length: int = 128):
        self.texts = texts
        self.labels = labels
        self.tokenizer = tokenizer
        self.max_length = max_length
        
    def __len__(self):
        return len(self.texts)
    
    def __getitem__(self, idx):
        text = self.texts[idx]
        label = self.labels[idx]
        
        # Tokenize
        encoding = self.tokenizer(
            text,
            max_length=self.max_length,
            padding='max_length',
            truncation=True,
            return_tensors='pt'
        )
        
        return {
            'input_ids': encoding['input_ids'].squeeze(0),
            'attention_mask': encoding['attention_mask'].squeeze(0),
            'label': torch.tensor(label, dtype=torch.long)
        }

# ============================================================================
# Generate Synthetic Multilingual Data
# ============================================================================

def create_synthetic_multilingual_data(num_samples_per_lang: int = 200) -> Dict[str, Tuple[List[str], List[int]]]:
    """
    Generate synthetic multilingual sentiment classification data
    Includes English, Chinese, French
    """
    # Synthetic data (should use real data in actual applications)
    data = {
        'en': {
            'positive': [
                "This movie is absolutely fantastic!",
                "I love this product, it's amazing.",
                "Great service, highly recommended!",
                "Excellent quality, will buy again.",
                "Wonderful experience, very satisfied."
            ] * (num_samples_per_lang // 5),
            'neutral': [
                "The movie was okay.",
                "Product works as expected.",
                "Service was average.",
                "Quality is acceptable.",
                "Experience was fine."
            ] * (num_samples_per_lang // 5),
            'negative': [
                "Terrible movie, waste of time!",
                "Product broke after one day.",
                "Very poor service, disappointed.",
                "Bad quality, do not buy.",
                "Horrible experience, never again."
            ] * (num_samples_per_lang // 5)
        },
        'zh': {
            'positive': [
                "这部电影太棒了！",
                "我很喜欢这个产品，太神奇了。",
                "服务很好，强烈推荐！",
                "质量很好，会再买。",
                "体验很棒，非常满意。"
            ] * (num_samples_per_lang // 5),
            'neutral': [
                "电影还可以。",
                "产品符合预期。",
                "服务一般。",
                "质量尚可。",
                "体验还行。"
            ] * (num_samples_per_lang // 5),
            'negative': [
                "电影太烂了，浪费时间！",
                "产品用一天就坏了。",
                "服务太差，很失望。",
                "质量不好，不要买。",
                "体验糟糕，再也不会了。"
            ] * (num_samples_per_lang // 5)
        },
        'fr': {
            'positive': [
                "Ce film est absolument fantastique!",
                "J'adore ce produit, c'est incroyable.",
                "Excellent service, hautement recommand é!",
                "Excellente qualit é, j'ach è terai encore.",
                "Exp é rience merveilleuse, tr è s satisfait."
            ] * (num_samples_per_lang // 5),
            'neutral': [
                "Le film é tait correct.",
                "Le produit fonctionne comme pr é vu.",
                "Le service é tait moyen.",
                "La qualit é est acceptable.",
                "L'exp é rience é tait bien."
            ] * (num_samples_per_lang // 5),
            'negative': [
                "Film terrible, perte de temps!",
                "Le produit s'est cass é apr è s un jour.",
                "Service tr è s mauvais, d éç u.",
                "Mauvaise qualit é, n'achetez pas.",
                "Exp é rience horrible, plus jamais."
            ] * (num_samples_per_lang // 5)
        }
    }
    
    # Organize data
    result = {}
    for lang, sentiment_data in data.items():
        texts = []
        labels = []
        
        for label_idx, (sentiment, examples) in enumerate(sentiment_data.items()):
            texts.extend(examples)
            labels.extend([label_idx] * len(examples))
        
        # Shuffle data
        indices = np.random.permutation(len(texts))
        texts = [texts[i] for i in indices]
        labels = [labels[i] for i in indices]
        
        result[lang] = (texts, labels)
    
    return result

# ============================================================================
# Training and Evaluation Functions
# ============================================================================

def train_epoch(model, dataloader, optimizer, criterion, device):
    """
    Train one epoch
    """
    model.train()
    total_loss = 0
    all_preds = []
    all_labels = []
    
    for batch in dataloader:
        input_ids = batch['input_ids'].to(device)
        attention_mask = batch['attention_mask'].to(device)
        labels = batch['label'].to(device)
        
        # Forward pass
        logits = model(input_ids, attention_mask)
        loss = criterion(logits, labels)
        
        # Backward pass
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
        
        # Statistics
        total_loss += loss.item()
        preds = torch.argmax(logits, dim=1).cpu().numpy()
        all_preds.extend(preds)
        all_labels.extend(labels.cpu().numpy())
    
    avg_loss = total_loss / len(dataloader)
    accuracy = accuracy_score(all_labels, all_preds)
    
    return avg_loss, accuracy

def evaluate(model, dataloader, device):
    """
    Evaluate model
    """
    model.eval()
    all_preds = []
    all_labels = []
    
    with torch.no_grad():
        for batch in dataloader:
            input_ids = batch['input_ids'].to(device)
            attention_mask = batch['attention_mask'].to(device)
            labels = batch['label'].to(device)
            
            # Forward pass
            logits = model(input_ids, attention_mask)
            preds = torch.argmax(logits, dim=1).cpu().numpy()
            
            all_preds.extend(preds)
            all_labels.extend(labels.cpu().numpy())
    
    accuracy = accuracy_score(all_labels, all_preds)
    
    return accuracy, all_preds, all_labels

# ============================================================================
# Main Experiment: Cross-Lingual Zero-Shot Transfer
# ============================================================================

def run_cross_lingual_experiment(
    source_lang: str = 'en',
    target_langs: List[str] = ['zh', 'fr'],
    num_epochs: int = 5,
    batch_size: int = 16,
    device: str = 'cpu'
):
    """
    Run cross-lingual transfer experiment
    """
    print("="*70)
    print("Cross-Lingual Transfer Learning Experiment")
    print("="*70)
    
    # Create data
    print("\nCreating synthetic multilingual data...")
    data = create_synthetic_multilingual_data(num_samples_per_lang=200)
    
    # Load tokenizer
    print("\nLoading multilingual BERT tokenizer...")
    tokenizer = BertTokenizer.from_pretrained('bert-base-multilingual-cased')
    
    # Create source language dataset
    print(f"\nPreparing source language ({source_lang}) data...")
    train_size = int(0.8 * len(data[source_lang][0]))
    
    source_train_texts = data[source_lang][0][:train_size]
    source_train_labels = data[source_lang][1][:train_size]
    source_test_texts = data[source_lang][0][train_size:]
    source_test_labels = data[source_lang][1][train_size:]
    
    train_dataset = MultilingualDataset(source_train_texts, source_train_labels, tokenizer)
    source_test_dataset = MultilingualDataset(source_test_texts, source_test_labels, tokenizer)
    
    train_loader = DataLoader(train_dataset, batch_size=batch_size, shuffle=True)
    source_test_loader = DataLoader(source_test_dataset, batch_size=batch_size, shuffle=False)
    
    # Create target language test sets
    target_test_loaders = {}
    for lang in target_langs:
        test_texts = data[lang][0]
        test_labels = data[lang][1]
        test_dataset = MultilingualDataset(test_texts, test_labels, tokenizer)
        target_test_loaders[lang] = DataLoader(test_dataset, batch_size=batch_size, shuffle=False)
    
    # Create model
    print("\nInitializing model...")
    model = MultilingualTextClassifier(num_classes=3).to(device)
    
    # Optimizer and loss function
    optimizer = optim.Adam(model.parameters(), lr=2e-5)
    criterion = nn.CrossEntropyLoss()
    
    # ========================================================================
    # Train on source language
    # ========================================================================
    print(f"\n{'='*70}")
    print(f"Training on source language: {source_lang}")
    print(f"{'='*70}")
    
    train_accuracies = []
    
    for epoch in range(num_epochs):
        train_loss, train_acc = train_epoch(model, train_loader, optimizer, criterion, device)
        train_accuracies.append(train_acc)
        
        # Evaluate on source language test set
        source_acc, _, _ = evaluate(model, source_test_loader, device)
        
        print(f"Epoch [{epoch+1}/{num_epochs}]")
        print(f"  Train Loss: {train_loss:.4f}, Train Acc: {train_acc:.4f}")
        print(f"  {source_lang} Test Acc: {source_acc:.4f}")
    
    # ========================================================================
    # Zero-shot cross-lingual evaluation
    # ========================================================================
    print(f"\n{'='*70}")
    print("Zero-Shot Cross-Lingual Evaluation")
    print(f"{'='*70}")
    
    results = {source_lang: source_acc}
    
    for lang in target_langs:
        acc, preds, labels = evaluate(model, target_test_loaders[lang], device)
        results[lang] = acc
        
        print(f"\n{source_lang} -> {lang} Zero-Shot Accuracy: {acc:.4f}")
        print(f"Classification Report:")
        print(classification_report(labels, preds, target_names=['positive', 'neutral', 'negative']))
    
    # Calculate average performance and transfer gaps
    avg_acc = np.mean(list(results.values()))
    transfer_gaps = {lang: results[source_lang] - results[lang] for lang in target_langs}
    
    print(f"\n{'='*70}")
    print("Summary")
    print(f"{'='*70}")
    print(f"Average Accuracy across all languages: {avg_acc:.4f}")
    print(f"\nTransfer Gaps:")
    for lang, gap in transfer_gaps.items():
        print(f"  {source_lang} -> {lang}: {gap:.4f}")
    
    return results, transfer_gaps

# ============================================================================
# Visualization
# ============================================================================

def plot_cross_lingual_results(results: Dict[str, float], transfer_gaps: Dict[str, float]):
    """
    Visualize cross-lingual transfer results
    """
    fig, axes = plt.subplots(1, 2, figsize=(14, 5))
    
    # 1. Accuracy per language
    languages = list(results.keys())
    accuracies = list(results.values())
    
    colors = ['#2ecc71' if lang == list(results.keys())[0] else '#3498db' for lang in languages]
    
    axes[0].bar(languages, accuracies, color=colors, alpha=0.8)
    axes[0].set_xlabel('Language', fontsize=12)
    axes[0].set_ylabel('Accuracy', fontsize=12)
    axes[0].set_title('Zero-Shot Cross-Lingual Accuracy', fontsize=14, fontweight='bold')
    axes[0].set_ylim([0, 1])
    axes[0].grid(True, alpha=0.3, axis='y')
    
    # Add value labels
    for i, (lang, acc) in enumerate(zip(languages, accuracies)):
        axes[0].text(i, acc + 0.02, f'{acc:.3f}', ha='center', fontsize=10)
    
    # 2. Transfer gaps
    target_langs = list(transfer_gaps.keys())
    gaps = list(transfer_gaps.values())
    
    axes[1].bar(target_langs, gaps, color='#e74c3c', alpha=0.8)
    axes[1].set_xlabel('Target Language', fontsize=12)
    axes[1].set_ylabel('Transfer Gap', fontsize=12)
    axes[1].set_title('Cross-Lingual Transfer Gap', fontsize=14, fontweight='bold')
    axes[1].grid(True, alpha=0.3, axis='y')
    
    # Add value labels
    for i, (lang, gap) in enumerate(zip(target_langs, gaps)):
        axes[1].text(i, gap + 0.01, f'{gap:.3f}', ha='center', fontsize=10)
    
    plt.tight_layout()
    plt.savefig('cross_lingual_transfer.png', dpi=150, bbox_inches='tight')
    plt.close()
    print("\nVisualization saved to cross_lingual_transfer.png")

# ============================================================================
# Main Function
# ============================================================================

def main():
    device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
    print(f"Using device: {device}")
    
    # Run experiment
    results, transfer_gaps = run_cross_lingual_experiment(
        source_lang='en',
        target_langs=['zh', 'fr'],
        num_epochs=5,
        batch_size=16,
        device=device
    )
    
    # Visualize
    plot_cross_lingual_results(results, transfer_gaps)
    
    print("\n" + "="*70)
    print("Experiment completed!")
    print("="*70)

if __name__ == "__main__":
    main()

Code Explanation

Core Components:

1. MultilingualTextClassifier: Classifier based on mBERT
2. MultilingualDataset: Multilingual data loading
3. Zero-shot transfer: Train on English, test on Chinese/French

Experimental Design:

1. Train sentiment classifier on source language (English)
2. Zero-shot transfer to target languages (Chinese, French)
3. Calculate transfer gaps and average performance

Key Details:

• Use mBERT's shared representation space
• No target language labeled data
• Evaluate cross-lingual transfer effectiveness

Challenges and Frontiers of Cross-Lingual Transfer

Impact of Language Differences

Language Family Similarity

Finding: Languages from similar families transfer better¹³.

Source → Target	Accuracy
English → French (same family)	78.3
English → Chinese (different family)	69.1
French → Spanish (same family)	81.7

Reasons: - Similar word order (SVO vs SOV) - Shared vocabulary (Romance languages) - Close grammatical structures

Writing Systems

Finding: Languages with same writing system transfer more easily.

Writing System	Example Languages	Transfer Difficulty
Latin alphabet	English, French, German, Spanish	Low
Chinese characters	Chinese, Japanese (partial)	Medium
Arabic alphabet	Arabic, Persian	Medium
Other (Thai, Korean)	-	High

Challenges for Low-Resource Languages

Problems:

1. Insufficient pre-training data: Few Wikipedia pages (e.g., Swahili has only thousands)
2. Low vocabulary coverage: Low-resource languages have small proportion in mBERT's 110K vocabulary
3. Language drift: High-resource languages dominate training, low-resource language representations degrade

Improvement directions:

1. Specialized vocabulary: Design separate subword vocabulary for low-resource languages
2. Data augmentation: Augment low-resource languages with high-resource language translations
3. Adaptive pre-training: Continue pre-training on low-resource languages

Bias in Multilingual Models

Problem: Multilingual models exhibit language bias¹⁴:

• English usually performs best (most pre-training data)
• Low-resource language performance drops significantly
• Culture-related tasks (e.g., sentiment classification) show large cross-lingual differences

Measurement: Inter-language performance variance:

Mitigation strategies:

1. Balanced sampling: Increase sampling probability for low-resource languages
2. Adversarial training: Minimize language discriminator accuracy
3. Multi-task learning: Add language identification task to force learning language differences

Frequently Asked Questions

Q1: mBERT doesn't use parallel corpus, why does cross-lingual work?

Key factors:

1. Anchor Words:
   • Numbers: 1, 2, 3 (shared across all languages)
   • Punctuation: , . ! ?
   • English loanwords: OK, Internet, COVID
2. Deep parameter sharing:
   • Forces different languages through same Transformer layers
   • Model forced to learn language-agnostic features
3. WordPiece decomposition:
   • Decomposes words into subword units
   • Increases cross-lingual vocabulary overlap

Experimental evidence¹⁵: Removing anchor words, cross-lingual performance drops 15-20%.

Q2: How to choose source language?

Empirical rules:

1. Data volume priority: Choose language with most labeled data (usually English)
2. Language family similarity: If target is French, Spanish is better than Chinese
3. Multi-source strategy: Combine multiple source languages (English+German → French)

Experiment: On XNLI, different source languages to French zero-shot accuracy:

Source Language	Accuracy
English	78.3
Spanish	81.2
German	79.7
Chinese	71.5

Spanish best (both Romance languages).

Q3: Translate-train vs zero-shot transfer, which is better?

Trade-offs:

Dimension	Translate-Train	Zero-Shot Transfer
Performance	Higher (+2-5%)	Lower
Cost	High (needs translation)	Low (no translation)
Inference latency	Low	Low
Translation quality dependency	Yes	No

Recommendation: - High-resource languages: Zero-shot transfer (translation quality high but unnecessary) - Low-resource languages: Translate-train (compensate for model weakness on low-resource languages)

Q4: What makes XLM-R better than mBERT?

Core improvements:

1. Larger scale:
   • mBERT: Few GB Wikipedia
   • XLM-R: 2.5TB CommonCrawl
2. More balanced language sampling:
   • mBERT: High-resource languages dominate
   • XLM-R:(mitigates imbalance)
3. More parameters:
   • mBERT: 110M
   • XLM-R: 550M

Performance improvement: On XNLI, XLM-R averages 10% higher than mBERT.

Q5: How to handle code-switching?

Strategies:

1. Data augmentation:
   • Randomly replace words with translations in other languages
   • Maintain syntactic structure
2. Multilingual pre-training:
   • Collect real code-switching data (e.g., Twitter)
   • Mix into pre-training corpus
3. Language tags:
   • Add language ID for each token
   • Model learns language switching patterns

Effect: On code-switching benchmark (GLUECoS), adding code-switching data augmentation improves accuracy by 5-10%.

Q6: Can cross-lingual transfer be used for generation tasks?

Yes! Common applications:

1. Machine translation: Source language training, target language generation
2. Cross-lingual summarization: English document → Chinese summary
3. Cross-lingual QA: Chinese question → English answer → translate back to Chinese

Models: mT5, mBART and other encoder-decoder models.

Challenges: - High fluency requirements for generation - Need to handle word order differences - Cultural adaptation (e.g., idiom translation)

Q7: Do multilingual models "forget" high-resource languages?

Yes! Phenomenon called "Language Competition"¹⁶.

Manifestation: - After fine-tuning on low-resource languages, English performance drops - Adding new language pre-training, old language performance degrades

Mitigation: - Multi-task learning: Optimize all languages simultaneously - Regularization: Methods like EWC (see Chapter 10 continual learning) - Language adapters: Independent parameters for each language

Q8: How to evaluate cross-lingual transfer quality?

Standard benchmarks:

1. XNLI: Cross-lingual natural language inference (15 languages)
2. XTREME: Cross-lingual multi-task benchmark (40 languages, 9 tasks)
3. MLQA: Multilingual question answering (7 languages)
4. TyDiQA: Typologically diverse QA (11 languages, covering low-resource languages)

Evaluation metrics: - Zero-shot accuracy - Transfer gap - Inter-language performance variance

Q9: What are theoretical limits of cross-lingual transfer?

Information theory perspective¹⁷:

Upper bound of cross-lingual transfer limited by Mutual Information between languages:

Intuition: More similar languages have higher mutual information, higher transfer upper bound.

Empirical: - Same language family:, transfer gap <5% - Different language family:, transfer gap >15%

Breakthrough directions: - Use intermediate language (Pivot Language) - Multilingual pre-training increases language commonality

Q10: How to add cross-lingual support for new language?

Process:

1. Collect monolingual data: Wikipedia, news, social media
2. Expand vocabulary: Add subwords for new language
3. Adaptive pre-training: Continue MLM on new language
4. Zero-shot evaluation: Test on downstream tasks
5. Few-shot fine-tuning: Fine-tune with small labeled data if available

Case study: Adding Swahili support:

Step	Zero-Shot Accuracy
Baseline (XLM-R)	68.4
+ Adaptive pre-training	72.1 (+3.7)
+ 100-sample fine-tuning	76.8 (+4.7)

Q11: What is inference overhead of multilingual models?

Comparison:

Model	Parameters	Inference Time (Relative)
BERT-base	110M	1.0x
mBERT	110M	1.0x (same)
XLM-R-base	270M	1.5x
XLM-R-large	550M	3.0x

Conclusion: Multilingual model inference overhead mainly depends on model size, not number of languages.

Optimization: - Model distillation: Distill XLM-R to smaller model - Language-specific pruning: Keep only target language vocabulary

Q12: Future directions for cross-lingual research?

Hot topics:

1. Extremely low-resource languages:
   • 7000+ languages on Earth, most without digital resources
   • Leverage linguistic knowledge (grammar, phonology)
2. Multimodal cross-lingual:
   • Image-text cross-lingual alignment
   • Video-text cross-lingual understanding
3. Cross-lingual commonsense reasoning:
   • Cultural differences in commonsense knowledge
   • How to transfer culture-related knowledge?
4. Interpretability:
   • Why does mBERT work cross-lingually?
   • Geometric structure of multilingual representations
5. Efficient multilingual models:
   • Parameter sharing vs language-specific parameters
   • Sparse activation (only activate relevant language parameters)

Summary

This article comprehensively introduced cross-lingual transfer techniques:

1. Problem definition: Zero-shot, few-shot, multi-source language transfer
2. Mathematical principles: Shared semantic space, bilingual word embedding alignment, language universals theory
3. Multilingual pre-training: Architecture and comparison of mBERT, XLM-R, mT5
4. Transfer strategies: Direct transfer, translate-train, translate-test, ensemble methods
5. Prompt learning: Multilingual prompt templates, automatic search, language-agnostic continuous prompts
6. Code-switching: Data augmentation, language mixing, adaptive pre-training
7. Complete code: 280+ lines implementing cross-lingual text classification from scratch
8. Challenges and frontiers: Language differences, low-resource languages, model bias, theoretical limits

Cross-lingual transfer enables AI to benefit 7 billion people globally, breaking down language barriers. In the next chapter, we will explore transfer learning applications in industry and best practices, seeing how to transform theory into productivity.

References

---

1. Conneau, A., Lample, G., Ranzato, M. A., et al. (2018). Word translation without parallel data. ICLR.↩︎

2. Conneau, A., & Lample, G. (2019). Cross-lingual language model pretraining. NeurIPS.↩︎

3. Artetxe, M., & Schwenk, H. (2019). Massively multilingual sentence embeddings for zero-shot cross-lingual transfer and beyond. TACL.↩︎

4. Devlin, J., Chang, M. W., Lee, K., & Toutanova, K. (2019). BERT: Pre-training of deep bidirectional transformers for language understanding. NAACL.↩︎

5. Pires, T., Schlinger, E., & Garrette, D. (2019). How multilingual is multilingual BERT? ACL.↩︎

6. Conneau, A., Khandelwal, K., Goyal, N., et al. (2020). Unsupervised cross-lingual representation learning at scale. ACL.↩︎

7. Xue, L., Constant, N., Roberts, A., et al. (2021). mT5: A massively multilingual pre-trained text-to-text transformer. NAACL.↩︎

8. Jiang, Z., Xu, F. F., Araki, J., & Neubig, G. (2020). How can we know what language models know? TACL.↩︎

9. Shin, T., Razeghi, Y., Logan IV, R. L., et al. (2020). AutoPrompt: Eliciting knowledge from language models with automatically generated prompts. EMNLP.↩︎

10. Wu, S., & Dredze, M. (2020). Are all languages created equal in multilingual BERT? RepL4NLP.↩︎

11. Winata, G. I., Madotto, A., Wu, Z., & Fung, P. (2019). Code-switching BERT: A task-agnostic language model for code-switching. arXiv:1908.05075.↩︎

12. Alabi, J., Amponsah-Kaakyire, K., Adelani, D., & Eskenazi, M. (2020). Massive vs. curated embeddings for low-resourced languages: the case of Yor ù b á and Twi. LREC.↩︎

13. Hu, J., Ruder, S., Siddhant, A., et al. (2020). XTREME: A massively multilingual multi-task benchmark for evaluating cross-lingual generalisation. ICML.↩︎

14. Lauscher, A., Ravishankar, V., Vulic, I., & Glavas, G. (2020). From zero to hero: On the limitations of zero-shot language transfer with multilingual transformers. EMNLP.↩︎

15. Pires, T., Schlinger, E., & Garrette, D. (2019). How multilingual is multilingual BERT? ACL.↩︎

16. Artetxe, M., Ruder, S., & Yogatama, D. (2020). On the cross-lingual transferability of monolingual representations. ACL.↩︎

17. Zhao, W., Eger, S., Bjerva, J., & Augenstein, I. (2021). Inducing language-agnostic multilingual representations. ACL.↩︎