Recommendation Systems (8): Knowledge Graph-Enhanced Recommendation
Chen Kai BOSS
  2026-02-03 23:11:11 2026-02-03 23:11  8.8k Words  54 Mins  

permalink: "en/recommendation-systems-8-knowledge-graph/" date: 2024-06-06 16:00:00 tags: - Recommendation Systems - Knowledge Graph - KG-enhanced categories: Recommendation Systems mathjax: true --- When you search for "The Dark Knight" on a movie recommendation platform, the system doesn't just know you watched it — it understands that Christian Bale played Batman, Christopher Nolan directed it, it's part of the Batman trilogy, and it's similar to other superhero films. This rich semantic understanding comes from knowledge graphs, structured representations that encode entities (movies, actors, directors) and their relationships (acted_in, directed_by, similar_to) as a graph. Knowledge graph-enhanced recommendation systems leverage these semantic relationships to provide more accurate, explainable, and diverse recommendations, especially for cold-start items and users with sparse interaction histories.

Knowledge graphs transform recommendation from pure pattern matching to semantic reasoning. Traditional collaborative filtering methods struggle when items have few interactions, but knowledge graphs provide rich auxiliary information: if a new movie shares actors or directors with movies you've enjoyed, the system can confidently recommend it even without historical interaction data. This article provides a comprehensive exploration of knowledge graph-enhanced recommendation systems, covering knowledge graph fundamentals, their role in recommendation, propagation-based methods like RippleNet, graph convolutional approaches (KGCN), attention mechanisms (KGAT, HKGAT), collaborative knowledge embedding (CKE), recent advances in RecKG, and practical implementations with 10+ code examples and detailed Q&A sections.

Foundations of Knowledge Graphs

What Is a Knowledge Graph?

A knowledge graph (KG) is a structured representation of knowledge as a graph, where nodes represent entities (objects, concepts, events) and edges represent relationships between entities. Formally, a knowledge graph is defined as\(\mathcal{G} = \{(h, r, t)\}\), where:

-\(h \in\)\(is the head entity -\)r \(\mathcal{R}\)is the relation type -\(t \in\)\(is the tail entity -\)\(is the set of entities -\)$is the set of relation types

Each triple\((h, r, t)\)represents a fact: "head entity\(h\)has relation\(r\)with tail entity\(t\)." For example, in a movie knowledge graph: -\((TheDarkKnight, directed_by, ChristopherNolan)\) -\((TheDarkKnight, starred_in, ChristianBale)\) -\((TheDarkKnight, genre, Action)\) -\((ChristianBale, acted_in, ThePrestige)\)

Knowledge Graph Structure

Knowledge graphs can be homogeneous (single entity/relation type) or heterogeneous (multiple entity/relation types). Recommendation systems typically use heterogeneous knowledge graphs that include:

Entity Types: - Users:\(U = \{u_1, u_2, \dots, u_m\}\) - Items:\(I = \{i_1, i_2, \dots, i_n\}\) - Attributes:\(A = \{a_1, a_2, \dots\}\) (genres, actors, directors, etc.)

Relation Types: - User-item interactions:\((u, interact, i)\) - Item-attribute relationships:\((i, has_genre, g)\),\((i, starred_by, a)\) - Attribute-attribute relationships:\((a_1, collaborated_with, a_2)\)

Knowledge Graph Representation

Knowledge graphs are typically stored as: 1. Triple stores: RDF (Resource Description Framework) format 2. Graph databases: Neo4j, Amazon Neptune, ArangoDB 3. Adjacency lists: Efficient for graph algorithms

For a knowledge graph with\(|\mathcal{E}|\)entities and\(|\mathcal{R}|\)relations, we can represent it as:

  • Adjacency matrix:\(A \in \mathbb{R}^{|\mathcal{E}| \times |\mathcal{E}|}\)where\(A_{ij} = 1\)if entities\(i\)and\(j\)are connected
  • Relation-specific adjacency:\(A_r \in \mathbb{R}^{|\mathcal{E}| \times |\mathcal{E}|}\)for each relation\(r \in\)$
  • Triple list: List of\((h, r, t)\)tuples

Knowledge Graph Embedding

Before using knowledge graphs in recommendation, we need to embed entities and relations into dense vector spaces. Common methods include:

TransE: Learns embeddings such that\(\mathbf{h} + \mathbf{r} \approx \mathbf{t}\)for valid triples\((h, r, t)\):\[\mathcal{L} = \sum_{(h,r,t) \in \mathcal{G}} \sum_{(h',r,t') \notin \mathcal{G}} [\gamma + ||\mathbf{h} + \mathbf{r} - \mathbf{t}||_2 - ||\mathbf{h}' + \mathbf{r} - \mathbf{t}'||_2]_+\]where\([\cdot]_+ = \max(0, \cdot)\)and\(\gamma\)is a margin.

TransR: Projects entities into relation-specific spaces:\[\mathbf{h}_r = \mathbf{h}M_r, \quad \mathbf{t}_r = \mathbf{t}M_r\]where\(M_r \in \mathbb{R}^{d \times d_r}\)is a relation-specific projection matrix.

DistMult: Uses bilinear scoring:\[f(h, r, t) = \mathbf{h}^T \text{diag}(\mathbf{r}) \mathbf{t}\]

ComplEx: Extends DistMult to complex space for asymmetric relations:\[f(h, r, t) = \text{Re}(\mathbf{h}^T \text{diag}(\mathbf{r}) \bar{\mathbf{t}})\]

Implementation: Knowledge Graph Construction

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
import numpy as np
import torch
import torch.nn as nn
from collections import defaultdict
from typing import List, Tuple, Dict

class KnowledgeGraph:
    """
    Simple knowledge graph representation for recommendation systems.
    """
    def __init__(self):
        self.entities = {}  # entity_name -> entity_id
        self.relations = {}  # relation_name -> relation_id
        self.triples = []  # List of (h_id, r_id, t_id)
        self.entity_adj = defaultdict(list)  # entity_id -> [(r_id, t_id), ...]
        self.relation_adj = defaultdict(lambda: defaultdict(list))  # r_id -> {h_id: [t_id, ...]}
        
    def add_entity(self, entity_name: str) -> int:
        """Add entity and return its ID."""
        if entity_name not in self.entities:
            entity_id = len(self.entities)
            self.entities[entity_name] = entity_id
            return entity_id
        return self.entities[entity_name]
    
    def add_relation(self, relation_name: str) -> int:
        """Add relation and return its ID."""
        if relation_name not in self.relations:
            relation_id = len(self.relations)
            self.relations[relation_name] = relation_id
            return relation_id
        return self.relations[relation_name]
    
    def add_triple(self, head: str, relation: str, tail: str):
        """Add a triple (head, relation, tail) to the knowledge graph."""
        h_id = self.add_entity(head)
        r_id = self.add_relation(relation)
        t_id = self.add_entity(tail)
        
        self.triples.append((h_id, r_id, t_id))
        self.entity_adj[h_id].append((r_id, t_id))
        self.relation_adj[r_id][h_id].append(t_id)
    
    def get_neighbors(self, entity_id: int) -> List[Tuple[int, int]]:
        """Get all neighbors of an entity: [(relation_id, tail_id), ...]."""
        return self.entity_adj[entity_id]
    
    def get_relation_triples(self, relation_id: int) -> List[Tuple[int, int]]:
        """Get all triples for a specific relation: [(head_id, tail_id), ...]."""
        triples = []
        for h_id, t_ids in self.relation_adj[relation_id].items():
            for t_id in t_ids:
                triples.append((h_id, t_id))
        return triples
    
    def get_entity_embedding_matrix(self, embedding_dim: int) -> torch.Tensor:
        """Initialize random entity embeddings."""
        num_entities = len(self.entities)
        return nn.Parameter(torch.randn(num_entities, embedding_dim))
    
    def get_relation_embedding_matrix(self, embedding_dim: int) -> torch.Tensor:
        """Initialize random relation embeddings."""
        num_relations = len(self.relations)
        return nn.Parameter(torch.randn(num_relations, embedding_dim))

# Example: Build a movie knowledge graph
kg = KnowledgeGraph()

# Add movie entities
kg.add_triple("TheDarkKnight", "directed_by", "ChristopherNolan")
kg.add_triple("TheDarkKnight", "starred_in", "ChristianBale")
kg.add_triple("TheDarkKnight", "has_genre", "Action")
kg.add_triple("TheDarkKnight", "has_genre", "Crime")
kg.add_triple("Inception", "directed_by", "ChristopherNolan")
kg.add_triple("Inception", "starred_in", "LeonardoDiCaprio")
kg.add_triple("Inception", "has_genre", "Sci-Fi")
kg.add_triple("ThePrestige", "directed_by", "ChristopherNolan")
kg.add_triple("ThePrestige", "starred_in", "ChristianBale")
kg.add_triple("ThePrestige", "has_genre", "Drama")

# Get neighbors
dark_knight_id = kg.entities["TheDarkKnight"]
neighbors = kg.get_neighbors(dark_knight_id)
print(f"Neighbors of TheDarkKnight: {neighbors}")

The Role of Knowledge Graphs in Recommendation

Why Knowledge Graphs Help Recommendation

Knowledge graphs address several fundamental limitations of traditional recommendation methods:

Cold Start Problem: New items with no interaction history can't be recommended by collaborative filtering. Knowledge graphs provide rich semantic information: if a new movie shares actors, directors, or genres with movies a user likes, we can recommend it confidently.

Data Sparsity: User-item interaction matrices are extremely sparse. Knowledge graphs add dense semantic connections, enabling the system to find relationships even when direct interactions are missing.

Explainability: Knowledge graphs provide interpretable paths for recommendations. Instead of "users who liked this also liked that," we can say "we recommend this because it's directed by the same director as movies you've enjoyed."

Diversity: Knowledge graphs help discover diverse recommendations by exploring different relation paths, avoiding the filter bubble effect.

Knowledge Graph-Enhanced Recommendation Pipeline

The typical pipeline for KG-enhanced recommendation:

  1. Knowledge Graph Construction: Build KG from structured data (DBpedia, Wikidata, domain-specific databases) or extract from unstructured text
  2. Entity Alignment: Link items/users in the recommendation system to entities in the knowledge graph
  3. Embedding Learning: Learn dense representations of entities and relations
  4. Recommendation: Use KG embeddings to enhance user/item representations and generate recommendations

Types of KG-Enhanced Recommendation Methods

Propagation-Based Methods: Propagate user preferences through the knowledge graph (RippleNet)

Graph Convolutional Methods: Apply graph convolutional networks to the knowledge graph (KGCN)

Attention-Based Methods: Use attention mechanisms to weight different relation paths (KGAT, HKGAT)

Embedding-Based Methods: Learn joint embeddings of users, items, and KG entities (CKE)

Hybrid Methods: Combine multiple approaches (RecKG)

RippleNet: Preference Propagation

RippleNet Architecture

RippleNet, introduced by Wang et al. in CIKM 2018, propagates user preferences through the knowledge graph like ripples in water. When a user interacts with an item, this preference "ripples" outward through the KG, activating related entities at different hops.

Key Idea: User preferences propagate along relation paths in the knowledge graph. Entities closer to the user's historical items receive stronger signals.

RippleNet Algorithm

Given a user\(u\)with historical items\(V_u = \{v_1, v_2, \dots\}\), RippleNet:

  1. Initializes the user's preference set as the historical items:\(\mathcal{S}_u^0 = V_u\)

  2. Propagates preferences through\(H\)hops:

    • At hop\(h\), finds entities connected to\(\mathcal{S}_u^{h-1}\)via relations
    • Aggregates these entities to form\(\mathcal{S}_u^h\)

  3. Aggregates preferences from all hops to form the user embedding

  4. Predicts user-item interaction probability

Mathematical Formulation

At hop\(h\), for each historical item\(v \in V_u\), RippleNet finds related entities:\[\mathcal{S}_u^h = \{t | (h, r, t) \in \mathcal{G}, h \in \mathcal{S}_u^{h-1}}\]The relevance of entity\(t\)at hop\(h\)is computed as:\[p_i^h = \text{softmax}(\mathbf{v}^T \mathbf{R}_i \mathbf{t})\]where\(\mathbf{v}\)is the item embedding,\(\mathbf{R}_i\)is the relation embedding matrix, and\(\mathbf{t}\)is the entity embedding.

The user's preference at hop\(h\)is:\[\mathbf{o}_u^h = \sum_{(h,r,t) \in \mathcal{S}_u^h} p_i^h \mathbf{t}\]The final user embedding aggregates all hops:\[\mathbf{u} = \sum_{h=1}^{H} \alpha_h \mathbf{o}_u^h + \mathbf{u}_0\]where\(\alpha_h\)are learnable weights and\(\mathbf{u}_0\)is the base user embedding.

Implementation: RippleNet

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
import torch
import torch.nn as nn
import torch.nn.functional as F
from typing import List, Dict, Set

class RippleNet(nn.Module):
    """
    RippleNet: Propagating User Preferences on the Knowledge Graph for Recommender Systems.
    """
    def __init__(self, num_users, num_items, num_entities, num_relations, 
                 embedding_dim=64, n_hop=2, n_memory=32, kg_adj=None):
        super(RippleNet, self).__init__()
        self.num_users = num_users
        self.num_items = num_items
        self.num_entities = num_entities
        self.num_relations = num_relations
        self.embedding_dim = embedding_dim
        self.n_hop = n_hop
        self.n_memory = n_memory
        self.kg_adj = kg_adj  # Knowledge graph adjacency: {entity_id: [(relation_id, tail_id), ...]}
        
        # Embeddings
        self.user_embeddings = nn.Embedding(num_users, embedding_dim)
        self.item_embeddings = nn.Embedding(num_items, embedding_dim)
        self.entity_embeddings = nn.Embedding(num_entities, embedding_dim)
        self.relation_embeddings = nn.Embedding(num_relations, embedding_dim)
        
        # Hop weights
        self.hop_weights = nn.Parameter(torch.ones(n_hop + 1) / (n_hop + 1))
        
        # Initialize embeddings
        nn.init.xavier_uniform_(self.user_embeddings.weight)
        nn.init.xavier_uniform_(self.item_embeddings.weight)
        nn.init.xavier_uniform_(self.entity_embeddings.weight)
        nn.init.xavier_uniform_(self.relation_embeddings.weight)
    
    def forward(self, user_ids, item_ids, ripple_sets):
        """
        Args:
            user_ids: (batch_size,) - user indices
            item_ids: (batch_size,) - item indices
            ripple_sets: List of ripple sets for each user
                         Each ripple set: List of n_hop sets
                         Each set: List of (relation_id, tail_id) tuples
        """
        # Get base embeddings
        user_emb = self.user_embeddings(user_ids)  # (batch_size, embedding_dim)
        item_emb = self.item_embeddings(item_ids)  # (batch_size, embedding_dim)
        
        # Propagate preferences through KG
        user_emb_enhanced = self._ripple_propagate(user_emb, item_emb, ripple_sets)
        
        # Predict interaction probability
        scores = torch.sum(user_emb_enhanced * item_emb, dim=1)  # (batch_size,)
        return scores
    
    def _ripple_propagate(self, user_emb, item_emb, ripple_sets):
        """
        Propagate user preferences through knowledge graph.
        """
        batch_size = user_emb.size(0)
        memories = [user_emb]  # Start with base user embedding
        
        for hop in range(self.n_hop):
            # Get ripple set for this hop
            hop_memories = []
            
            for i in range(batch_size):
                ripple_set = ripple_sets[i][hop] if hop < len(ripple_sets[i]) else []
                
                if len(ripple_set) == 0:
                    # No neighbors at this hop, use zero vector
                    hop_memories.append(torch.zeros(self.embedding_dim))
                    continue
                
                # Sample n_memory entities if too many
                if len(ripple_set) > self.n_memory:
                    ripple_set = ripple_set[:self.n_memory]
                
                # Get embeddings
                relations = torch.LongTensor([r for r, t in ripple_set])
                tails = torch.LongTensor([t for r, t in ripple_set])
                
                relation_emb = self.relation_embeddings(relations)  # (n_mem, embedding_dim)
                tail_emb = self.entity_embeddings(tails)  # (n_mem, embedding_dim)
                
                # Compute relevance scores
                # p = softmax(v^T R t)
                item_emb_i = item_emb[i].unsqueeze(0)  # (1, embedding_dim)
                scores = torch.sum(item_emb_i * relation_emb * tail_emb, dim=1)  # (n_mem,)
                probs = F.softmax(scores, dim=0)  # (n_mem,)
                
                # Weighted aggregation
                hop_memory = torch.sum(probs.unsqueeze(1) * tail_emb, dim=0)  # (embedding_dim,)
                hop_memories.append(hop_memory)
            
            hop_memories = torch.stack(hop_memories)  # (batch_size, embedding_dim)
            memories.append(hop_memories)
        
        # Aggregate all hops
        memories = torch.stack(memories, dim=1)  # (batch_size, n_hop+1, embedding_dim)
        weights = F.softmax(self.hop_weights, dim=0)  # (n_hop+1,)
        user_emb_enhanced = torch.sum(weights.unsqueeze(0).unsqueeze(2) * memories, dim=1)  # (batch_size, embedding_dim)
        
        return user_emb_enhanced
    
    def get_ripple_sets(self, user_id, user_items, kg_adj, max_hops=2):
        """
        Generate ripple sets for a user.
        
        Args:
            user_id: User ID
            user_items: List of item IDs the user has interacted with
            kg_adj: Knowledge graph adjacency
            max_hops: Maximum number of hops
        
        Returns:
            List of ripple sets: [hop_0_set, hop_1_set, ...]
        """
        ripple_sets = []
        current_set = set(user_items)  # Start with user's items
        
        for hop in range(max_hops):
            next_set = []
            for entity_id in current_set:
                if entity_id in kg_adj:
                    for relation_id, tail_id in kg_adj[entity_id]:
                        next_set.append((relation_id, tail_id))
            
            ripple_sets.append(next_set)
            current_set = set([t for _, t in next_set])
        
        return ripple_sets

# Example usage
if __name__ == "__main__":
    num_users = 1000
    num_items = 500
    num_entities = 2000  # Includes items + attributes
    num_relations = 10
    embedding_dim = 64
    
    # Build knowledge graph adjacency
    kg_adj = {
        0: [(0, 100), (1, 101)],  # Item 0 connected to entities 100, 101
        1: [(0, 102), (2, 103)],
    }
    
    model = RippleNet(
        num_users=num_users,
        num_items=num_items,
        num_entities=num_entities,
        num_relations=num_relations,
        embedding_dim=embedding_dim,
        n_hop=2,
        n_memory=32,
        kg_adj=kg_adj
    )
    
    # Generate ripple sets for a user
    user_items = [0, 1]  # User interacted with items 0 and 1
    ripple_sets = model.get_ripple_sets(0, user_items, kg_adj, max_hops=2)
    
    # Forward pass
    user_ids = torch.LongTensor([0])
    item_ids = torch.LongTensor([2])
    batch_ripple_sets = [ripple_sets]
    
    scores = model(user_ids, item_ids, batch_ripple_sets)
    print(f"Prediction score: {scores.item():.4f}")

KGCN: Knowledge Graph Convolutional Networks

KGCN Architecture

KGCN (Knowledge Graph Convolutional Network), introduced by Wang et al. in WWW 2019, applies graph convolutional networks directly to the knowledge graph to learn item embeddings. Unlike RippleNet which propagates user preferences, KGCN learns item representations by aggregating information from neighboring entities in the KG.

Key Idea: Items are represented by aggregating features from their neighbors in the knowledge graph. Different relations contribute differently to the final representation.

KGCN Algorithm

For an item\(i\)in the knowledge graph, KGCN:

  1. Samples neighbors:$_i = {(r, e) | (i, r, e) } $

  2. Aggregates neighbor embeddings with relation-aware weights

  3. Updates item embedding by combining its own embedding with aggregated neighbors

  4. Repeats for multiple layers to capture multi-hop relationships

Mathematical Formulation

At layer\(l\), for item\(i\), KGCN aggregates neighbors:\[\mathbf{e}_{\mathcal{N}_i}^l = \sum_{(r,e) \in \mathcal{N}_i} \pi_r^l(i, e) \mathbf{e}_e^{l-1}\]where\(\pi_r^l(i, e)\)is the attention weight for relation\(r\)and neighbor\(e\):\[\pi_r^l(i, e) = \frac{\exp(\mathbf{W}_r^l \mathbf{e}_i^{l-1} + \mathbf{b}_r^l)}{\sum_{(r',e') \in \mathcal{N}_i} \exp(\mathbf{W}_{r'}^l \mathbf{e}_i^{l-1} + \mathbf{b}_{r'}^l)}\]The item embedding is updated as:\[\mathbf{e}_i^l = \sigma(\mathbf{W}^l [\mathbf{e}_i^{l-1} || \mathbf{e}_{\mathcal{N}_i}^l] + \mathbf{b}^l)\]where\(||\)denotes concatenation and\(\sigma\)is an activation function.

Implementation: KGCN

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
import torch
import torch.nn as nn
import torch.nn.functional as F
from torch_geometric.nn import MessagePassing
from torch_geometric.utils import add_self_loops, degree

class KGCNLayer(nn.Module):
    """
    Single layer of Knowledge Graph Convolutional Network.
    """
    def __init__(self, embedding_dim, num_relations, aggregator='sum'):
        super(KGCNLayer, self).__init__()
        self.embedding_dim = embedding_dim
        self.num_relations = num_relations
        self.aggregator = aggregator
        
        # Relation-specific transformation matrices
        self.relation_transforms = nn.ModuleList([
            nn.Linear(embedding_dim, embedding_dim, bias=False)
            for _ in range(num_relations)
        ])
        
        # Attention weights
        self.attention_weights = nn.ModuleList([
            nn.Linear(embedding_dim, 1)
            for _ in range(num_relations)
        ])
        
        # Aggregation function
        self.aggregate = nn.Linear(embedding_dim * 2, embedding_dim)
    
    def forward(self, item_embeddings, neighbors, relations):
        """
        Args:
            item_embeddings: (num_items, embedding_dim) - current item embeddings
            neighbors: List of neighbor lists for each item
                      neighbors[i] = [(relation_id, neighbor_id), ...]
            relations: Relation embeddings (num_relations, embedding_dim)
        """
        num_items = item_embeddings.size(0)
        aggregated = []
        
        for i in range(num_items):
            if len(neighbors[i]) == 0:
                # No neighbors, use zero vector
                aggregated.append(torch.zeros(self.embedding_dim))
                continue
            
            # Get neighbor embeddings and relations
            neighbor_embs = []
            attention_scores = []
            
            for r_id, n_id in neighbors[i]:
                neighbor_emb = item_embeddings[n_id]  # (embedding_dim,)
                
                # Transform neighbor embedding with relation
                transformed = self.relation_transforms[r_id](neighbor_emb)  # (embedding_dim,)
                neighbor_embs.append(transformed)
                
                # Compute attention score
                # Attention based on item and relation compatibility
                item_emb = item_embeddings[i]  # (embedding_dim,)
                relation_emb = relations[r_id]  # (embedding_dim,)
                combined = item_emb + relation_emb  # (embedding_dim,)
                score = self.attention_weights[r_id](combined)  # (1,)
                attention_scores.append(score)
            
            neighbor_embs = torch.stack(neighbor_embs)  # (num_neighbors, embedding_dim)
            attention_scores = torch.stack(attention_scores).squeeze()  # (num_neighbors,)
            
            # Apply attention
            attention_weights = F.softmax(attention_scores, dim=0)  # (num_neighbors,)
            aggregated_emb = torch.sum(attention_weights.unsqueeze(1) * neighbor_embs, dim=0)  # (embedding_dim,)
            aggregated.append(aggregated_emb)
        
        aggregated = torch.stack(aggregated)  # (num_items, embedding_dim)
        
        # Combine with original embeddings
        combined = torch.cat([item_embeddings, aggregated], dim=1)  # (num_items, embedding_dim * 2)
        output = self.aggregate(combined)  # (num_items, embedding_dim)
        
        return output

class KGCN(nn.Module):
    """
    Knowledge Graph Convolutional Network for Recommendation.
    """
    def __init__(self, num_users, num_items, num_entities, num_relations,
                 embedding_dim=64, n_layers=2, aggregator='sum'):
        super(KGCN, self).__init__()
        self.num_users = num_users
        self.num_items = num_items
        self.num_entities = num_entities
        self.num_relations = num_relations
        self.embedding_dim = embedding_dim
        self.n_layers = n_layers
        
        # Embeddings
        self.user_embeddings = nn.Embedding(num_users, embedding_dim)
        self.item_embeddings = nn.Embedding(num_items, embedding_dim)
        self.entity_embeddings = nn.Embedding(num_entities, embedding_dim)
        self.relation_embeddings = nn.Embedding(num_relations, embedding_dim)
        
        # KGCN layers
        self.kgcn_layers = nn.ModuleList([
            KGCNLayer(embedding_dim, num_relations, aggregator)
            for _ in range(n_layers)
        ])
        
        # Initialize embeddings
        nn.init.xavier_uniform_(self.user_embeddings.weight)
        nn.init.xavier_uniform_(self.item_embeddings.weight)
        nn.init.xavier_uniform_(self.entity_embeddings.weight)
        nn.init.xavier_uniform_(self.relation_embeddings.weight)
    
    def forward(self, user_ids, item_ids, item_neighbors):
        """
        Args:
            user_ids: (batch_size,) - user indices
            item_ids: (batch_size,) - item indices
            item_neighbors: List of neighbor lists for each layer
                           item_neighbors[l][i] = [(relation_id, neighbor_id), ...]
        """
        # Get base embeddings
        user_emb = self.user_embeddings(user_ids)  # (batch_size, embedding_dim)
        
        # Initialize item embeddings (use entity embeddings for items)
        item_emb = self.entity_embeddings(item_ids)  # (batch_size, embedding_dim)
        
        # Apply KGCN layers
        relation_emb = self.relation_embeddings.weight  # (num_relations, embedding_dim)
        
        # For simplicity, we'll use item_neighbors for the first layer
        # In practice, you'd propagate through multiple layers
        current_emb = self.entity_embeddings.weight[:self.num_items]  # (num_items, embedding_dim)
        
        for layer_idx, kgcn_layer in enumerate(self.kgcn_layers):
            if layer_idx < len(item_neighbors):
                current_emb = kgcn_layer(current_emb, item_neighbors[layer_idx], relation_emb)
                current_emb = F.relu(current_emb)
        
        # Get item embeddings for batch
        item_emb_enhanced = current_emb[item_ids]  # (batch_size, embedding_dim)
        
        # Predict interaction
        scores = torch.sum(user_emb * item_emb_enhanced, dim=1)  # (batch_size,)
        return scores

# Example usage
if __name__ == "__main__":
    num_users = 1000
    num_items = 500
    num_entities = 2000
    num_relations = 10
    embedding_dim = 64
    
    model = KGCN(
        num_users=num_users,
        num_items=num_items,
        num_entities=num_entities,
        num_relations=num_relations,
        embedding_dim=embedding_dim,
        n_layers=2
    )
    
    # Example neighbors: item 0 has neighbors via relations
    item_neighbors = [
        {0: [(0, 100), (1, 101)], 1: [(0, 102)]},  # Layer 0
        {100: [(2, 200)], 101: [(2, 201)]}  # Layer 1
    ]
    
    user_ids = torch.LongTensor([0, 1])
    item_ids = torch.LongTensor([0, 1])
    
    scores = model(user_ids, item_ids, item_neighbors)
    print(f"Prediction scores: {scores}")

KGAT: Knowledge Graph Attention Network

KGAT Architecture

KGAT (Knowledge Graph Attention Network), introduced by Wang et al. in KDD 2019, extends KGCN by using attention mechanisms to learn relation-aware item embeddings. KGAT explicitly models high-order connectivity in the knowledge graph through an attention-based aggregator.

Key Innovation: KGAT uses attention to automatically learn the importance of different relations and entities when aggregating information in the knowledge graph.

KGAT Algorithm

KGAT builds a collaborative knowledge graph (CKG) that combines user-item interactions with the knowledge graph:\[\mathcal{G}_1 = \{(u, interact, i) | u \in \mathcal{U}, i \in \mathcal{I}}\] \[\mathcal{G}_2 = \{(h, r, t) | h, t \in \mathcal{E}, r \in \mathcal{R}}\] \[\mathcal{G} = \mathcal{G}_1 \cup \mathcal{G}_2\]For an entity\(e\)(user or item), KGAT aggregates neighbors with attention:\[\mathbf{e}_{\mathcal{N}_e} = \sum_{(r,e') \in \mathcal{N}_e} \pi(e, r, e') \mathbf{e}_{e'}\]where the attention weight is:\[\pi(e, r, e') = \frac{\exp(\text{LeakyReLU}(\mathbf{a}^T [\mathbf{e}_e || \mathbf{e}_r || \mathbf{e}_{e'}]))}{\sum_{(r',e'') \in \mathcal{N}_e} \exp(\text{LeakyReLU}(\mathbf{a}^T [\mathbf{e}_e || \mathbf{e}_{r'} || \mathbf{e}_{e''}]))}\]The entity embedding is updated as:\[\mathbf{e}_e^{(l)} = \sigma(\mathbf{W}^{(l)} [\mathbf{e}_e^{(l-1)} || \mathbf{e}_{\mathcal{N}_e}^{(l-1)}] + \mathbf{b}^{(l)})\]

Multi-Head Attention

KGAT uses multi-head attention to capture different aspects:\[\mathbf{e}_{\mathcal{N}_e}^{(h)} = \sum_{(r,e') \in \mathcal{N}_e} \pi^{(h)}(e, r, e') \mathbf{e}_{e'}^{(h)}\] \[\mathbf{e}_{\mathcal{N}_e} = ||_{h=1}^{H} \mathbf{e}_{\mathcal{N}_e}^{(h)}\]where\(H\)is the number of attention heads and\(||\)denotes concatenation.

Implementation: KGAT

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
import torch
import torch.nn as nn
import torch.nn.functional as F
import math

class KGATLayer(nn.Module):
    """
    Single layer of Knowledge Graph Attention Network.
    """
    def __init__(self, embedding_dim, num_relations, num_heads=1, dropout=0.1):
        super(KGATLayer, self).__init__()
        self.embedding_dim = embedding_dim
        self.num_relations = num_relations
        self.num_heads = num_heads
        self.head_dim = embedding_dim // num_heads
        self.dropout = dropout
        
        assert embedding_dim % num_heads == 0, "embedding_dim must be divisible by num_heads"
        
        # Attention mechanism
        self.attention = nn.Linear(embedding_dim * 3, num_heads)  # [e || r || e']
        
        # Transformation matrices for each head
        self.W_q = nn.Linear(embedding_dim, embedding_dim)
        self.W_k = nn.Linear(embedding_dim, embedding_dim)
        self.W_v = nn.Linear(embedding_dim, embedding_dim)
        
        # Output projection
        self.output_proj = nn.Linear(embedding_dim, embedding_dim)
        
        self.dropout_layer = nn.Dropout(dropout)
        self.leaky_relu = nn.LeakyReLU(0.2)
    
    def forward(self, entity_embeddings, relation_embeddings, neighbors):
        """
        Args:
            entity_embeddings: (num_entities, embedding_dim)
            relation_embeddings: (num_relations, embedding_dim)
            neighbors: List of neighbor lists for each entity
                      neighbors[i] = [(relation_id, neighbor_id), ...]
        """
        num_entities = entity_embeddings.size(0)
        outputs = []
        
        for i in range(num_entities):
            if len(neighbors[i]) == 0:
                outputs.append(entity_embeddings[i])
                continue
            
            # Get query, key, value
            query = self.W_q(entity_embeddings[i])  # (embedding_dim,)
            query = query.view(self.num_heads, self.head_dim)  # (num_heads, head_dim)
            
            neighbor_embs = []
            attention_scores = []
            
            for r_id, n_id in neighbors[i]:
                neighbor_emb = entity_embeddings[n_id]  # (embedding_dim,)
                relation_emb = relation_embeddings[r_id]  # (embedding_dim,)
                
                # Key and value
                key = self.W_k(neighbor_emb).view(self.num_heads, self.head_dim)  # (num_heads, head_dim)
                value = self.W_v(neighbor_emb).view(self.num_heads, self.head_dim)  # (num_heads, head_dim)
                
                neighbor_embs.append(value)
                
                # Compute attention score
                # Concatenate [entity || relation || neighbor]
                combined = torch.cat([
                    entity_embeddings[i],
                    relation_emb,
                    neighbor_emb
                ])  # (embedding_dim * 3,)
                
                score = self.attention(combined)  # (num_heads,)
                attention_scores.append(score)
            
            neighbor_embs = torch.stack(neighbor_embs)  # (num_neighbors, num_heads, head_dim)
            attention_scores = torch.stack(attention_scores)  # (num_neighbors, num_heads)
            
            # Apply attention
            attention_weights = F.softmax(attention_scores, dim=0)  # (num_neighbors, num_heads)
            attention_weights = self.dropout_layer(attention_weights)
            
            # Weighted aggregation
            aggregated = torch.sum(
                attention_weights.unsqueeze(2) * neighbor_embs,
                dim=0
            )  # (num_heads, head_dim)
            
            # Concatenate heads
            aggregated = aggregated.view(self.embedding_dim)  # (embedding_dim,)
            
            # Combine with original embedding
            output = entity_embeddings[i] + aggregated  # (embedding_dim,)
            output = self.output_proj(output)
            output = F.relu(output)
            
            outputs.append(output)
        
        outputs = torch.stack(outputs)  # (num_entities, embedding_dim)
        return outputs

class KGAT(nn.Module):
    """
    Knowledge Graph Attention Network for Recommendation.
    """
    def __init__(self, num_users, num_items, num_entities, num_relations,
                 embedding_dim=64, n_layers=2, num_heads=1, dropout=0.1):
        super(KGAT, self).__init__()
        self.num_users = num_users
        self.num_items = num_items
        self.num_entities = num_entities
        self.num_relations = num_relations
        self.embedding_dim = embedding_dim
        self.n_layers = n_layers
        
        # Embeddings
        self.user_embeddings = nn.Embedding(num_users, embedding_dim)
        self.entity_embeddings = nn.Embedding(num_entities, embedding_dim)
        self.relation_embeddings = nn.Embedding(num_relations, embedding_dim)
        
        # KGAT layers
        self.kgat_layers = nn.ModuleList([
            KGATLayer(embedding_dim, num_relations, num_heads, dropout)
            for _ in range(n_layers)
        ])
        
        # Initialize embeddings
        nn.init.xavier_uniform_(self.user_embeddings.weight)
        nn.init.xavier_uniform_(self.entity_embeddings.weight)
        nn.init.xavier_uniform_(self.relation_embeddings.weight)
    
    def forward(self, user_ids, item_ids, entity_neighbors):
        """
        Args:
            user_ids: (batch_size,) - user indices
            item_ids: (batch_size,) - item indices
            entity_neighbors: List of neighbor lists for each layer
        """
        # Combine user and entity embeddings
        # In CKG, users and items are both entities
        all_entity_emb = self.entity_embeddings.weight  # (num_entities, embedding_dim)
        
        # Replace user positions with user embeddings
        user_emb = self.user_embeddings.weight  # (num_users, embedding_dim)
        all_entity_emb[:self.num_users] = user_emb
        
        relation_emb = self.relation_embeddings.weight  # (num_relations, embedding_dim)
        
        # Apply KGAT layers
        current_emb = all_entity_emb
        for layer_idx, kgat_layer in enumerate(self.kgat_layers):
            if layer_idx < len(entity_neighbors):
                current_emb = kgat_layer(current_emb, relation_emb, entity_neighbors[layer_idx])
        
        # Get user and item embeddings
        user_emb_enhanced = current_emb[user_ids]  # (batch_size, embedding_dim)
        item_emb_enhanced = current_emb[self.num_users + item_ids]  # (batch_size, embedding_dim)
        
        # Predict interaction
        scores = torch.sum(user_emb_enhanced * item_emb_enhanced, dim=1)  # (batch_size,)
        return scores

# Example usage
if __name__ == "__main__":
    num_users = 1000
    num_items = 500
    num_entities = 2000  # Includes users + items + attributes
    num_relations = 10
    embedding_dim = 64
    
    model = KGAT(
        num_users=num_users,
        num_items=num_items,
        num_entities=num_entities,
        num_relations=num_relations,
        embedding_dim=embedding_dim,
        n_layers=2,
        num_heads=2
    )
    
    # Example neighbors
    entity_neighbors = [
        {0: [(0, 1000), (1, 1001)], 1000: [(2, 1500)]},  # Layer 0
        {1000: [(3, 1800)]}  # Layer 1
    ]
    
    user_ids = torch.LongTensor([0, 1])
    item_ids = torch.LongTensor([0, 1])
    
    scores = model(user_ids, item_ids, entity_neighbors)
    print(f"Prediction scores: {scores}")

HKGAT: Hybrid Knowledge Graph Attention Network

HKGAT Architecture

HKGAT (Hybrid Knowledge Graph Attention Network), introduced in recent work, extends KGAT by explicitly modeling heterogeneous entity types and relations. HKGAT uses type-specific attention mechanisms to handle the heterogeneity in knowledge graphs.

Key Innovation: HKGAT distinguishes between different entity types (users, items, attributes) and applies type-aware transformations and attention mechanisms.

HKGAT Algorithm

HKGAT maintains separate embeddings for different entity types:

  • User embeddings:\(\mathbf{U} \in \mathbb{R}^{|\mathcal{U}| \times d}\)
  • Item embeddings:\(\mathbf{I} \in \mathbb{R}^{|\mathcal{I}| \times d}\)
  • Attribute embeddings:\(\mathbf{A} \in \mathbb{R}^{|\mathcal{A}| \times d}\)For each entity type\(t\), HKGAT applies type-specific attention:\[\mathbf{e}_{\mathcal{N}_e}^t = \sum_{(r,e') \in \mathcal{N}_e, \text{type}(e')=t} \pi^t(e, r, e') \mathbf{e}_{e'}\]The type-specific attention weight is:\[\pi^t(e, r, e') = \frac{\exp(\mathbf{a}_t^T [\mathbf{W}_t \mathbf{e}_e || \mathbf{W}_r \mathbf{e}_r || \mathbf{W}_t \mathbf{e}_{e'}])}{\sum_{(r',e'') \in \mathcal{N}_e, \text{type}(e'')=t} \exp(\mathbf{a}_t^T [\mathbf{W}_t \mathbf{e}_e || \mathbf{W}_{r'} \mathbf{e}_{r'} || \mathbf{W}_t \mathbf{e}_{e''}])}\]where\(\mathbf{W}_t\)and\(\mathbf{W}_r\)are type-specific and relation-specific transformation matrices.

Implementation: HKGAT

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
import torch
import torch.nn as nn
import torch.nn.functional as F
from collections import defaultdict

class HKGATLayer(nn.Module):
    """
    Single layer of Hybrid Knowledge Graph Attention Network.
    """
    def __init__(self, embedding_dim, num_entity_types, num_relations, num_heads=1, dropout=0.1):
        super(HKGATLayer, self).__init__()
        self.embedding_dim = embedding_dim
        self.num_entity_types = num_entity_types
        self.num_relations = num_relations
        self.num_heads = num_heads
        self.head_dim = embedding_dim // num_heads
        self.dropout = dropout
        
        # Type-specific transformation matrices
        self.type_transforms = nn.ModuleList([
            nn.Linear(embedding_dim, embedding_dim)
            for _ in range(num_entity_types)
        ])
        
        # Relation-specific transformation matrices
        self.relation_transforms = nn.ModuleList([
            nn.Linear(embedding_dim, embedding_dim)
            for _ in range(num_relations)
        ])
        
        # Type-specific attention mechanisms
        self.type_attention = nn.ModuleList([
            nn.Linear(embedding_dim * 3, num_heads)
            for _ in range(num_entity_types)
        ])
        
        # Output projection
        self.output_proj = nn.Linear(embedding_dim, embedding_dim)
        self.dropout_layer = nn.Dropout(dropout)
    
    def forward(self, entity_embeddings, relation_embeddings, entity_types, neighbors):
        """
        Args:
            entity_embeddings: (num_entities, embedding_dim)
            relation_embeddings: (num_relations, embedding_dim)
            entity_types: (num_entities,) - type ID for each entity
            neighbors: List of neighbor lists: neighbors[i] = [(relation_id, neighbor_id, neighbor_type), ...]
        """
        num_entities = entity_embeddings.size(0)
        outputs = []
        
        for i in range(num_entities):
            entity_type = entity_types[i].item()
            entity_emb = entity_embeddings[i]  # (embedding_dim,)
            
            if len(neighbors[i]) == 0:
                outputs.append(entity_emb)
                continue
            
            # Group neighbors by type
            neighbors_by_type = defaultdict(list)
            for r_id, n_id, n_type in neighbors[i]:
                neighbors_by_type[n_type].append((r_id, n_id))
            
            # Aggregate for each type
            type_aggregated = []
            for t in range(self.num_entity_types):
                if t not in neighbors_by_type:
                    continue
                
                type_neighbors = neighbors_by_type[t]
                neighbor_embs = []
                attention_scores = []
                
                for r_id, n_id in type_neighbors:
                    neighbor_emb = entity_embeddings[n_id]  # (embedding_dim,)
                    relation_emb = relation_embeddings[r_id]  # (embedding_dim,)
                    
                    # Apply type-specific and relation-specific transformations
                    transformed_entity = self.type_transforms[entity_type](entity_emb)
                    transformed_neighbor = self.type_transforms[t](neighbor_emb)
                    transformed_relation = self.relation_transforms[r_id](relation_emb)
                    
                    neighbor_embs.append(transformed_neighbor)
                    
                    # Compute attention score
                    combined = torch.cat([
                        transformed_entity,
                        transformed_relation,
                        transformed_neighbor
                    ])  # (embedding_dim * 3,)
                    
                    score = self.type_attention[t](combined)  # (num_heads,)
                    attention_scores.append(score)
                
                if len(neighbor_embs) > 0:
                    neighbor_embs = torch.stack(neighbor_embs)  # (num_neighbors, embedding_dim)
                    attention_scores = torch.stack(attention_scores)  # (num_neighbors, num_heads)
                    
                    # Apply attention
                    attention_weights = F.softmax(attention_scores, dim=0)  # (num_neighbors, num_heads)
                    attention_weights = self.dropout_layer(attention_weights)
                    
                    # Weighted aggregation
                    aggregated = torch.sum(
                        attention_weights.unsqueeze(2) * neighbor_embs.unsqueeze(1),
                        dim=0
                    )  # (num_heads, embedding_dim)
                    
                    # Average over heads
                    aggregated = aggregated.mean(dim=0)  # (embedding_dim,)
                    type_aggregated.append(aggregated)
            
            # Combine type-specific aggregations
            if len(type_aggregated) > 0:
                aggregated_all = torch.stack(type_aggregated).mean(dim=0)  # (embedding_dim,)
                output = entity_emb + aggregated_all
            else:
                output = entity_emb
            
            output = self.output_proj(output)
            output = F.relu(output)
            outputs.append(output)
        
        outputs = torch.stack(outputs)  # (num_entities, embedding_dim)
        return outputs

class HKGAT(nn.Module):
    """
    Hybrid Knowledge Graph Attention Network for Recommendation.
    """
    def __init__(self, num_users, num_items, num_attributes, num_relations,
                 embedding_dim=64, n_layers=2, num_heads=1, dropout=0.1):
        super(HKGAT, self).__init__()
        self.num_users = num_users
        self.num_items = num_items
        self.num_attributes = num_attributes
        self.num_relations = num_relations
        self.embedding_dim = embedding_dim
        self.n_layers = n_layers
        self.num_entity_types = 3  # User, Item, Attribute
        
        # Separate embeddings for each entity type
        self.user_embeddings = nn.Embedding(num_users, embedding_dim)
        self.item_embeddings = nn.Embedding(num_items, embedding_dim)
        self.attribute_embeddings = nn.Embedding(num_attributes, embedding_dim)
        self.relation_embeddings = nn.Embedding(num_relations, embedding_dim)
        
        # HKGAT layers
        self.hkgat_layers = nn.ModuleList([
            HKGATLayer(embedding_dim, self.num_entity_types, num_relations, num_heads, dropout)
            for _ in range(n_layers)
        ])
        
        # Initialize embeddings
        nn.init.xavier_uniform_(self.user_embeddings.weight)
        nn.init.xavier_uniform_(self.item_embeddings.weight)
        nn.init.xavier_uniform_(self.attribute_embeddings.weight)
        nn.init.xavier_uniform_(self.relation_embeddings.weight)
    
    def forward(self, user_ids, item_ids, entity_neighbors, entity_types):
        """
        Args:
            user_ids: (batch_size,) - user indices
            item_ids: (batch_size,) - item indices
            entity_neighbors: List of neighbor lists for each layer
            entity_types: List of entity type arrays for each layer
        """
        # Combine all entity embeddings
        num_entities = self.num_users + self.num_items + self.num_attributes
        all_entity_emb = torch.zeros(num_entities, self.embedding_dim)
        
        all_entity_emb[:self.num_users] = self.user_embeddings.weight
        all_entity_emb[self.num_users:self.num_users+self.num_items] = self.item_embeddings.weight
        all_entity_emb[self.num_users+self.num_items:] = self.attribute_embeddings.weight
        
        relation_emb = self.relation_embeddings.weight
        
        # Apply HKGAT layers
        current_emb = all_entity_emb
        for layer_idx, hkgat_layer in enumerate(self.hkgat_layers):
            if layer_idx < len(entity_neighbors):
                current_emb = hkgat_layer(
                    current_emb,
                    relation_emb,
                    entity_types[layer_idx],
                    entity_neighbors[layer_idx]
                )
        
        # Get user and item embeddings
        user_emb_enhanced = current_emb[user_ids]  # (batch_size, embedding_dim)
        item_emb_enhanced = current_emb[self.num_users + item_ids]  # (batch_size, embedding_dim)
        
        # Predict interaction
        scores = torch.sum(user_emb_enhanced * item_emb_enhanced, dim=1)  # (batch_size,)
        return scores

# Example usage
if __name__ == "__main__":
    num_users = 1000
    num_items = 500
    num_attributes = 200
    num_relations = 10
    embedding_dim = 64
    
    model = HKGAT(
        num_users=num_users,
        num_items=num_items,
        num_attributes=num_attributes,
        num_relations=num_relations,
        embedding_dim=embedding_dim,
        n_layers=2,
        num_heads=2
    )
    
    # Entity types: 0=User, 1=Item, 2=Attribute
    entity_types = [
        torch.LongTensor([0] * num_users + [1] * num_items + [2] * num_attributes),
        torch.LongTensor([0] * num_users + [1] * num_items + [2] * num_attributes)
    ]
    
    # Example neighbors: (relation_id, neighbor_id, neighbor_type)
    entity_neighbors = [
        {0: [(0, 1000, 1), (1, 1500, 2)]},  # Layer 0
        {1000: [(2, 1600, 2)]}  # Layer 1
    ]
    
    user_ids = torch.LongTensor([0, 1])
    item_ids = torch.LongTensor([0, 1])
    
    scores = model(user_ids, item_ids, entity_neighbors, entity_types)
    print(f"Prediction scores: {scores}")

CKE: Collaborative Knowledge Base Embedding

CKE Architecture

CKE (Collaborative Knowledge base Embedding), introduced by Zhang et al. in KDD 2016, jointly learns embeddings for users, items, and knowledge graph entities. CKE combines collaborative filtering signals with knowledge graph structure through a unified embedding framework.

Key Idea: Items are represented by combining their collaborative filtering embeddings with their knowledge graph entity embeddings, enabling the model to leverage both interaction patterns and semantic relationships.

CKE Components

CKE consists of three components:

  1. Collaborative Filtering Module: Learns user and item embeddings from interaction data
  2. Structural Knowledge Module: Learns entity embeddings from knowledge graph structure
  3. Textual Knowledge Module: Learns embeddings from item descriptions/textual content

Mathematical Formulation

For an item\(i\), CKE represents it as:\[\mathbf{i} = \mathbf{i}_{CF} + \mathbf{i}_{KG} + \mathbf{i}_{text}\]where: -\(\mathbf{i}_{CF}\)is the collaborative filtering embedding -\(\mathbf{i}_{KG}\)is the knowledge graph entity embedding -\(\mathbf{i}_{text}\)is the textual content embedding

The collaborative filtering component uses matrix factorization:\[\hat{r}_{ui} = \mathbf{u}^T \mathbf{i}_{CF}\]The knowledge graph component uses TransR:\[\mathcal{L}_{KG} = \sum_{(h,r,t) \in \mathcal{G }} \sum_{(h',r,t') \notin \mathcal{G }} [\gamma + ||\mathbf{h}_r + \mathbf{r} - \mathbf{t}_r||_2^2 - ||\mathbf{h}'_r + \mathbf{r} - \mathbf{t}'_r||_2^2]_+\]where\(\mathbf{h}_r = \mathbf{h}M_r\)and\(\mathbf{t}_r = \mathbf{t}M_r\)are relation-specific projections.

The textual component uses a CNN or RNN to encode item descriptions:\[\mathbf{i}_{text} = \text{CNN}(\text{description}_i)\]The final loss combines all components:\[\mathcal{L} = \mathcal{L}_{CF} + \lambda_1 \mathcal{L}_{KG} + \lambda_2 \mathcal{L}_{text} + \lambda_3 \mathcal{L}_{reg}\]

Implementation: CKE

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
import torch
import torch.nn as nn
import torch.nn.functional as F
from torch.nn.utils.rnn import pad_sequence

class TransR(nn.Module):
    """
    TransR: Learning entity and relation embeddings for knowledge graph.
    """
    def __init__(self, num_entities, num_relations, embedding_dim, relation_dim):
        super(TransR, self).__init__()
        self.num_entities = num_entities
        self.num_relations = num_relations
        self.embedding_dim = embedding_dim
        self.relation_dim = relation_dim
        
        # Entity embeddings
        self.entity_embeddings = nn.Embedding(num_entities, embedding_dim)
        
        # Relation embeddings
        self.relation_embeddings = nn.Embedding(num_relations, relation_dim)
        
        # Relation-specific projection matrices
        self.projection_matrices = nn.Embedding(num_relations, embedding_dim * relation_dim)
        
        # Initialize
        nn.init.xavier_uniform_(self.entity_embeddings.weight)
        nn.init.xavier_uniform_(self.relation_embeddings.weight)
        nn.init.xavier_uniform_(self.projection_matrices.weight)
    
    def forward(self, head_ids, relation_ids, tail_ids, neg_head_ids=None, neg_tail_ids=None):
        """
        Args:
            head_ids: (batch_size,) - head entity indices
            relation_ids: (batch_size,) - relation indices
            tail_ids: (batch_size,) - tail entity indices
            neg_head_ids: (batch_size,) - negative head entity indices
            neg_tail_ids: (batch_size,) - negative tail entity indices
        """
        # Get embeddings
        head_emb = self.entity_embeddings(head_ids)  # (batch_size, embedding_dim)
        relation_emb = self.relation_embeddings(relation_ids)  # (batch_size, relation_dim)
        tail_emb = self.entity_embeddings(tail_ids)  # (batch_size, embedding_dim)
        
        # Get projection matrices
        proj_mat = self.projection_matrices(relation_ids)  # (batch_size, embedding_dim * relation_dim)
        proj_mat = proj_mat.view(-1, self.embedding_dim, self.relation_dim)  # (batch_size, embedding_dim, relation_dim)
        
        # Project entities to relation space
        head_proj = torch.bmm(head_emb.unsqueeze(1), proj_mat).squeeze(1)  # (batch_size, relation_dim)
        tail_proj = torch.bmm(tail_emb.unsqueeze(1), proj_mat).squeeze(1)  # (batch_size, relation_dim)
        
        # Compute scores: ||h_r + r - t_r||_2^2
        scores = torch.norm(head_proj + relation_emb - tail_proj, p=2, dim=1) ** 2  # (batch_size,)
        
        if neg_head_ids is not None and neg_tail_ids is not None:
            # Negative samples
            neg_head_emb = self.entity_embeddings(neg_head_ids)
            neg_tail_emb = self.entity_embeddings(neg_tail_ids)
            
            neg_head_proj = torch.bmm(neg_head_emb.unsqueeze(1), proj_mat).squeeze(1)
            neg_tail_proj = torch.bmm(neg_tail_emb.unsqueeze(1), proj_mat).squeeze(1)
            
            neg_scores = torch.norm(neg_head_proj + relation_emb - neg_tail_proj, p=2, dim=1) ** 2
            
            # Margin-based loss
            margin = 1.0
            loss = F.relu(margin + scores - neg_scores).mean()
            return loss
        
        return scores

class TextEncoder(nn.Module):
    """
    CNN-based text encoder for item descriptions.
    """
    def __init__(self, vocab_size, embedding_dim, num_filters=100, filter_sizes=[3, 4, 5]):
        super(TextEncoder, self).__init__()
        self.embedding_dim = embedding_dim
        self.num_filters = num_filters
        self.filter_sizes = filter_sizes
        
        # Word embeddings
        self.word_embeddings = nn.Embedding(vocab_size, embedding_dim)
        
        # Convolutional layers
        self.convs = nn.ModuleList([
            nn.Conv1d(embedding_dim, num_filters, kernel_size=fs)
            for fs in filter_sizes
        ])
        
        # Initialize
        nn.init.xavier_uniform_(self.word_embeddings.weight)
    
    def forward(self, text_sequences):
        """
        Args:
            text_sequences: List of token sequences, each is a LongTensor
        """
        # Embed words
        embedded = []
        for seq in text_sequences:
            emb = self.word_embeddings(seq)  # (seq_len, embedding_dim)
            embedded.append(emb)
        
        # Pad sequences
        embedded = pad_sequence(embedded, batch_first=True)  # (batch_size, max_len, embedding_dim)
        embedded = embedded.transpose(1, 2)  # (batch_size, embedding_dim, max_len)
        
        # Apply convolutions
        conv_outputs = []
        for conv in self.convs:
            conv_out = F.relu(conv(embedded))  # (batch_size, num_filters, conv_len)
            pooled = F.max_pool1d(conv_out, kernel_size=conv_out.size(2))  # (batch_size, num_filters, 1)
            conv_outputs.append(pooled.squeeze(2))  # (batch_size, num_filters)
        
        # Concatenate
        output = torch.cat(conv_outputs, dim=1)  # (batch_size, num_filters * len(filter_sizes))
        return output

class CKE(nn.Module):
    """
    Collaborative Knowledge base Embedding for Recommendation.
    """
    def __init__(self, num_users, num_items, num_entities, num_relations,
                 embedding_dim=64, relation_dim=32, vocab_size=10000,
                 kg_lambda=0.1, text_lambda=0.1, reg_lambda=0.01):
        super(CKE, self).__init__()
        self.num_users = num_users
        self.num_items = num_items
        self.num_entities = num_entities
        self.num_relations = num_relations
        self.embedding_dim = embedding_dim
        self.kg_lambda = kg_lambda
        self.text_lambda = text_lambda
        self.reg_lambda = reg_lambda
        
        # Collaborative filtering embeddings
        self.user_embeddings = nn.Embedding(num_users, embedding_dim)
        self.item_cf_embeddings = nn.Embedding(num_items, embedding_dim)
        
        # Knowledge graph embeddings (TransR)
        self.transr = TransR(num_entities, num_relations, embedding_dim, relation_dim)
        
        # Text encoder
        self.text_encoder = TextEncoder(vocab_size, embedding_dim)
        
        # Projection layer to combine KG and text embeddings
        self.item_kg_proj = nn.Linear(embedding_dim, embedding_dim)
        self.item_text_proj = nn.Linear(300, embedding_dim)  # 300 = num_filters * len(filter_sizes)
        
        # Initialize
        nn.init.xavier_uniform_(self.user_embeddings.weight)
        nn.init.xavier_uniform_(self.item_cf_embeddings.weight)
    
    def forward(self, user_ids, item_ids, item_texts=None, kg_triples=None):
        """
        Args:
            user_ids: (batch_size,) - user indices
            item_ids: (batch_size,) - item indices
            item_texts: List of text sequences for items
            kg_triples: (head_ids, relation_ids, tail_ids) for KG loss
        """
        # Collaborative filtering embeddings
        user_emb = self.user_embeddings(user_ids)  # (batch_size, embedding_dim)
        item_cf_emb = self.item_cf_embeddings(item_ids)  # (batch_size, embedding_dim)
        
        # Knowledge graph embeddings
        # Get entity embeddings for items (assuming item_ids map to entity_ids)
        item_kg_emb = self.transr.entity_embeddings(item_ids)  # (batch_size, embedding_dim)
        item_kg_emb = self.item_kg_proj(item_kg_emb)  # (batch_size, embedding_dim)
        
        # Text embeddings
        if item_texts is not None:
            item_text_emb = self.text_encoder(item_texts)  # (batch_size, text_dim)
            item_text_emb = self.item_text_proj(item_text_emb)  # (batch_size, embedding_dim)
        else:
            item_text_emb = torch.zeros_like(item_cf_emb)
        
        # Combine item embeddings
        item_emb = item_cf_emb + item_kg_emb + item_text_emb  # (batch_size, embedding_dim)
        
        # Predict rating
        scores = torch.sum(user_emb * item_emb, dim=1)  # (batch_size,)
        
        return scores
    
    def compute_loss(self, user_ids, item_ids, ratings, item_texts=None, kg_triples=None):
        """
        Compute total loss including CF, KG, and text components.
        """
        # CF loss
        pred_scores = self.forward(user_ids, item_ids, item_texts, kg_triples)
        cf_loss = F.mse_loss(pred_scores, ratings.float())
        
        # KG loss
        kg_loss = 0.0
        if kg_triples is not None:
            head_ids, relation_ids, tail_ids = kg_triples
            # Sample negative triples
            neg_head_ids = torch.randint(0, self.num_entities, head_ids.shape)
            neg_tail_ids = torch.randint(0, self.num_entities, tail_ids.shape)
            kg_loss = self.transr(head_ids, relation_ids, tail_ids, neg_head_ids, neg_tail_ids)
        
        # Regularization
        reg_loss = (
            torch.norm(self.user_embeddings.weight) ** 2 +
            torch.norm(self.item_cf_embeddings.weight) ** 2 +
            torch.norm(self.transr.entity_embeddings.weight) ** 2
        )
        
        total_loss = cf_loss + self.kg_lambda * kg_loss + self.reg_lambda * reg_loss
        return total_loss

# Example usage
if __name__ == "__main__":
    num_users = 1000
    num_items = 500
    num_entities = 2000
    num_relations = 10
    embedding_dim = 64
    
    model = CKE(
        num_users=num_users,
        num_items=num_items,
        num_entities=num_entities,
        num_relations=num_relations,
        embedding_dim=embedding_dim
    )
    
    user_ids = torch.LongTensor([0, 1, 2])
    item_ids = torch.LongTensor([0, 1, 2])
    ratings = torch.FloatTensor([5.0, 4.0, 3.0])
    
    # Example text sequences
    item_texts = [
        torch.LongTensor([1, 2, 3, 4]),
        torch.LongTensor([5, 6, 7]),
        torch.LongTensor([8, 9, 10, 11, 12])
    ]
    
    # Example KG triples
    kg_triples = (
        torch.LongTensor([0, 1, 2]),  # head_ids
        torch.LongTensor([0, 1, 2]),  # relation_ids
        torch.LongTensor([100, 101, 102])  # tail_ids
    )
    
    loss = model.compute_loss(user_ids, item_ids, ratings, item_texts, kg_triples)
    print(f"Total loss: {loss.item():.4f}")

Recent Advances: RecKG and Beyond

RecKG: Knowledge Graph for Recommendation

RecKG represents a family of recent methods that integrate knowledge graphs more deeply into recommendation systems. Key advances include:

Dynamic Knowledge Graph Construction: Building KGs that evolve with user interactions and item updates

Multi-hop Reasoning: Explicitly modeling multi-hop paths in the knowledge graph for explainable recommendations

Heterogeneous Graph Neural Networks: Applying advanced GNN architectures (GraphSAGE, GAT, Graph Transformer) to knowledge graphs

Key Research Directions

Temporal Knowledge Graphs: Incorporating temporal information into knowledge graphs to model evolving preferences and item popularity

Multi-modal Knowledge Graphs: Combining structured knowledge with images, text, and other modalities

Federated Knowledge Graphs: Learning from distributed knowledge graphs across different domains or platforms

Explainable Recommendations: Using knowledge graph paths to generate natural language explanations

Implementation: Multi-hop Path Reasoning

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
import torch
import torch.nn as nn
import torch.nn.functional as F
from collections import deque

class PathReasoning(nn.Module):
    """
    Multi-hop path reasoning for explainable recommendation.
    """
    def __init__(self, num_entities, num_relations, embedding_dim=64):
        super(PathReasoning, self).__init__()
        self.num_entities = num_entities
        self.num_relations = num_relations
        self.embedding_dim = embedding_dim
        
        # Entity and relation embeddings
        self.entity_embeddings = nn.Embedding(num_entities, embedding_dim)
        self.relation_embeddings = nn.Embedding(num_relations, embedding_dim)
        
        # Path scoring network
        self.path_scorer = nn.Sequential(
            nn.Linear(embedding_dim * 3, embedding_dim),
            nn.ReLU(),
            nn.Linear(embedding_dim, 1)
        )
        
        nn.init.xavier_uniform_(self.entity_embeddings.weight)
        nn.init.xavier_uniform_(self.relation_embeddings.weight)
    
    def find_paths(self, kg_adj, start_entity, end_entity, max_hops=3, max_paths=10):
        """
        Find paths from start_entity to end_entity in the knowledge graph.
        
        Args:
            kg_adj: Knowledge graph adjacency: {entity_id: [(relation_id, tail_id), ...]}
            start_entity: Starting entity ID
            end_entity: Target entity ID
            max_hops: Maximum number of hops
            max_paths: Maximum number of paths to return
        
        Returns:
            List of paths: [(relation_ids, entity_ids), ...]
        """
        paths = []
        queue = deque([(start_entity, [], [])])  # (current_entity, relation_path, entity_path)
        visited = set()
        
        while queue and len(paths) < max_paths:
            current, rel_path, ent_path = queue.popleft()
            
            if current == end_entity and len(rel_path) > 0:
                paths.append((rel_path, ent_path + [end_entity]))
                continue
            
            if len(rel_path) >= max_hops:
                continue
            
            if current not in kg_adj:
                continue
            
            for r_id, t_id in kg_adj[current]:
                if (current, r_id, t_id) not in visited:
                    visited.add((current, r_id, t_id))
                    queue.append((t_id, rel_path + [r_id], ent_path + [current]))
        
        return paths[:max_paths]
    
    def score_path(self, path):
        """
        Score a path using the path scoring network.
        
        Args:
            path: (relation_ids, entity_ids) tuple
        
        Returns:
            Path score
        """
        rel_ids, ent_ids = path
        
        if len(rel_ids) == 0:
            return torch.tensor(0.0)
        
        # Get embeddings
        start_emb = self.entity_embeddings(torch.LongTensor([ent_ids[0]]))
        rel_emb = self.relation_embeddings(torch.LongTensor(rel_ids)).mean(dim=0, keepdim=True)
        end_emb = self.entity_embeddings(torch.LongTensor([ent_ids[-1]]))
        
        # Combine
        combined = torch.cat([start_emb, rel_emb, end_emb], dim=1)  # (1, embedding_dim * 3)
        score = self.path_scorer(combined).squeeze()
        
        return score
    
    def recommend_with_explanation(self, user_id, user_items, kg_adj, candidate_items, max_hops=3):
        """
        Recommend items with path-based explanations.
        
        Args:
            user_id: User ID
            user_items: List of items the user has interacted with
            kg_adj: Knowledge graph adjacency
            candidate_items: List of candidate item IDs
            max_hops: Maximum hops for path finding
        
        Returns:
            List of (item_id, score, explanation_path) tuples
        """
        recommendations = []
        
        for item_id in candidate_items:
            # Find paths from user items to candidate item
            all_paths = []
            for user_item in user_items:
                paths = self.find_paths(kg_adj, user_item, item_id, max_hops=max_hops)
                all_paths.extend(paths)
            
            if len(all_paths) == 0:
                continue
            
            # Score paths
            path_scores = []
            for path in all_paths:
                score = self.score_path(path)
                path_scores.append((score.item(), path))
            
            # Get best path
            path_scores.sort(reverse=True, key=lambda x: x[0])
            best_score, best_path = path_scores[0]
            
            recommendations.append((item_id, best_score, best_path))
        
        # Sort by score
        recommendations.sort(reverse=True, key=lambda x: x[1])
        return recommendations

# Example usage
if __name__ == "__main__":
    num_entities = 1000
    num_relations = 20
    embedding_dim = 64
    
    model = PathReasoning(num_entities, num_relations, embedding_dim)
    
    # Example knowledge graph
    kg_adj = {
        0: [(0, 100), (1, 101)],  # Entity 0 connected to 100, 101
        100: [(2, 200)],  # Entity 100 connected to 200
        200: [(3, 300)],  # Entity 200 connected to 300
    }
    
    # Find paths
    paths = model.find_paths(kg_adj, start_entity=0, end_entity=300, max_hops=3)
    print(f"Found {len(paths)} paths from entity 0 to entity 300")
    
    for i, (rel_path, ent_path) in enumerate(paths):
        score = model.score_path((rel_path, ent_path))
        print(f"Path {i+1}: {ent_path} via relations {rel_path}, score: {score.item():.4f}")

Practical Implementation Tips

Knowledge Graph Construction

Data Sources: - Structured databases: DBpedia, Wikidata, Freebase - Domain-specific databases: MovieLens metadata, product catalogs - Text extraction: Named entity recognition and relation extraction from descriptions

Entity Alignment: - String matching: Exact or fuzzy matching of item names - Embedding-based: Use pre-trained embeddings to find similar entities - Manual curation: Domain experts align entities

Training Strategies

Joint Training: Train KG embedding and recommendation objectives together

Pre-training: Pre-train KG embeddings, then fine-tune for recommendation

Multi-task Learning: Share embeddings between KG and recommendation tasks

Evaluation Metrics

Recommendation Metrics: - Hit Rate (HR@K) - Normalized Discounted Cumulative Gain (NDCG@K) - Mean Reciprocal Rank (MRR)

Knowledge Graph Metrics: - Link prediction accuracy - Entity ranking metrics - Path quality metrics

Implementation: Complete Training Pipeline

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
import torch
import torch.nn as nn
import torch.optim as optim
from torch.utils.data import Dataset, DataLoader
import numpy as np

class RecommendationDataset(Dataset):
    """
    Dataset for knowledge graph-enhanced recommendation.
    """
    def __init__(self, user_ids, item_ids, ratings, kg_triples=None):
        self.user_ids = torch.LongTensor(user_ids)
        self.item_ids = torch.LongTensor(item_ids)
        self.ratings = torch.FloatTensor(ratings)
        self.kg_triples = kg_triples
    
    def __len__(self):
        return len(self.user_ids)
    
    def __getitem__(self, idx):
        sample = {
            'user_id': self.user_ids[idx],
            'item_id': self.item_ids[idx],
            'rating': self.ratings[idx]
        }
        if self.kg_triples is not None:
            sample['kg_triple'] = self.kg_triples[idx]
        return sample

def train_kg_recommender(model, train_loader, val_loader, num_epochs=10, lr=0.001):
    """
    Training function for knowledge graph-enhanced recommender.
    """
    optimizer = optim.Adam(model.parameters(), lr=lr)
    criterion = nn.MSELoss()
    
    best_val_loss = float('inf')
    
    for epoch in range(num_epochs):
        # Training
        model.train()
        train_loss = 0.0
        
        for batch in train_loader:
            user_ids = batch['user_id']
            item_ids = batch['item_id']
            ratings = batch['rating']
            
            optimizer.zero_grad()
            
            if 'kg_triple' in batch:
                kg_triples = batch['kg_triple']
                loss = model.compute_loss(user_ids, item_ids, ratings, kg_triples=kg_triples)
            else:
                predictions = model(user_ids, item_ids)
                loss = criterion(predictions, ratings)
            
            loss.backward()
            optimizer.step()
            
            train_loss += loss.item()
        
        train_loss /= len(train_loader)
        
        # Validation
        model.eval()
        val_loss = 0.0
        
        with torch.no_grad():
            for batch in val_loader:
                user_ids = batch['user_id']
                item_ids = batch['item_id']
                ratings = batch['rating']
                
                if 'kg_triple' in batch:
                    kg_triples = batch['kg_triple']
                    loss = model.compute_loss(user_ids, item_ids, ratings, kg_triples=kg_triples)
                else:
                    predictions = model(user_ids, item_ids)
                    loss = criterion(predictions, ratings)
                
                val_loss += loss.item()
        
        val_loss /= len(val_loader)
        
        print(f"Epoch {epoch+1}/{num_epochs}, Train Loss: {train_loss:.4f}, Val Loss: {val_loss:.4f}")
        
        if val_loss < best_val_loss:
            best_val_loss = val_loss
            torch.save(model.state_dict(), 'best_model.pt')
    
    return model

# Example usage
if __name__ == "__main__":
    # Generate synthetic data
    num_users = 1000
    num_items = 500
    num_entities = 2000
    num_relations = 10
    
    # Training data
    train_user_ids = np.random.randint(0, num_users, 10000)
    train_item_ids = np.random.randint(0, num_items, 10000)
    train_ratings = np.random.uniform(1, 5, 10000)
    
    # Validation data
    val_user_ids = np.random.randint(0, num_users, 2000)
    val_item_ids = np.random.randint(0, num_items, 2000)
    val_ratings = np.random.uniform(1, 5, 2000)
    
    # Create datasets
    train_dataset = RecommendationDataset(train_user_ids, train_item_ids, train_ratings)
    val_dataset = RecommendationDataset(val_user_ids, val_item_ids, val_ratings)
    
    train_loader = DataLoader(train_dataset, batch_size=64, shuffle=True)
    val_loader = DataLoader(val_dataset, batch_size=64, shuffle=False)
    
    # Initialize model (using CKE as example)
    model = CKE(
        num_users=num_users,
        num_items=num_items,
        num_entities=num_entities,
        num_relations=num_relations,
        embedding_dim=64
    )
    
    # Train
    trained_model = train_kg_recommender(model, train_loader, val_loader, num_epochs=10)

❓ Q&A: Knowledge Graph-Enhanced Recommendation Common Questions

How do knowledge graphs help with the cold start problem?

Knowledge graphs provide rich semantic information about items even when they have no interaction history. For example, a new movie can be recommended based on shared actors, directors, or genres with movies a user has enjoyed. The knowledge graph creates connections between items through attributes, enabling recommendations for cold-start items.

What's the difference between RippleNet and KGCN?

RippleNet propagates user preferences through the knowledge graph: when a user interacts with an item, this preference "ripples" outward, activating related entities. KGCN, on the other hand, learns item embeddings by aggregating information from neighboring entities in the knowledge graph. RippleNet is user-centric (propagates from user items), while KGCN is item-centric (learns item representations).

How does KGAT improve upon KGCN?

KGAT extends KGCN by using attention mechanisms to automatically learn the importance of different relations and entities when aggregating information. KGAT also builds a collaborative knowledge graph that combines user-item interactions with the knowledge graph, enabling joint learning of collaborative and semantic signals.

What is the advantage of HKGAT over KGAT?

HKGAT explicitly models heterogeneous entity types (users, items, attributes) and applies type-specific transformations and attention mechanisms. This allows the model to better handle the heterogeneity in knowledge graphs, where different entity types may require different treatment.

How does CKE combine different information sources?

CKE represents items as the sum of three components: collaborative filtering embeddings (from interaction data), knowledge graph embeddings (from KG structure), and textual embeddings (from item descriptions). This allows the model to leverage multiple information sources simultaneously.

What are the computational challenges of knowledge graph-enhanced recommendation?

Knowledge graphs can be very large (millions of entities and relations), making neighbor sampling and aggregation computationally expensive. Solutions include: sampling strategies (e.g., RippleNet's n_memory parameter), hierarchical aggregation, and efficient graph neural network implementations.

How do you evaluate knowledge graph-enhanced recommendation systems?

Evaluation includes both recommendation metrics (Hit Rate, NDCG, MRR) and knowledge graph metrics (link prediction accuracy, entity ranking). Additionally, explainability can be evaluated by measuring the quality of generated explanations based on knowledge graph paths.

Can knowledge graphs improve recommendation diversity?

Yes, knowledge graphs help discover diverse recommendations by exploring different relation paths. Instead of recommending only similar items, the system can recommend items connected through different attributes (e.g., same director but different genre), increasing diversity.

How do you handle missing or noisy knowledge graph data?

Missing data can be handled through: (1) learning embeddings that are robust to missing relations, (2) using attention mechanisms that can downweight noisy connections, (3) data augmentation through relation inference, and (4) multi-task learning that shares information across tasks.

Recent trends include: (1) temporal knowledge graphs that model evolving preferences, (2) multi-modal knowledge graphs combining structured data with images and text, (3) federated learning across distributed knowledge graphs, (4) transformer-based architectures for knowledge graphs, and (5) explainable AI using knowledge graph paths.

How do you choose the number of hops/layers in knowledge graph methods?

The number of hops/layers depends on the graph structure and task. Too few hops may miss important relationships, while too many hops can introduce noise. Common approaches: (1) start with 2-3 hops and tune based on validation performance, (2) use attention mechanisms to automatically weight different hops, (3) analyze the graph diameter to determine maximum useful hops.

What's the relationship between knowledge graphs and graph neural networks?

Knowledge graphs provide the structure (entities and relations), while graph neural networks provide the learning mechanism (how to aggregate information from neighbors). Most modern KG-enhanced recommendation methods use GNNs (GCN, GAT, GraphSAGE) to learn from knowledge graphs.

How do you align items in your recommendation system with entities in external knowledge graphs?

Entity alignment methods include: (1) string matching (exact or fuzzy), (2) embedding-based similarity (learn embeddings for both and find nearest neighbors), (3) manual curation by domain experts, (4) joint learning that aligns entities during training.

Can knowledge graphs help with explainable recommendation?

Yes, knowledge graphs enable explainable recommendations by providing interpretable paths. For example, "We recommend Movie X because it's directed by Director Y, who also directed Movie Z that you rated highly." These explanations are generated by finding paths in the knowledge graph from user items to recommended items.

What are the limitations of knowledge graph-enhanced recommendation?

Limitations include: (1) dependency on knowledge graph quality and completeness, (2) computational cost for large graphs, (3) difficulty in handling dynamic knowledge (new entities/relations), (4) potential bias from knowledge graph construction, and (5) challenges in multi-domain scenarios where knowledge graphs may not align well.

Summary

Knowledge graph-enhanced recommendation systems represent a significant advancement in recommendation technology, combining the pattern recognition capabilities of collaborative filtering with the semantic reasoning enabled by structured knowledge. Methods like RippleNet, KGCN, KGAT, HKGAT, and CKE demonstrate different approaches to leveraging knowledge graphs: propagation-based methods spread user preferences through the graph, convolutional methods learn item representations from neighbors, attention-based methods weight different relations, and embedding methods jointly learn from multiple information sources.

The key advantages of knowledge graph-enhanced recommendation include: addressing cold start problems through rich semantic information, improving explainability through interpretable paths, increasing diversity by exploring different relation types, and enabling better handling of sparse data through dense semantic connections.

As research continues, we're seeing advances in temporal modeling, multi-modal integration, federated learning, and explainable AI. These developments promise to make knowledge graph-enhanced recommendation systems even more powerful and practical for real-world applications.

References and Further Reading

  1. Wang, H., et al. "RippleNet: Propagating User Preferences on the Knowledge Graph for Recommender Systems." CIKM 2018. arXiv:1803.03467

  2. Wang, X., et al. "KGCN: Knowledge Graph Convolutional Network for Recommender Systems." WWW 2019. arXiv:1904.12575

  3. Wang, X., et al. "KGAT: Knowledge Graph Attention Network for Recommendation." KDD 2019. arXiv:1905.07854

  4. Zhang, F., et al. "Collaborative Knowledge Base Embedding for Recommender Systems." KDD 2016. DOI:10.1145/2939672.2939673

  • Post title：Recommendation Systems (8): Knowledge Graph-Enhanced Recommendation
  • Post author：Chen Kai
  • Create time：2026-02-03 23:11:11
  • Post link：https://www.chenk.top/recommendation-systems-8-knowledge-graph/
  • Copyright Notice：All articles in this blog are licensed under BY-NC-SA unless stating additionally.
 Comments