Text Enriched Sparse Hyperbolic Graph Convolutional NetworksThanks: N. Choudhary, and C. K. Reddy are with the Department of Computer Science, Virginia Tech, Arlington, VA, USA 22203. C. K. Reddy is also affiliated with Amazon, Palo Alto, CA, USA 94301. E-mail: nurendra@vt.edu, reddy@cs.vt.eduN. Rao, and K. Subbian are with Amazon, Palo Alto, CA, USA 94301. E-mail: nikhilsr@amazon.com, ksubbian@amazon.com

Early research on graph representations relied on learning effective node representations, primarily, through two broad methods, namely, matrix factorization and random walks. In matrix factorization based approaches [20], the sparse graph adjacency matrix $A$ is factorized into low-dimensional dense matrix $L$ such that the information loss $\|L^{T}L-A\|$ is minimized. In the random walk based approaches [6, 5, 21], a node’s neighborhood is collected with random walks through its edges, and the neighborhood is used to predict the node’s representation in a dense network framework. Earlier methods such as LINE [22] and SDNE [23] use first-order (nodes connected by an edge) and second-order (nodes with similar neighborhood) proximity to learn the node representations. These methods form a vector space model for graphs and have shown some preliminary success. However, they are node-specific and do not consider the neighborhood information of a node or the overall graph structure. In more recent works, aggregating information from a nodes’ neighborhood is explored using the neural network models. Graph neural networks (GNN) [24], typically applied to node classification, aggregate information from a nodes’ neighborhood to predict the label for the root node. Several approaches based on different neural network architectures for neighborhood aggregation have been developed in recent years and some of the popular ones include GraphSage [7] (LSTM), Graph Convolution Networks (GCN) [8], and Graph Attention Networks (GAT) [9]. Another line of work specifically tailored for heterogeneous graphs [16, 25, 26], utilizes the rich relational information through metapath aggregation. These approaches, while efficient at aggregating neighborhood information, do not consider the node’s semantic attributes or the global graph structure. In the proposed TESH-GCN model, we aim to utilize the node’s semantic signal, in congruence with global adjacency tensor, to capture both the node’s semantic attributes and its position in the overall graph structure.

In recent research [11], graph datasets have been shown to possess an inherent hierarchy between nodes thus demonstrating a non-Euclidean geometry. In [11], the authors provide the gyrovector space model including the hyperbolic variants of the algebraic operations required to design neural networks. The algebraic operations for the Poincaré ball of curvature $c$ are the following: Möbius addition $(\oplus_{c})$, exponential map $(\exp_{x}^{c})$, logarithmic map $(\log_{x}^{c})$, Möbius scalar multiplication $(\otimes_{c})$, and hyperbolic activation $(\sigma_{c})$.

where $\lambda_{x}^{c}=\frac{2}{(1-c\|x\|^{2})}$ is the metric conformal factor. Based on these approaches, hyperbolic networks such as HGNN [11], HGCN [12], HAN [27], and HypE [28] have shown to outperform their Euclidean counterparts on graph datasets. However, these approaches still focus on the nodes’ local neighborhood and not the overall graph structure. Furthermore, hyperbolic transformations are performed on entire vectors and are thus inefficient on sparse tensors. In our model, we utilize the $\beta-$split and $\beta-$concatenation operations [29] to optimize the hyperbolic graph convolution for sparse adjacency tensors.

Let us consider a heterogeneous graph $\mathcal{G}=(V,E)$ with $K$ edge types, where $v\in V$ is the set of its nodes and $e_{k}(v_{i},v_{j})\in E\in\mathbb{B}^{K\times|V|\times|V|}$ is a sparse Boolean adjacency tensor (which indicates if edge type $e_{k}$ exists between nodes $v_{i}$ and $v_{j}$ or not). Each node $v_{i}$ also contains a corresponding text sequence $s_{i}$. The sparsity of the adjacency tensor and hierarchy of the graph $G$ is quantified by the sparsity ratio ($R$, Definition 1) and hyperbolicity ($\delta$, Definition 2), respectively. Higher sparsity ratio implies that $E$ is sparser, whereas lower hyperbolicity implies $G$ has more hierarchical relations.

For the task of link prediction, given input nodes $v_{i}$ and $v_{j}$ with corresponding text sequence $s_{i}$ and $s_{j}$, respectively and an incomplete training adjacency tensor $E$, our goal is to train TESH-GCN to optimize a predictor $P_{\theta}$ parameterized by $\theta$ such that;

where z is a Boolean indicator that indicates if an edge between the two nodes exists ($z=1$) or not ($z=0$) and y is a class predictor for each $k\in K$ edge types. $\hat{y_{k}}$ is the probability of each class $k\in K$ predicted by TESH-GCN and $y_{k}$ is the ground truth class label.

In this section, we describe the message aggregation framework of TESH-GCN, which allows us to aggregate the node’s text-enriched local neighborhood and long metapath features (through semantic signals and reformulated hyperbolic graph convolution) from sparse adjacency tensors in the hyperbolic space. In this section, We detail the (i) methodology of integrating semantic features with graph tensors, (ii) sparse HGCN layer to encode hierarchical and graph structure information efficiently, and (iii) aggregation through self-attention to improve model robustness.

Let us say, the $d$-dimensional hyperbolic vector in Poincaré ball of curvature $c$ is $x\in\mathbb{H}^{d}_{c}$ and $\beta_{d}=B\left(\frac{d}{2},\frac{1}{2}\right)$ is a scalar beta coefficient, where B is the beta function. Then, the $\beta$-split and $\beta$-concatenation are defined as follows.

The final encoding of an adjacency tensor $A_{k}$ is, thus, the output features of the last convolution layer transformed to the tangent space with the logarithmic map $h_{k,L}=log^{c_{L}}_{0}(h_{k,L})$⁵⁵ 5 The transformation from hyperbolic space to tangent space with logarithmic map is required for attention-based aggregation as such formulation is not well-defined for the hyperbolic space..

In this work, we consider a generalized link prediction problem in heterogeneous networks where there are two sub-tasks. (i) To predict if a link exists between two nodes and (ii) To predict the class/type of link (if one exists). One method to achieve this goal is to add the non-existence of link as another class. Let us assume we add a class $z$ which indicates the existence of the link ($z=1$) and $z=0$ when the link is absent. Then, for the task of link prediction, we need to support the independence assumption, i.e., $z\perp\!\!\!\perp e_{k},~~\forall e_{k}\in E$, which is not true. Prediction of an edge type $e_{k}$ is conditional on $z=1$. Hence, we setup a multi-step loss that first predicts the existence of a link and then classifies it into an edge type.

where $x$ and $\theta$ are the input and model parameters, respectively. $L$ is the cross entropy loss that needs to be minimized. Although we use this generalized link prediction as the task of interest in this paper, TESH-GCN can be applied to any task such as node/graph classification by replacing the loss with the appropriate loss.

We implemented TESH-GCN using Pytorch [33] on eight NVIDIA V100 GPUs with 16 GB VRAM. For gradient descent, we used Riemmanian Adam [34] with standard $\beta$ values of 0.9 and 0.999 and an initial learning rate of 0.001. Number of dimensions ($D$) and number of layers ($L$) is empirically selected based on performance-memory trade-off. Figure 6 presents the memory-performance trade-off for different choices of parameters D and L. We observe that the $D=8$ and $L=8$ provides the best performance for the memory required. Hence, we chose them for the final implementation of our model. For non-linearity, we used the hyperbolic activation function, given in Eq. (1). The sparsity in the model variables is handled using the torch-sparse library⁶⁶ 6 https://github.com/rusty1s/pytorch_sparse. While this library and other similar ones handle the operational sparsity of the graphs, previous GNN-based approaches need to locally convert the sparse tensors to the corresponding dense format for their layer operations. In TESH-GCN, the conversion is not required because all operations in Sparse-HGCN are directly performed on the sparse tensor as it only considers the non-zero elements of the tensor. Each convolution operation moves up one-hop in the nodes’ neighborhood. Hence, the number of graph convolution layers should at least be the maximum shortest path between any two nodes in the graph. For a dataset, this is empirically calculated by sampling nodes from the graph and calculating the maximum shortest path between them. For the datasets in our experiments, we used 8 layers ($L=8$) to extract local neighborhoods in the early layers and metapath structures in the later layers. The main adjacency tensor can be split either over the number of semantic signals (D) or the number of edge types (K). We chose the latter because each adjacency tensor needed a separate GPU and it was more efficient and convenient to control the training process, given that the number of edge types is lesser than the number of semantic signals in our experiments. Algorithm 1 provides the pseudocode for training the model.

For the datasets, we select the following widely used publicly available network benchmark datasets where the nodes contain certain semantic information in the form of text attributes. Also, the choice of the datasets is driven by the diversity of their hyperbolicity to test performance on different levels of latent hierarchy (lower hyperbolicity implies more latent hierarchy).

1. 1.

   Amazon [35] is a heterogeneous e-commerce graph dataset that contains electronic products as nodes with title text connected by edges based on the purchase information. The edge types are also_buy (products bought together) and also_view (products viewed in the same user session).

2. 2.

   DBLP [36] is a heterogeneous relational dataset that contains papers, authors, conferences, and terms from the DBLP bibliography website connected by three edge types: paper-author, paper-conf and paper-term. For the semantic information, we include the paper’s titles, author’s names, conference’s names, and the terms’ text.

3. 3.

   Twitter [2] dataset is a user follower network graph with unidentifiable profile information given as node’s features. The node features are pre-encoded to remove sensitive identifiable information.

4. 4.

   Cora [37] is a citation graph that contains publications with title text and author information connected by citation edges.

5. 5.

   MovieLens [38] dataset is a standard user-movie heterogeneous rating dataset with three edge types: user-movie, user-user, and movie-genre. We utilize the movie’s title and genre’s name as the textual information.

More detailed statistics of the datasets such as the no. of nodes, edges, edge types, along with hyperbolicity and data sparsity are given in Table II.

We compare the performance of the proposed model with the following state-of-the-art models in the following categories: text-based (1-3), graph-based (4-6), and hybrid text-graph (7-9) approaches.

1. 1.

   C-DSSM [39] is an extension of DSSM [40] that utilizes convolution layers to encode character trigrams of documents for matching semantic features.

2. 2.

   BERT [17] is a popular transformer based language model that pre-trains on large amount of text data and is fine-tuned on sequence classification task for efficient text matching.

3. 3.

   XLNet [41] is an improvement over the BERT model which uses position invariant autoregressive training to pre-train the language model.

4. 4.

   GraphSage [7] is one of the first approaches that aggregate the neighborhood information of a graph’s node. It includes three aggregators mean, LSTM [42], and max pooling. For our baseline, we choose the best performing LSTM aggregator.

5. 5.

   GCN [8] utilizes convolutional networks to aggregate neighborhood information.

6. 6.

   HGCN [12] utilizes convolutional networks in the hyperbolic space that typically performs better than the Euclidean counterparts, especially, for datasets with low hyperbolicity (i.e., more latent hierarchy).

7. 7.

   TextGNN [13] initializes node attributes with semantic embeddings to outperform previous approaches especially for the task of link prediction.

8. 8.

   TextGCN [14] constructs a word-document graph based on TF-IDF scores and then applies graph convolution for feature detection towards link prediction between nodes.

9. 9.

   Graphormer [15] adds manually constructed global features using spatial encoding, centrality encoding, and edge encoding to the node vector to aggregate the neighborhood in a transformer network architecture for graph-level prediction tasks.

To analyze the performance of TESH-GCN, we compare it against the state-of-the-art baselines using standard graph datasets on the task of link prediction. We input the node-pairs ($v_{i},v_{j}$) with the corresponding text sequence ($s_{i},s_{j}$) to the model and predict the probability that an edge type $e_{k}$ connects them as $y_{k}=P_{\theta}(e_{k}|(v_{i},v_{j},s_{i},s_{j}))$. We evaluate our model using 5-fold cross validation splits on the following standard performance metrics: Accuracy (ACC), Area under ROC curve (AUC), Precision (P), and F-score (F1). For our experimentation, we perform 5-fold cross validation with a training, validation and test split of 8:1:1 on the edges of the datasets. Table IV provides the number of samples and sparsity of each split in the dataset. The results on the test set are presented in Table III.

From the experimental results, we observe that TESH-GCN is able to outperform the previous approaches by a significant margin on different evaluation metrics. Additionally, we notice that the performance improvement of hyperbolic models (HGCN and TESH-GCN) is more on datasets with lower hyperbolicity (higher latent hierarchy). This shows that hyperbolic space is better at extracting hierarchical features from the graph structures. Furthermore, we see that the performance decreases a little without the residual network. However, it does not justify the additional parameters but it adds robustness against noisy graph and text (evaluation in Section IV-F), so we use this variant in our final model. Another point of note is that text-based frameworks are better than graph approaches in datasets with good semantic information such as Amazon, whereas, graph-based approaches are better on well-connected graphs such as Cora. However, TESH-GCN is able to maintain good performance in both the scenarios, demonstrating its ability to capture both semantic and structural information from the dataset.

In this section, we study the importance of different components and their contribution to the overall performance of our model. The different components we analyze in our ablation study are: (i) the semantic text signal, (ii) the hyperbolic transformations, (iii) the residual network, and (iv) the multi-step loss. The ablation study is conducted on the same datasets by calculating the evaluation metrics after freezing the parameters of the component of interest in the model. The results of the study are presented in Table III.

The results show that the text signal contributes to $7\%$ performance gain in our model, implying the importance of utilizing the nodes’ semantic information in aggregating features from the adjacency tensors. The hyperbolic transformations lead to a $18\%$ increase in TESH-GCN’s performance, demonstrating the importance of hierarchical features in extracting information from graphs. This also provides additional evidence of the latent hierarchy in the graph networks. Furthermore, removing the residual network shows a decrease of $1\%$ in our model’s performance which shows that text signals capture the semantic signal in the graph convolution layers and the residual network works only towards increasing the robustness in the final link prediction task. In addition to this, we notice that replacing multi-step loss with a standard cross entropy loss (with non-existence of links added as another class) leads to a $2\%$ reduction in performance. This provides evidence for the advantages of conditioning link classification on link prediction (as in multi-step loss) compared to a standard multi-class loss function.

One of the major contributions of TESH-GCN is its ability to efficiently handle sparse adjacency tensors in its graph convolution operations. To compare its performance to previous graph-based and hybrid approaches, we analyze the space and time complexity of our models and the baselines. The space complexity is studied through the number of parameters and time complexity is reported using the training and inference times of the models. We compare the space and time complexity of our models using large graphs of different sparsity ratios ($R$) (by varying the number of edges/links on a graph with $10^{4}$ nodes). The different sparsity ratios considered in the evaluation are $1-10^{-r}~\forall r\in\llbracket 0,5\rrbracket$. Figure 8 and Table V shows the comparison of different GCN based models’ training time on varying sparsity ratios and inference times on different datasets, respectively. Table VI presents the number of parameters and space complexity of the different baselines in comparison to TESH-GCN.

From the time complexity analysis, we notice that TESH-GCN consistently takes much less training time than the other GCN-based and hybrid approaches in high sparsity graphs. This shows that the current GCN-based approaches do not handle the sparsity of the adjacency tensor. However, the overhead of specialized graph convolution layer in TESH-GCN leads to a poor time complexity for cases with high graph density ($R<0.9$). From the comparison of inference times, given in Table V, we notice that TESH-GCN’s inference time is comparable to the graph-based baselines and significantly lesser than hybrid baselines. Figure 9 provides the effect of sparsity on the inference time of our model and the baselines. We note that TESH-GCN is able to outperform other hybrid graph-text baselines and needs similar inference time as the baselines that only consider the local neighborhood of its nodes. TESH-GCN is faster for high sparsity graphs but the overhead of specialized graph convolutions takes more time than other baselines on high density graphs.

The space complexity analysis clearly shows that TESH-GCN uses much lesser number of model parameters than baselines with comparable performance. Also, the complexity shows the dependence of text-based approaches on only the textual sequence length, whereas, the graph based are dependent on the number of nodes. However, TESH-GCN is able to reduce the space complexity by a factor of the sparsity ratio and only consider informative non-zero features from the adjacency tensors, leading to a decrease in the number of trainable parameters.

To test the robustness of our model, we introduce varying levels of noise into the Amazon graph by (i) node drop: dropping n% percentage of nodes, (ii) text replacement: replacing n% percentage of the text, and (iii) hybrid noise: dropping n% of nodes and replacing n% of text. We compare the performance of our model and the baselines across different values of $n=10,20,30,40$, and $50$. The results for the robustness evaluation are given in Figure 7.

First, we highlight the main observations, that node drop and text replacement only affects graph-based and text-based approaches, respectively (and does not affect them vice versa). In the case of hybrid baselines, we still note a decrease in performance for both the noise variants. This implies that the text and graph features in the baselines do not complement each other. In the case of TESH-GCN, we note that both the noise variants do not cause any significant performance loss. This shows that the complementary nature of the semantic residual network and hyperbolic graph convolution network leads to an increased robustness against noise in either the text or graph. In the third scenario with hybrid noise, we see a reduction of $\approx 25\%$ performance in text-based and graph-based baselines and $\approx 50\%$ in hybrid baseline with a $50\%$ noise. However, we notice that, although TESH-GCN is a hybrid model, we only observe a $25\%$ performance loss with $50\%$ noise, implying the effectiveness of text-graph correspondence in the scenario of hybrid noise as well. Thus, we conclude that TESH-GCN is robust against noise in either graph or text, but vulnerable, albeit less than other hybrid baselines, to a joint attack on both the graph and text.

Model comprehension is a critical part of our architecture as it helps us form a better understanding of the results and explain the model’s final output. To understand TESH-GCN’s link prediction, we look at the different metapaths that connect the input nodes as well as the text in the metapaths’ nodes that receive the most attention ($\alpha_{k}$). For this, we follow the graph convolution and attention pooling operations through the layers in the network and extract the most critical metapaths chosen by the model to arrive at the prediction. The methodology for extracting the metapaths with their corresponding weightage in the final link prediction is presented in Algorithm 2. Figure 10 depicts some metapaths extracted from the Amazon dataset. In Figures 10(a) and 10(b), we note that TESH-GCN aggregates information from multiple long (4-hop) metapaths between the input nodes for prediction. Additionally, we see tokens in the node’s text being emphasized (having higher attention weight) based on the edges through which they propagate their semantic information, e.g., in Figure 10(b), we observe that key tokens: Pirates of the Caribbean and Necklace propagate the semantic information to match with additional relevant tokens such as Cursed Aztec, Medallion, Pendant and coin to establish the edge also_buy between the input nodes. Thus, we observe the role of different metapaths as well as semantic information in the message propagation towards the downstream task of link prediction.