Characterized Diffusion and Spatial-Temporal Interaction Network for Trajectory Prediction in Autonomous Driving (2024)

In order to predict the future trajectories of neighboring agents, the Characterized Diffusion Module treats trajectory prediction as the reverse process of motion Characterized Diffusion and gradually eliminates the uncertainty of future trajectories by learning a parameterized Markov chain with the observed historical states. More specifically, during the diffusion process, the uncertainty inherent in future trajectories is simulated by iteratively introducing Gaussian noise. Conversely, in the inverse diffusion process, this uncertainty is iteratively mitigated to derive the anticipated future trajectories accurately. The detailed procedure is shown in Fig. 3. Mathematically, let C be the future trajectory of the neighboring agents. Firstly, We initialize the diffused unit $\textbf{C}^{0}$:

We use a forward diffusion operation $f_{\textit{diffuse}}(\cdot)$ to add uncertainty to $\textbf{C}^{\delta-1}$ and transition to diffused unit $\textbf{C}^{\delta}$:

where $\textbf{C}^{\delta}$ is the diffused unit at the $\delta^{th}$ diffusion step, $\Gamma$ is the total number of steps.

After $n$ iterations, our model is able to capture a comprehensive spectrum of uncertain traffic scenarios with maximum coverage. Then, the inverse diffusion process is applied to accurately derive the future trajectories of neighboring vehicles. This step-by-step refinement ensures high fidelity in predicting vehicle movements, effectively addressing the inherent complexity of dynamic traffic scenarios. Due to the indeterminacy of future trajectories, in the denoising procedure, it is usually more reliable to capture more than one inverse unit to extract sufficient trajectory information. Therefore, we draw $K$ independent and identically distributed samples to initialize the denoising unit $\widehat{\mathbf{C}}_{k}^{\Gamma}$ from a normal distribution.

We formulate the trajectories generation process as a reverse diffusion by iteratively applying a denoising operation $f_{\textit{denoise}}(\cdot)$ to obtain the denoised unit $\widehat{\mathbf{C}}_{k}^{\delta}$ conditioned on historical states $\textbf{X}_{0}$, $\textbf{X}_{i}$ and, unit $\widehat{\mathbf{C}}_{k}^{\delta+1}$:

In a denoising module, two parts are trainable: a transformer-based context encoder $f_{\textit{context}}(\cdot)$ to learn a social-temporal embedding and an uncertainty estimation module $f_{\epsilon}(\cdot)$ to estimate the uncertainty to reduce. Mathematically, the $\delta^{th}$ denoising step works as follows:

where $\alpha_{\delta}$ and $\bar{\alpha}_{\delta}=\prod_{i=1}^{\delta}\alpha_{i}$ are parameters in the diffusion process and $\mathbf{z}\sim\mathcal{N}(\mathbf{z};\mathbf{0},\mathbf{I})$ is the uncertainty. We leverage a context encoder $f_{\text{context }}(\cdot)$ on historical states $\left(\mathbf{X},\mathbb{X}_{\mathcal{N}}\right)$ to obtain the context condition $\mathbf{C}_{encoder}$, and estimates the uncertainty $\epsilon_{\theta}^{\delta}$ in the uncertain trajectory $\widehat{\mathbf{Y}}_{k}^{\delta+1}$ through uncertainty estimation $f_{\epsilon}(\cdot)$ implemented by multi-layer perceptions with the context C; Eq. 7 provides a standard denoising step.

The final ${K}$ extracted units are $\widehat{\mathbf{C}}=\{\widehat{\mathbf{C}}_{1}^{0},\widehat{\mathbf{C}}_{2}^{$ Finally, we obtain the future trajectories with:

where $\Omega$ denotes a multi-layer perception with learnable parameter matrix $W_{\textit{cf}}$.

1) Temporal Encoder: To begin with, a temporal embedding vector $F^{t}$ is obtained from the historical states $x^{t}$ at the $t^{th}$ timestamp using a fully connected layer with a learnable parameter matrix $W_{\textit{emb}}$ as follows:

where $\delta$ is the fully connected layer and $\Phi$ is the LeakyReLU activation function. It can be defined as follows:

where $W_{init}$ denotes the learnable parameter matrix of encoder $f_{\textit{tem}}$ and $h^{t}$ denotes the temporal feature at the $t^{th}$ timestamp, which is updated at each timestamp based on the hidden state at the previous timestamp and the embedding vector at the current timestamp. Here, we apply $f_{\textit{tem}}$ to every single agent shared with the parameters of encoding $f_{\textit{tem}}$ to reduce variation across numerous agents at different timestamps. Finally, for the target agent, we obtain $H_{0}=[h^{-T_{p}+1}_{0},h^{-T_{p}+2}_{0},...,h^{0}_{0}]\in R^{T_{p}\times D}$ representing the temporal feature over $T_{p}$ timestamps, and $\bar{H}_{i}=[\bar{h}^{-T_{p}+1}_{i},\bar{h}^{-T_{p}+2}_{i},...,\bar{h}^{0}_{i}$ denoting the temporal feature of $i^{th}$ neighboring agents and $D$ denotes the number of hidden dimensions.

2) Spatial Encoder: Apparently, naive use of one temporal feature per agent does not capture agent-to-agent spatial relations. Therefore, capturing the spatial relations between agents in the same scene is highly necessary. Given the success of attention mechanisms in sequence-based prediction Hu (2020), we adopt a multi-head attention mechanism to obtain spatial relations between agents, which can be represented as follows:

where $f_{sp}$ is the spatial attention.

In a detailed elaboration, $Q=[{q^{-T_{p}+1},q^{-T_{p}+2},...,q^{0}}]$, $K=[k^{-T_{p}+1},k^{-T_{p}+2},...,k^{0}]$, $V=[v^{-T_{p}+1},v^{-T_{p}+2},...,v^{0}]$ respectively denote linear projected vectors query, key, value and $W_{q}$,$W_{k}$,$W_{v}$ indicate three learnable parameter matrices. We apply normalization $\Pi$ to query and key to represent the importance of agents influencing each other. Formally,

where $\omega$ is the attention score representing the similarity between $Q$ and $K$,$(\cdot)$ denotes the operation of matrix multiplication. Subsequently, we leverage attention scores to discern the significant connections among agents. Mathematically,

where $\upsilon$ is the output of the single-head attention. Compared to single-head, multi-head attention can comprehensively capture local and global spatial relations. Therefore, we apply the multi-head attention mechanism to obtain $\Upsilon=[\upsilon_{1},\upsilon_{2},...,\upsilon_{n}]$, carrying spatial relations. We introduce an innovative gating mechanism $H_{g}$ to control the importance of different heads and selectively amplify or suppress specific heads. This mechanism acts as a gatekeeper for the $\Upsilon$ by adjusting the activation level consisting of two linear layers as:

where $\sigma$ denotes the activation function sigmoid and $\odot$ denotes the multiplication of the corresponding elements of the matrix, $\kappa$ denotes the linear layer. Note that the output of the spatial encoder we simplify as follows:

3) ST Fusion: Following the spatial encoder, we introduce a ST Fusion $f_{\textit{ST}}$ to deeply capture spatial-interaction from $S$ produced by the preceding module. Formally,

where $\bar{Q}=[\bar{q}^{t}_{1},\bar{q}^{t}_{2},...,\bar{q}^{t}_{M}]$, $\bar{K}=[\bar{k}^{t}_{1},\bar{k}^{t}_{2},...,\bar{k}^{t}_{M}]$, $\bar{V}=[\bar{v}^{t}_{1},\bar{v}^{t}_{2},...,\bar{v}^{t}_{M}]$ respectively denote linear projected vectors query, key, value at the $t^{th}$ timestamp, and $\bar{W}_{q}$,$\bar{W}_{k}$,$\bar{W}_{v}$ indicate three learnable parameter matrices. In analogy with the spatial interaction module, we also use normalization and gating mechanisms to obtain long-term temporal features simplified as follows:

This study defines the trajectory prediction task as a conditional probabilistic prediction problem. Specifically, the decoder is designed to predict the future trajectory for the target agent based on different lateral and longitudinal maneuver classes:

where $P({\hat{Y}})$ is the conditional probabilistic distribution for the predicted trajectory. In detail, we use the LSTM decoder to implement the final multi-modal trajectory prediction.

where $\hat{y}^{t}$ is the predicted 2D spatial coordinate at future $t^{th}$ timestamp, $W_{decoder}$ denotes the parameter matrix to be learned in the LSTM. Based on the comprehensive feature vectors, we can only apply the simplest LSTM decoder to predict the accurate trajectory easily with fewer parameters.

We adopt a two-stage training approach to train our model. In the first stage, our model is trained to predict a future trajectory with the Mean Squared Error (MSE) Pan (2020) loss function as follows:

where $(\hat{y}_{x}^{t},\hat{y}_{y}^{t})$ is the 2D spatial coordinate predicted and $(y_{x}^{t},y_{y}^{t})$ denotes the corresponding ground-truth coordinate. It helps our model to learn the exact position information that approximates the true trajectory. When the model converges to a point using the MSE loss function, we use a Negative Log-Likelihood (NLL) Kim et al. (2020) loss function. This transition facilitates a more comprehensive exploration of uncertainty within the trajectory prediction.

where $\Delta_{x}^{t}$,$\Delta_{y}^{t}$ represents $(y_{x}^{t}-\hat{y}_{x}^{t})$, $(y_{y}^{t}-\hat{y}_{y}^{t})$, and $\sigma_{x}^{t}$ , $\sigma_{y}^{t}$ denote the standard deviation of x-coordinate and y-coordinate at the $t$th timestamp. $\rho_{xy}^{t}$ is the correlation coefficient between the $x$ and $y$ coordinates at $t$th timestamp. $P^{t}=\rho^{T}_{x}\rho^{t}_{y}\sqrt{1-(\rho_{xy}^{t})^{2}}$ is the standard deviation of the probability density function in $x$ and $y$ coordinate. $\alpha$ is an empirical constant value to simplify the calculation.