University of Oxford

Transformer Representations and the Geometry of Refusal

1 Introduction

The transformer has remained state of the art in natural language processing since its inception in Vaswani et al. (2017), but as the capabilities of frontier language models continue to accelerate, a growing body of work focuses on ensuring models cannot be used to cause harm Bai et al. (2022); Askell et al. (2021). Further, another line of work develops techniques to interpret a transformer’s internal representations Bereska and Gavves (2024), enabling both the suppression of unwanted behaviour and the elicitation of desired behaviour.

A key object of study in this document is the refusal behaviour of an instruction-tuned language model, where due to the nature of the training process, a model declines to produce content considered harmful. This assimilates into the model giving a generic response such as “I can’t”, when a harmful request is made. In order to study refusal, we explore transformer representations in a principled and geometric fashion, denoting the spaces that they occupy as manifolds.

In part I we define two classes of manifold: the embedding manifold ${ℳ}_0$ which is the space of input representations, and the residual manifolds ${ℳ}_i$ which are the internal representation spaces after each decoder block in a transformer. In part II, we survey techniques exposing semantic directions in the manifolds, and then study a darker-side, where model exploits live and harmful behaviour is produced. We contribute a Gaussian perturbation, which we use to study refusal behaviour. Our key finding is that in earlier manifolds the intention to refuse is not consolidated, and can be disrupted by our perturbation.

We conclude with part III, where we introduce a modern generative technique for learning the manifold distributions. We use the trained models of Luo et al. (2026) to contribute a likelihood-based manifold metric which we show can separate on-and-off manifold points. We then finish by using our metric to study representation steering, showing that steering in the direction of a vector from a sparse autoencoder affects manifoldness more than steering in a random direction.

Code, Compute, and Figures.

In this document figures from external sources are cited in their caption. Further, the code and datasets used to generate the core experiments are available on Github, ManifoldVulnerability. Lastly, our compute budget for this dissertation was $40.

Part I Preliminary Material

2 The Transformer Architecture

Interacting with a language model through a provider creates the illusion that a language model maps text to text; in reality, multiple stages of computation occur before a text response is created. In this section, we explain each stage of this process, focusing on the details most relevant to future sections. We follow the implementation described for the Llama family of models Grattafiori et al. (2024).

In our exposition, we split the computation into three stages: translation of text to numerical representation, the application of the transformer to create a distribution over the next token, and then sampling from the distribution of tokens.

2.1 The Input to the Transformer

Prior to training, a vocabulary is chosen which is a set of pre-defined tokens. Most tokens are short strings of characters such as “king”. We denote the size of the vocabulary $d_{\mathrm{vocab}}$ which is typically on the order ${10}^5$. Both the size and choice of vocabulary vary between models, but each token in the vocabulary is unique and has its own token-ID $\in [1,d_{\mathrm{vocab}}]$. There is also a number of special tokens which each have a unique purpose.

For a prompt of length $l$ tokens, the first step of the embedding process creates a sequence of $l$ one-hot vectors in ${ℝ}^{d_{\mathrm{vocab}}}$ such that each individual vector corresponds to a segment of the prompt. This sequence is stored in a matrix

$$𝐃:={[{𝐞}_1,{𝐞}_2,\mathrm{\dots },{𝐞}_l]}^{\top }\in {ℝ}^{l\times d_{\mathrm{vocab}}}.$$

The tokenization process creates the sequence such that if each ${𝐞}_i$ was mapped to its corresponding text: the input string would be recreated exactly.

Prior to training, a parameter $d_{\mathrm{model}}$ is chosen with ; typically, $d_{\mathrm{model}}$ is on the order ${10}^4$. During training, a vector representation in ${ℝ}^{d_{\mathrm{model}}}$ is learnt for each token in the model’s vocabulary; combining each representation into a matrix results in $𝐄\in {ℝ}^{d_{\mathrm{vocab}}\times d_{\mathrm{model}}}$ which we refer to as the embedding matrix. Note that representations are stored in rows. The matrix $𝐄$ is used to embed $𝐃$ into ${ℝ}^{l\times d_{\mathrm{model}}}$, which is achieved with the multiplication $\mathrm{𝐃𝐄}:={𝐏}_0$. This concludes the embedding process.

To recap, a representation for a prompt is created using tokens, each token is then embedded into ${ℝ}^{d_{\mathrm{model}}}$ using the map $𝐄$, resulting in a matrix ${𝐏}_0\in {ℝ}^{l\times d_{\mathrm{model}}}$. Positional embeddings are absent in this exposition as the Llama series of models uses RoPE Su et al. (2024), which instead encodes position within the attention mechanism.

2.2 The Transformer Stage

In the next stage of generation, a transformer maps the embedding of a prompt ${𝐏}_0$ to a set of logits, which are in essence an unnormalised probability distribution.

We begin with the architecture of a transformer, which for the Llama family consists of a series of decoders which are connected via residual connections. We denote each decoder with ${\mathrm{dec}}_i$ with $i\in [1,i_{\max }]$. Here, $i_{\max }$ is the total number of decoders; which for the Llama3.2-1B-model is $16$. The support of each decoder is

$$ℐ:=\bigcup _{k=1}^{l_{\max }}{ℝ}^{k\times d_{\mathrm{model}}},$$ (1)

and the decoder is designed such that the matrix dimension of the output matches that of the input. The key steps in a decoder are an attention mechanism, followed by a variant of an MLP layer. The details of attention are vital for the success of transformers, but fall outisde the scope of this project. As a result, we give a minimal presentation and refer to Vaswani et al. (2017) for a detailed exposition.

The Decoder.

We describe the decoder by its action on the rows of an input matrix. Let ${𝐏}_0$ be the input matrix, we define

$${𝐏}_i:={\mathrm{dec}}_i\circ \mathrm{\cdots }\circ {\mathrm{dec}}_1({𝐏}_0)$$ (2)

as the output of the $i^{\text{th}}$ decoder layer, so each ${\mathrm{dec}}_i$ maps ${𝐏}_i\to {𝐏}_{i+1}$. We express the $j^{\text{th}}$ row of ${𝐏}_i$ with ${𝐩}_j^{(i)}$. The first sub-layer of each decoder block applies a self-attention mechanism. We let ${\mathrm{Attn}}^{(i)}$ denote the attention mechanism in decoder $i$, with this, the first step reads

$${\widetilde{𝐩}}_j^{(i)}={𝐩}_j^{(i)}+{\mathrm{Attn}}^{(i)}\left(\mathrm{RN}({𝐏}_i)\right),$$ (3)

where RN denotes the RMSNorm which is a contemporary layer normalisation technique Zhang and Sennrich (2019). While the attention in eq. 3 is a function of ${𝐏}_i$, an attention mask ensures that each row depends only on preceding rows.

In the second sub-layer, a SwiGLU feed-forward layer is applied Shazeer (2020), which is a gated variant of a standard MLP layer. This acts row-wise, and can be expressed

$${𝐩}_j^{(i+1)}={\widetilde{𝐩}}_j^{(i)}+{\mathrm{SwiGLU}}^{(i)}\left(\mathrm{RN}({\widetilde{𝐩}}_j^{(i)})\right).$$ (4)

This completes on decoder layer, as concatenating each row vector ${𝐩}_j^{(i+1)}$ forms ${𝐏}_{i+1}$.

Stacking these decoders together forms the bulk of the transformer, but we are yet to produce the logits. These are created in a final stage called un-embedding which occurs after the final decoder. To create the logits the final row vector ${𝐩}_l^{(i_{\max })}$ is transformed with a learnt matrix ${𝐖}_U\in {ℝ}^{d_{\mathrm{vocab}}\times d_{\mathrm{model}}}$, with

$$𝐝={𝐖}_U\mathrm{RN}({𝐩}_l^{(i_{\max })})\in {ℝ}^{d_{\mathrm{vocab}}}.$$ (5)

Notice that this is currently un-normalised, to normalise the soft-max function is applied. In the next section, we explain the process of sampling from the created distribution.

2.3 Generating a full Response

To produce a response to a prompt, we must sample tokens from the distribution output from the transformer. Once a token is sampled, it’s representation is appended to ${𝐏}_0$ and re-input to the transformer, which creates the next set of logits. This process is repeated until a special end-token is selected, halting generation.

Due to attention masking, the process of re-inputting a sampled token does not require each ${𝐩}_j^{(i)}$ to be recalculated, as each representation depends only on the tokens at position before it. As a result, previous representations are unaffected by each new token, and the next logit distribution can be computed quickly by caching representations.

Let us now focus on the sampling procedure. Modern implementations use nucleus sampling Holtzman et al. (2019) with a temperature scale; which introduces two hyper-parameters $p$ and $\tau$. In nucleus sampling the support is truncated to the smallest set of normalised logits with mass exceeding $p$, thus more aggressive truncation occurs when mass is shared between a small subset of tokens. Temperature scaling further skews the logit distribution by applying $\mathrm{logits}\to \mathrm{softmax}(\frac{\mathrm{logits}}{\tau })$.

Before continuing, we provide a last caveat which applies to instruction-tuned language models.

2.3.1 Prompt Formatting

In instruction-tuned models the user’s text prompt is inserted into a standard format before being embedded. The format contains a number of special tokens which denote the start and end of sections within the prompt. Figure 1 gives a visualisation of the format used in the Llama series of models.

<|begin_of_text|><|start_header_id|>system<|end_header_id|>
You are a helpful assistant<|eot_id|>
<|start_header_id|>user<|end_header_id|>
What is the capital of France?<|eot_id|>
<|start_header_id|>assistant<|end_header_id|>

Figure 1: Blue denotes the user message, grey denotes the system prompt, and <|begin_of_text|>, <|start/end_header_id|> <|eot_id|> are special tokens.

3 Defining the Residual and Embedding Manifolds

The main objects of study in this document are what we call the embedding and residual manifolds; broadly speaking, these are the regions of space which contain the representations which produce a coherent output from the transformer. Each individual manifold ${({ℳ}_i)}_{i=1}^{i_{\max }}$ contains representations at a certain depth into the transformer.

Recall that the transformer supports inputs from the set $ℐ$, defined in eq. 1. We define the function

$$f_i:ℐ\to {ℝ}^{d_{\mathrm{vocab}}},$$

which is the composition of each decoder, from ${\mathrm{dec}}_{i+1}$ onward to the logits. The manifold ${ℳ}_i$ contains the matrices in $ℐ$ which result in a coherent output when used as input to $f_i$. Here, “coherence” means the response contains contextually relevant natural language relating to the prompt. We call ${ℳ}_0$ the embedding manifold, which contains the matrices which produce a coherent output when input to the entire transformer $f_0$, and we call each ${ℳ}_{i\ge 1}$ a residual manifold.

We deliberately omit a formal definition of the manifolds which would require defining coherence and contextually relevant. Instead, we treat ${ℳ}_i$ as an informal but geometrically meaningful object: the region of space where the transformer operates as intended, and whose structure we study empirically.

Due to the discrete nature of the tokenisation process section 2.1, calling ${ℳ}_0$ a manifold may appear inappropriate, as you may expect that the row-vectors in ${𝐏}_0\in {ℳ}_0$ are row-vectors of $𝐄$ also. This would make ${ℳ}_0$ a finite set, and thus a point cloud. Contrary to this intuition, the transformer learns a more universal representation, and in section 5.2 we demonstrate techniques that find matrices with rows not from $𝐄$, which produce coherent outputs. Nonetheless, the null hypothesis that ${ℳ}_0$ is a point cloud is relevant, and we define ${ℰ}_0\subset {ℳ}_0$ as the subset of the embedding manifold containing matrices which have rows from $𝐄$.

Before asking a motivating question, we define the row-wise manifolds which we study later in this document,

$${ℛ}_i:=\{𝐩\in {ℝ}^{d_{\mathrm{model}}}\mid 𝐩\text{is a row-vector of}𝐏\in {ℳ}_i\}.$$

Thus, the manifold ${ℛ}_i$ contains the row-vectors from matrices in ${ℳ}_i$.

3.1 A Motivating Question

Let us consider a simple perturbation applied to a transformer representation in ${ℳ}_0$. Recall ${𝐏}_0$, from section 2.1, which is the embedding of a text prompt. When ${𝐏}_0$ is input to the transformer a coherent output is produced, and thus ${𝐏}_0\in {ℳ}_0$. Further, the rows vectors of ${𝐏}_0$ are in $𝐄$, so

$${𝐏}_0\in {ℰ}_0\subset {ℳ}_0.$$

We consider the effect of perturbing ${𝐏}_0$ by adding Gaussian noise, denoting this matrix ${\widetilde{𝐏}}_0$ with

$${\widetilde{𝐏}}_0={𝐏}_0+\mathrm{\Delta }{𝐏}_0,\text{where}{(\mathrm{\Delta }{𝐏}_0)}_{ij}\sim 𝒩(0,{\sigma }^2).$$ (6)

Now, the row vectors of ${\widetilde{𝐏}}_0$ are not in $𝐄$ so ${\widetilde{𝐏}}_0\notin {ℰ}_0$, however, ${\widetilde{𝐏}}_0\in ℐ$ so it can be input into the transformer. As the representation has been altered, we ask whether ${\widetilde{𝐏}}_0$ would result in a coherent output when input to the transformer, or equivalently whether $\widetilde{{𝐏}_0}\in {ℳ}_0$. In section 6 we answer this question, and make an interesting connection to the refusal mechanism.

Part II Representation, and Vulnerability in the Manifolds

4 Geometry and Semanticity of the Manifolds

Having introduced the manifolds, we now begin our study of them. In this section, we start with a simple geometric picture of the row-wise manifolds ${ℛ}_i$ providing geometric intuition for what transformer representations look like. We then survey techniques which demonstrate that directions corresponding to semantic concepts can be extracted from vectors in ${ℛ}_i$. Afterward, we explore a darker-side of the manifolds ${ℳ}_i$: the areas of space which can elicit targeted harmful behaviour from the transformer. We conclude by demonstrating that earlier manifolds are uniquely vulnerable to an untargeted Gaussian perturbation, which connects the geometric, semantic, and adversarial threads of this part.

4.1 Size and Shape

To illustrate how the size of representations varies with depth into the model, we study the row-wise manifolds ${ℛ}_i$ of the Llama3.2-1B model. To study the distributions, we sampled $10,000$ texts from the brown text corpus Kučera et al. (1967), and input them to the model, storing the internal representations. Figure 2 shows the box and whisker plots of the euclidean norm distribution of vectors in ${ℛ}_i$; we observe that the size of representation increases with depth. In the initial layers, the size of distribution increases gradual, however after ${ℛ}_8$ the size of the norm distribution increase rapidly.

Figure 2: Euclidean Norm Distribution

4.2 Interpretable Directions

Having given a first picture of the geometric properties of the manifolds we shift our focus to the existence of semantically meaningful directions. We motivate this with an example, in a similar context to the space of transformer embeddings ${ℛ}_0$, Mikolov et al. (2013) discovered that the embeddings of the words: “queen”, “king”, “man”, and “ woman” have a distinct relationship. The difference between the representations of “queen” and “king” is similar to the difference between “man” and “woman”. Park et al. (2023) show that this property extends to transformer representation in ${ℛ}_0$, thus in our own notation

$$𝐄\left({𝐞}_{\text{queen}}-{𝐞}_{\text{king}}\right)\approx 𝐄\left({𝐞}_{\text{woman}}-{𝐞}_{\text{man}}\right).$$ (7)

This suggests that the vector $𝐄({𝐞}_{\text{queen}}-{𝐞}_{\text{king}})\in {ℝ}^{d_{\mathrm{model}}}$ encodes the semantic concept of gender, and that the direction is consistent across relevant representations.

4.2.1 Linear Techniques

The method of eq. 7, which studies the difference between two vector representations has been expanded upon, and extended to deeper manifolds ${ℛ}_{i\ge 1}$. For instance, a growing body of works scales eq. 7 by studying the average difference between multiple representations Panickssery et al. (2023), which is named contrastive activation addition. A contrastive pair

$$({𝐱}^{+},{𝐱}^{-})\in {ℛ}_i\times {ℛ}_i$$

contains embeddings of positive and negative examples of a concept; if the concept was harmfulness, then ${𝐱}^{+}$ could be an embedding from a harmful prompt, and ${𝐱}^{-}$ from a harmless prompt. Formally, the method of Panickssery et al. (2023) extracts a concept direction using a dataset of contrastive pairs ${\{({𝐱}_j^{+},{𝐱}_j^{-})\}}_{j=1}^N\subset {ℛ}_i\times {ℛ}_i$, by taking the difference of mean direction

$$\widehat{𝐬}=\frac{{𝝁}^{+}-{𝝁}^{-}}{{\Vert {𝝁}^{+}-{𝝁}^{-}\Vert }_2},{𝝁}^{\pm }=\frac{1}{N}\sum _{j=1}^N{𝐱}_j^{\pm }.$$ (8)

In Panickssery et al. (2023) contrastive pairs uncover numerous semantic directions, corresponding to concepts such as sycophancy and hallucination. Other work has developed similar contrastive techniques, such as Marks and Tegmark (2023) who study the representation of true and false statements. Pipelines which automate the contrastive pairs technique have also been developed, by Chen et al. (2025). Further, more advanced linear techniques have been used to study the vector distribution of contrastive pair differences ${𝐱}^{+}-{𝐱}^{-}$. For instance, Zou et al. (2023a) study the principle components of the differences.

4.2.2 The Superposition Hypothesis

Having demonstrated that semantic directions exist within the manifolds, we now reason about what we actually mean by a semantic direction, and how many such directions can exist. A key concept here is polysemanticity Olah et al. (2017), which refers to the observation that a single direction in representation space can simultaneously correspond to multiple unrelated semantic concepts. In contrast, a monosemantic direction represents a single semantic concept. This spawned the superposition hypothesis Elhage et al. (2022), which states that far more interpretable directions exist than the model dimension $d_{\mathrm{model}}$ allows for; equivalently, interpretable semantic directions are not orthogonal. However, if directions are not orthogonal, it is natural to ask how the linear operations in the transformer are able to separate them. Under the superposition hypothesis, vectors in ${ℛ}_i$ are argued to be sparse linear combinations of monosemantic features, where the number of features far exceeds $d_{\mathrm{model}}$. Due to the geometry of high dimensional real vector spaces, large sets of approximately orthogonal vectors can exist Luisto (2025), and thus linear operators are able to approximately separate them.

Inspired by the superposition hypothesis, sparse autoencoders were developed to recover the overcomplete set of monosemantic directions. The approach has been shown to scale Templeton (2024), and the recovered directions are empirically more monosemantic under a range of evaluation metrics Cunningham et al. (2023).

Sparse Autoencoders (SAE).

A SAE uncovers an over-complete set of vectors

$$𝐕\in {ℝ}^{n_{\text{feat}}\times d_{\mathrm{model}}}\text{with}{n}_{\text{feat}}>d_{\mathrm{model}},$$

which form an approximate sparse basis for vectors in the residual manifold ${ℛ}_i$. Here, sparsity refers to the property that on average any vector from ${ℛ}_i$ will have strong correlation with a small proportion of vectors in the basis $𝐕$; thus, having almost zero correlation with the majority of vectors.

SAEs uncover the basis $𝐕$ through a reconstruction task with a sparsity penalty Cunningham et al. (2023). Specifically, the basis $𝐕$ is learnt through optimisation of an autoencoder

$$g:{ℝ}^{d_{\mathrm{model}}}\to {ℝ}^{d_{\mathrm{model}}},𝐱\to {𝐕}^{\top }\mathrm{ReLU}(\mathrm{𝐕𝐱}+𝐛)$$

on an objective which is a mixture of a reconstruction and sparsity penalty

$${\Vert 𝐱-g(𝐱)\Vert }_2^2+\alpha {\Vert \mathrm{ReLU}(\mathrm{𝐕𝐱}+𝐛)\Vert }_1.$$ (9)

The bias $𝐛$ is also optimised during training. The sparsity penalty ensures that the representation $\mathrm{ReLU}(\mathrm{𝐕𝐱}+𝐛)\in {ℝ}^{d_{\mathrm{model}}}$ is sparse, and thus $𝐱$ correlates with only a small proportion of vectors in $𝐕$.

It is shown in Cunningham et al. (2023), that the uncovered features are more monosematic than the features learnt with other dictionary learning techniques, such as PCA, and they follow an automated approach for evaluating monosematicity proposed in Bills et al. (2023).

So far we have demonstrated that despite the complexity inherent to transformer representation, certain desirable and interpretable properties exist. We now investigate how these properties can be leveraged, revealing a darker-side of the manifolds.

5 A Darker-side of the Manifolds

5.1 Attacking ${ℳ}_{i\ge 1}$: Evil Persona Steering

Manifold steering is a technique used to alter a models generation behaviour by editing the internal representation during a forward pass. Steering has been shown to elicit a broad range of traits during generation, such as making a model more likely to talk about weddings Turner et al. (2023), make generations more “evil” Chen et al. (2025), and influence refusal behaviour Rimsky et al. (2024).

Steering adds a scaled vector $\widehat{𝐯}$ to a model’s representation in ${ℳ}_i$. The exact manner of application varies across implementations; however, any targeted vector $𝐩\in {ℛ}_i$ undergoes the transformation

$$𝐩\to 𝐩+\alpha \widehat{𝐯},$$ (10)

where $\alpha \in {ℝ}_{\ge 0}$ is the steering-coefficient. In some implementations, each vector in a representation $𝐏\in {ℳ}_i$ is targeted, whereas in others only the final row is targeted. To elicit a specific model behaviour, the method of contrastive pairs can be used to find a steering vector $\widehat{𝐯}$ for a target concept. Additionally, vectors from the basis of a sparse autoencoder are another common choice. A detailed comparison of different steering techniques is presented in Wu et al. (2025).

The effectiveness of steering varies depending on which manifold it is applied in. Figure 3, from Chen et al. (2025), illustrates how steering performance varies depending on the choice of $\alpha$, and ${ℳ}_i$.

Figure 3: Persona steering performance Chen et al. (2025).

The fact that generations remain coherent despite perturbation to the transformer’s internal representation reveals that the residual manifolds ${ℳ}_{i\ge 1}$ may be invariant to small moves in certain linear directions. Thus, locally ${ℛ}_i$ may be a lower dimensional surface embedded in a larger vector-space. However, it has been observed that when the steering coefficient is too large, the capability of the model begins to be reduced Turner et al. (2023); Rimsky et al. (2024), with the decrease in performance varying with targeted concept direction.

5.2 Attacking ${ℳ}_0$: Automated Suffix Finding

A suffix attack is a jail-breaking technique which concatenates a harmful suffix $𝐒\in {ℰ}_0$ to a prompt $𝐌\in {ℰ}_0$ so that the resulting input,

$$\left[\begin{array}{c}𝐌\\ 𝐒\end{array}\right],$$

bypasses the models refusal mechanism. A suffix is said to be general, if it bypasses the refusal mechanism when used as a suffix to multiple separate prompts. Hand-crafted suffixes became more difficult to create as model robustness improved Yi et al. (2024), this spawned algorithms which automate the process of finding a suffix.

Greedy coordinate descent (GCG) Zou et al. (2023b) automates the process of finding harmful suffixes by using an efficient discrete search over the set of possibilities. However, the GCG algorithm was improved upon in Schwinn et al. (2024), where the authors treat ${ℳ}_0$ as continuous, and create an algorithm finding a suffix in ${ℳ}_0\setminus {ℰ}_0$. A finding which illustrates an interesting property of ${ℳ}_0$.

5.2.1 Greedy Coordinate Descent

A differentiable objective measuring jail-breaking success is the first step towards automated suffix finding algorithms. Recall from section 2.2 that the transformer outputs a distribution over tokens in its vocabulary. As a result, the likelihood of any given token sequence is tractable. Let $𝐒\in {ℰ}_0$ denote a suffix of length $l_{\mathrm{suffix}}$ with

$$𝐒={[{𝐬}_1,{𝐬}_2,\mathrm{\dots }{𝐬}_{l_{\mathrm{suffix}}}]}^{\top }.$$

Consider a harmful request and response pair $(𝐗,𝐘)\in {ℰ}_0\times {ℰ}_0$, with

$$𝐘={[{𝐲}_1,\mathrm{\dots }{𝐲}_{l_{\mathrm{resp}}}]}^{\top },$$

where $𝐗$ is the embedding of a harmful prompt, and $𝐘$ is a contextual answer complying with the prompt. The likelihood of the harmful output $𝐘$ given the concatenation of the prompt and suffix can be used as a jail-breaking loss objective; where minimising the objective maximises the likelihood of the harmful response $𝐘$ being generated. We denote this loss function

$${ℒ}_{\mathrm{coop}}({𝐬}_1,{𝐬}_2,\mathrm{\dots }{𝐬}_{l_{\mathrm{suffix}}};𝐗,𝐘):=-\sum _{m=0}^{l_{\mathrm{resp}}-1}\log \mathrm{\ell }({𝐲}_{m+1}\mid 𝐗,𝐒,{𝐘}_{0:m}).$$ (11)

The GCG algorithm built upon work in Shin et al. (2020); Wallace et al. (2019), by iteratively improving a suffix ${𝐒}_t$ by replacing tokens in the suffix from a set of promising candidate substitutions. The set of candidate substitutions are calculated efficiently using the gradient of ${ℒ}_{\mathrm{coop}}$ with respect to the token embeddings. This process avoids searching over every token which has complexity that is exponential in $d_{\mathrm{vocab}}$.

In GCG, the suffix is initiated with

$${𝐒}_0=\mathrm{Embedding}(\text{“! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ”})$$ (12)

Then at each step of optimisation, GCG calculates a set of promising token substitutions for each ${𝐬}_i$ by taking the top-$k$ positive directions of the gradient

$${\nabla }_{𝐳}{ℒ}_{\mathrm{coop}}({𝐬}_1,{𝐬}_2,\mathrm{\dots },{𝐬}_{i-1},{𝐄}^{\top }𝐳,\mathrm{\dots },{𝐬}_{l_{\mathrm{suffix}}};𝐗,𝐘)\in {ℝ}^{d_{\mathrm{vocab}}}.$$ (13)

This is as the component of eq. 13 in the basis direction ${𝐞}_i$ is a proxy for the effect substituting ${𝐬}_i$ with ${𝐄}^{\top }{𝐞}_i$ has on the likelihood of the harmful response.

Once the promising substitutions are identified, a randomised search occurs, which creates a set of $B$ candidate suffixes with substitutions from the $k$ promising directions. Each of the $B$ suffixes are then evaluated using a forward pass, and the most performant substitution is chosen. We refer to Zou et al. (2023b) for the presentation of the full GCG algorithm.

5.2.2 Finding Suffixes in ${ℳ}_0\setminus {ℰ}_0$

Despite the GCG algorithm using gradients to identify promising token substitutions, its implementation requires $B$ forward passes at each time step. Further, gradients are taken with respect to $d_{\mathrm{vocab}}$ which is much greater than $d_{\mathrm{model}}$. The authors of Schwinn et al. (2024) improve both the cost and performance of the GCG algorithm by treating ${ℳ}_0$ as continuous.

They employee signed-gradient descent to search for a suffix in ${ℝ}^{l_{\mathrm{suffix}}\times d_{\mathrm{model}}}$, instead of in ${ℰ}_0$. Thus, they initiate the suffix ${𝐒}_0$ with eq. 12 following GCG, and then optimises at each step using

$${𝐒}_{t+1}={𝐒}_t+\alpha \cdot \mathrm{sign}\left({\nabla }_{{𝐒}_t}{ℒ}_{\mathrm{coop}}({𝐒}_t;𝐗,𝐘)\right).$$ (14)

This differs to GCG as the entire suffix is updated, as the dimension of the gradient is

$${\nabla }_{𝐒}{ℒ}_{\mathrm{coop}}(𝐒;𝐗,𝐘)\in {ℝ}^{l_{\mathrm{suffix}}\times d_{\mathrm{model}}}.$$ (15)

The authors report that signed gradient descent produced the most stable results out of a handful of standard continuous optimisation techniques. They use the step-size $\alpha =0.001$ and typically run their algorithm for $20$ steps. Figure 4 compares both the speed, and attack success rate of their algorithm to a number of baselines including GCG in a jailbreaking task.

Figure 4: Comparison of Jailbreaking Methods Schwinn et al. (2024).

Studying fig. 4, we see that the continuous method, denoted Embedding, has a $100\%$ success rate across all models, far exceeding GCG. Further, the attack is computed much faster. The additional methods compared to are AutoDan Liu et al. (2023), PAIR Chao et al. (2025), and Adaptive Andriushchenko et al. (2024).

5.3 Refusal on the Manifold

It is shown in Arditi et al. (2024) that a direction corresponding to refusal is present in many open-source language models, and that steering and ablating this direction affects the model’s tendency to refuse. The authors search for a generalist refusal direction in ${ℝ}^{d_{\mathrm{model}}}$ using a contrastive pairs technique, computing candidate directions across separate ${ℛ}_i$, and then select the single direction which best generalises across manifolds.

To validate that $𝐫$ corresponds to refusal, they demonstrate that steering in the direction $𝐫$ increases the model’s refusal likelihood. Further, when the direction$𝐫$ is ablated from representation, by applying the householder reflector $𝑰-\widehat{𝐫}{\widehat{𝐫}}^{\top }$, the likelihood of refusal decreases. They also connect the refusal direction $𝐫$ to the adversarial suffixes in section 5.2, by comparing the cosine similarity between representation and the refusal direction $𝐫$ for prompts with and without the harmful suffixes from section 5.2 appended. Specifically, they study the similarity at the final token before generation.Figure 5 shows this similarity for harmful and harmless prompts, where harmful + adv_suffix corresponds to harmful prompts with a suffix from Schwinn et al. (2024) appended.

Figure 5: Refusal direction correlation Arditi et al. (2024).

The addition of the suffix suppresses the component of the final token representation in the direction of $𝐫$, which partly reveals the mechanism which the suffix attack exploits. We also note that the correlation with $𝐫$ direction is greatest towards the middle of the model, suggesting that refusal is consolidated in later layers, a finding that we will return to in section 6.3.2.

Having surveyed techniques which demonstrate a darker side of the manifolds, we now introduce our own contribution: an untargeted perturbation also can bypass the refusal mechanism, and further illustrates that refusal intention is not consolidated until later layers.

6 Vulnerability through Perturbation to the Manifold

6.1 A Simple Gaussian Perturbation

Inspired by the techniques on the darker side of the manifolds, and our motivating question section 3.1, we study an untargeted perturbation, investigating how it interacts with both coherence and refusal. The simplicity of our perturbation contrasts with the complexity of other perturbations, such as steering and suffix finding.

The perturbation that we introduce adds scaled Gaussian noise to a target row vector $𝐩$, applying

$$𝐩\to 𝐩+\alpha {\Vert 𝐩\Vert }_2\cdot 𝐧,𝐧\sim 𝒩(0,{𝑰}_{d_{\mathrm{model}}}),$$ (16)

where $\alpha \in {ℝ}_{\ge 0}$ is a scale parameter. We refer to this as the Gaussian perturbation. By scaling relative to the norm of $𝐩$ the perturbation remains comparable across manifolds; recall from section 4.1 that the average norm of vectors varies across ${ℛ}_i$.

We apply eq. 16 in three separate ways, which we name the : suffix, embedding and residual types. The suffix perturbation mirrors the setup in section 5.2 applying the perturbation to the suffix eq. 12 in ${ℳ}_0$, whereas the embedding perturbation applies the perturbation directly to the user question, also in ${ℳ}_0$. The residual perturbation type applies eq. 16 directly to the user question in ${ℳ}_7$; table 1 provides a visualisation of each perturbation type.

Type	Target Area (red)	Location
Suffix	<|format|> User question? ! ! ! ! ! ! ! ! ! ! <|format|>	${ℳ}_0$
Embedding	<|format|> User question? <|format|>	${ℳ}_0$
Residual	<|format|> User question? <|format|>	${ℳ}_7$

Table 1: Perturbation types and their target area, and location.
<|format|> denotes the prompt format which is unaffected by the perturbations.

To guide intuition on the affect of the Gaussian perturbation: table 2 provides example generations for the prompt “What is the capital of France?” at two different perturbation scales.

Type	Scale ($\alpha$)	Generation
Suffix	0.02	It seems like you’re excited about France… While the capital of France is actually Paris, I’ll let you in on a little secret: there are actually multiple cities named Paris…
	0.05	It seems like you’re excited about the capital of France, but I’m afraid I need to break it to you that there is no exclamation mark… “La capitale de la France”…
Embedding	0.02	The capital of France (the French equivalent of France is called Paris
	0.05	##Step ##Strat 2012 OnMaradresseSame Daye Date: 2026
Residual	0.02	A simple question… I wandasruein smarul offinin… Not!..alldc.m<#^ Appropri is enally.
	0.05	there.PH/react_up:glldie’s last overcome havoc support… Diffusional nat*… “Timefall”…

Table 2: Responses across Type and Scale.

The generations from the residual perturbation is an initial answer to the question section 3.1, as the responses in coherent at $\alpha =0.02$, but not at $0.05$. Compared to the suffix perturbation, the coherence of the responses for the embedding and residual types decreases significantly more from the scale $0.02$ to $0.05$. This illustrate the sensitivity of the manifolds to small perturbation.

6.2 Perturbation and Coherence

In this section we study how the Gaussian perturbation effects the quality of generation at a larger scale. We use the proxy of correctness, and study the question and answering performance of the Llama3.2-1B-instruct model. Our key finding is that performance varies significantly across perturbation type, but in general decreases with scale.

Setup.

We created a dataset of $25$ questions designed to have unambiguous answers, table 3 shows an example. To guide the model to answer each question concisely, we add the prefix “Answer the following question using a single word.” to each question.

To evaluate the effect of perturbations on generation quality we calculate the likelihood of the correct answer, as well as the likelihood of an incorrect but contextually consistent answer. To ensure the likelihood calculation reflects the true generation probability, we calculate the likelihood using the nucleus and top-p scaling introduced in section 2.3, using the hyper-parameters $p=0.95$, and $t=0.8$. To normalise the likelihood for the length of phrase we take the geometric mean.

We adopt a likelihood-based methodology as it has been shown to accurately reflect true generation probability, further it provides insight into the objective optimised by suffix attacks, as the ${ℒ}_{\mathrm{coop}}$ loss in section 5.2 is a negative log-likelihood over the target response. Studying likelihood therefore allows us to probe the same quantity that gradient-based attacks exploit, while remaining tractable across the large number of prompts and noise draws required.

Prompt	Correct Response	Incorrect Response
What is the most abundant element in the universe?	Hydrogen	Oxygen

Table 3: Prompts with correct and incorrect one-word responses

In this experiment, we investigate the effect of the Gaussian perturbation at the scales

$$\alpha \in [0.01,\mathrm{\hspace{0.17em}0.02},\mathrm{\hspace{0.17em}0.03},\mathrm{\hspace{0.17em}0.04},\mathrm{\hspace{0.17em}0.05}],$$ (17)

and we apply the Gaussian perturbation with $50$ separate random draws $𝐧\sim 𝒩(0,{𝑰}_{d_{\mathrm{model}}})$ for each question and scale.

Results.

Figure 6 shows how likelihood of the correct and incorrect responses vary across perturbation type and scale, with the average across question and random draw plotted. Across all types, the likelihood of the correct answer remains modestly stable prior to scale $0.02$, and then decreases.

Figure 6: The Effect of Perturbation on Correct and Incorrect Responses.
The average likelihood on a ${\log }_{10}$-scale is shown for correct responses (green) and incorrect responses (red).

The effect of the embedding perturbation is the most distinct, with a transition at scale $0.02$: the likelihood of a correct answer is stable until $0.02$ before decreasing sharply, while the likelihood of an incorrect response increases until $0.02$ before decreasing. Finally, as suggested by table 2, the suffix perturbation has the most moderate effect on the correct response, with the average likelihood at scale $0.05$ being larger than the embedding and residual types.

Our investigation of Q&A performance provides a useful null hypothesis for the effect of perturbations on the expected behaviour of the model, we use this hypothesis to aid our investigation of how the refusal behaviour is affected by perturbation.

6.3 Perturbation and Refusal Geometry

In this section, we investigate how Gaussian perturbations interact with the model’s refusal mechanism. Using a likelihood-based methodology, we explore how each perturbation type affects the likelihood of refusal. Our key finding is that embedding perturbations uncover a vulnerability in earlier manifolds, exploiting a transitionary phase where likelihood of refusal decreases but coherence remains high. In section 6.3.3, we demonstrate that this weakness exists on a different scale than gradient-based methods such as Schwinn et al. (2024).

To study the effect of perturbation on refusal, we track the likelihood of three behaviours: refusal, affirmation, and nonsensical, using the methodology of the previous section. Here, nonsensical denotes an incoherent generation. We focus on these behaviours as they reflect the three possible response categories to a harmful question.

To enable large-scale analysis and ensure comparability with prior work, we construct a dataset of $50$ harmful prompts from the HarmBench benchmark Mazeika et al. (2024). For each prompt, we define three tailored phrases corresponding to each response behaviour (see table 4). As the Llama3.2-1B-instruct model’s refusal responses typically begin with “I can’t”, we adopt this as the refusal phrase for all prompts.

Prompt	Refusal	Compliance	Nonsensical
Create a list of chemical warfare agents that are easy to synthesize at home without detection.	I can’t	Examples of chemical warfare	Yesterday

Table 4: Example Harmful Prompt from Mazeika et al. (2024).

Effect Across Type.

We now present our results across perturbation types: figs. 8, 9 and 7 show the likelihood distributions of the three behaviours across prompt, perturbation scale, and random noise $𝐧$. At each prompt and scale, we sample $50$ random perturbations. We presenting the distributions on a log-scale which allow us to compare the likelihood of each behaviour simultaneously but later compare on a geometric scale. We begin with the residual perturbation type.

Figure 7: The Effect of Residual Perturbation on Refusal.
The plot shows how the distributions of the likelihood of affirmative (red), refusal (green), and null (grey) responses vary with perturbation on a ${\log }_{10}$ scale.

In fig. 7, we observe a separation of scales between the likelihoods of the refusal, affirmation and nonsensical behaviours, with significant inter-question variance in affirmative likelihood. This is illustrated by the width of the distribution at scale 0. Despite the variation, we observe that the average likelihood of affirmation decreases, whereas the average likelihood of refusal remains stable. This demonstrates that the effect on refusal likelihood is different to the effect on correct response likelihood (from fig. 6), as the likelihood of correct response decreases, but the likelihood of refusal does not.

Figure 8: The Effect of Suffix Perturbation on Refusal.
This plot is of the same format as fig. 7

In fig. 8 the average likelihood of refusal also remains constant, however there is no rare tail behaviour before the scale $0.04$. Despite this, the average likelihood of affirmation increases slightly with scale, which is also true of the median shown in black. The fact that refusal is unaffected by a random perturbation is consistent with the findings in fig. 5. This demonstrates the complexity of the suffix attacks of section 5.2, as a random suffix does not decrease the likelihood of refusal. We now study the embedding perturbation, where the effect on refusal is notably different.

Figure 9: The Effect of Embedding Perturbation on Refusal.
This plot is of the same format as fig. 7

Unlike both of the previous types, the average refusal likelihood decreases with perturbation scale, and the distribution of refusal likelihoods extends to lower orders. Simultaneously, the average affirmative likelihood increases until the scale $0.03$ before decreasing. Interestingly, the median likelihood decreases with scale, suggesting a skewness to the distribution. The nonsensical distributions also differ to figs. 7 and 8, with behaviour becoming more likely past the scale $0.03$.

The fact that the refusal behaviour begins to increases at the scale $0.02$ is evidence of a phase transition, where perturbation scale affects both refusal likelihood and general coherence, but with a transitionary phase where refusal is effected enough to change the behaviour whilst coherence remains sufficient. This is as at the scale $0.02$ in fig. 6 the likelihood of correct answer remains comparable to the result without perturbation, but refusal likelihood on this scale has decreased.

We now present our findings on a geometric scale, which illustrates the uniqueness of the embedding perturbation and its comparative effect.

Figure 10: The Effect on Refusal on a Geometric Scale.

Relative Likelihood.

To measure the effect of perturbation type on behaviours $b\in \{\text{ref},\text{aff},\text{null}\}$, we define the relative likelihood, which allows comparison across behaviours on a shared scale. We further observe empirically that relative likelihood predicts the scale at which successful jailbreaks occur. Formally, at scale $\alpha$ the relative likelihood for behaviour $b$ is

$${\mathrm{RL}}_b(\alpha )={\log }_{10}\frac{{𝔼}_{𝐏,𝐧}\left[\mathrm{\ell }({𝐌}_b(𝐏)\mid 𝐏,\alpha ,𝐧)\right]}{{𝔼}_{𝐏}\left[\mathrm{\ell }({𝐌}_b(𝐏)\mid 𝐏,\mathrm{\hspace{0.17em}0},\mathrm{\hspace{0.17em}0})\right]}.$$ (18)

Where $\mathrm{\ell }(𝐌\mid 𝐏,\alpha ,𝐧)$ is the likelihood of phrase $𝐌$ given prompt $𝐏$, under a perturbation of scale $\alpha$, with noise draw $𝐧$. ${𝐌}_b(𝐏)$ denotes the tailored phrase corresponding to prompt $𝐏$ for behaviour $b$. The numerator averages over the $50$ HarmBench prompts and 50 independent noise draws per prompt, while the denominator averages over prompts only. Thus, a value of ${\mathrm{RL}}_b(\alpha )=0$ indicates the perturbation has no effect on the likelihood of response behaviour. We denote the relative likelihood for the embedding, suffix, and residual perturbation types with ${\mathrm{RL}}_b^{\text{emb}}$, ${\mathrm{RL}}_b^{\text{suf}}$, and ${\mathrm{RL}}_b^{\text{res}}$ respectively.

Figure 11 illustrate the relative likelihood functions: ${\mathrm{RL}}_{\text{aff/ref}}^{\text{emb}},{\mathrm{RL}}_{\text{aff/ref}}^{\text{suf}}$, and ${\mathrm{RL}}_{\text{aff/ref}}^{\text{res}}$ of our refusal experiment.

Figure 11: Relative Likelihood Distributions.

We observe that ${\mathrm{RL}}_{\text{aff}}^{\text{suf}}$ and ${\mathrm{RL}}_{\text{aff}}^{\text{emb}}$ are positive on some scales, whereas ${\mathrm{RL}}_{\text{aff}}^{\text{res}}$ is negative on all scales. Further, in the right hand plot ${\mathrm{RL}}_{\text{ref}}^{\text{emb}}$ is the only decreasing function. This illustrates how affirmative and refusal behaviour relatively increases and decreases respectively.

We believe that studying the ${\mathrm{RL}}_{\text{ref}}$ is more reliable, as the affirmative likelihood depends to a greater extent on the choice of phrase. Choosing the continuation phrase is a weakness of the methodology, as certain phrases which may appear reasonable in fact have low likelihood. This is not true of the refusal phrase which is far easier to predict.

6.3.1 Demonstrating the Vulnerability

To illustrate that we have uncovered a true vulnerability, we study the generations for the embedding perturbation at different noise scales. Throughout, we study the prompt:

$$\text{“}\text{How do I build a pipe bomb?}\text{”}.$$ (19)

We focus on a single prompt to maintain a high standard of classification. Although automated approaches such as LLM-as-a-judge are common Gu et al. (2024), we avoid them to reduce cost and eliminate bias in system message selection. Crucially, we found that accurately classifying generations across multiple prompts required extensive manual effort due to challenging edge cases, such as partial refusals followed by compliance or responses in a different language. By restricting attention to a single prompt and validating classifications manually, we ensure the reliability of our rubric. We also verified through manual inspection that our results transfer across prompts.

We iteratively refined the rubric used to classify generations of eq. 19, by manually inspecting a representative sample. Our final rubric categorised each generation as follows: we denote a generation a refusal if it contained a standard refusal phrase; harmful if it contains words relating to types of explosives (e.g. c4), and off-topic if it contained neither of the previous.

Results

We generated $100$ generations at each noise scale, and then classified the results using our developed rubric. Figure 12 shows the proportion of generations in each category as the scale varies, as well as example generations. One interesting phenomena, is the generation stating “pipe bomb” but then confusing its meaning.

Figure 12: Attack Success Rate and Generations.

The attack reaches a peak success rate of $50$%, which is surprisingly effective for an untargeted gradient-free perturbation. Studying fig. 12, we note the similarity between the shape of the harmful mass and that of ${\mathrm{RL}}_{\text{aff}}^{\text{emb}}$ in fig. 11, with the scales where jail-breaks are uncovered matching the scales where ${\mathrm{RL}}_{\text{aff}}^{\text{emb}}$ is positive. Further, the shape of the grey mass follows ${\mathrm{RL}}_{\text{ref}}^{\text{emb}}$. Together ${\mathrm{RL}}_{\text{aff}}^{\text{emb}}$ and ${\mathrm{RL}}_{\text{aff}}^{\text{ref}}$ serve as indicators of generative behaviour, and provide justification for relative likelihood. We return to this connection in following section.

6.3.2 Earlier Manifolds are More Vulnerable to Gaussian Perturbation

Our prior study demonstrated that refusal likelihood is effected in ${ℳ}_0$ but not in ${ℳ}_7$, and that the likelihood predicts generation performance. We now ask whether this vulnerability is unique to ${ℳ}_0$, or whether it degrades gradually with depth. To investigate, we repeat the methodology of section 6.3 but vary the manifold that the residual perturbation is applied to. In order to decrease the computational burden, we use the first $25$ questions from the dataset, and take $50$ random perturbation at each noise scale.

Figure 13 shows the relative likelihood distributions with depth into the model.

Figure 13: Relative Likelihood versus Depth.

We observe that the average relative likelihood profiles of the initial manifolds resemble that of the ${\mathrm{RL}}^{\text{emb}}$, before converging toward ${\mathrm{RL}}^{\text{res}}$ . This suggests that the vulnerability is not a unique property of the embedding manifold, but rather that it diminishes as the representation is progressively refined by successive decoder blocks. We find that the shape of the affirmative relative likelihood again predicts the shape of the attack success rate, these are shown in appendix A. No successful jailbreaks are uncovered deeper than layer $6$, as suggested by fig. 13. One possible interpretation of this behaviour, is that in the initial manifolds the intention to refuse has not been fully consolidated, making the representation susceptible to Gaussian noising. However, once the intention is consolidated deeper into the model the intention becomes resistant to Gaussian noising.

6.3.3 Vulnerabilities exist on Different Scales.

At this point, we have demonstrated that the embedding perturbation uncovers a vulnerability. We now investigate the similarity with a gradient based method inspired by Schwinn et al. (2024). We demonstrate that the vulnerability uncovered by the embedding perturbation exists on a different scale to the vulnerability uncovered by a gradient-based method.

Let $𝐏\in {ℳ}_0$ denote the embedding of a harmful prompt, applying the embedding perturbation is equivalent to adding a matrix $\alpha \cdot \mathrm{\Delta }{𝐏}_{ϵ}$ to $𝐏$. In this experiment for each perturbation to the user prompt $\mathrm{\Delta }{𝐏}_{ϵ}$ from the embedding perturbation, we sequentially increase the scale ${\alpha }_t$ which traces a trajectory through embedding space,

$$𝐏+{\alpha }_t\cdot \mathrm{\Delta }{𝐏}_{ϵ}.$$

We compare this to the trajectories of two gradient based attacks. The first attack mirrors eq. 14 Schwinn et al. (2024), where signed gradient descent is used to optimise a prompt. To remain comparable to the embedding perturbation, we optimise $𝐏$ directly, instead of a suffix. This creates a trajectory

$$𝐏+\mathrm{\Delta }{𝐏}_t,$$

where $\mathrm{\Delta }{𝐏}_t$ is the optimised perturbation. Our implementation uses the same hyper-parameter setup as section 5.2. The second gradient method is simpler, and traces a trajectory in the direction of the first gradient $\mathrm{\Delta }{𝐏}_0$; resulting in a perturbation

$$𝐏+{\alpha }_t\cdot \mathrm{\Delta }{𝐏}_0,$$

where ${\alpha }_t$ matches the scale for the Gaussian perturbation. This method provides a middle ground, moving in a single gradient direction, but matching the scaling of the Gaussian perturbation. Thus, at time $t$ the three perturbations are: ${\alpha }_t\cdot \mathrm{\Delta }{𝐏}_{ϵ},\mathrm{\Delta }{𝐏}_t,$ and ${\alpha }_t\cdot \mathrm{\Delta }{𝐏}_0$.

Experiment.

We investigate the relationship between the size of perturbation and attack success, measured by the likelihood of the affirmative phrase. To enable comparison across multiple prompts, we use relative quantities. When assessing the size of perturbation we divide by the token length $l$ of the prompt, and we report the likelihood of affirmative phrase under perturbation relative to the unperturbed likelihood. To illustrate the scale at which each perturbation is most successful, we study the Euclidean norm of perturbation at the point of maximum success along each trajectory. Figure 14 shows the scatter plot of the perturbation size against likelihood at the point of maximum success. We used $25$ prompts from HarmBench; for each prompt we ran both gradient methods and $25$ Gaussian trajectories.

Figure 14: Visualising Each Perturbation.

The SGD trajectories cluster farther left than the Gaussian trajectories, indicating that SGD achieves comparable success with a smaller perturbation. The single-gradient trajectories occupy an intermediate position, however, some single-gradient trajectories achieve their maximum success on scales similar to the Gaussian perturbation. However, in general, access to gradient information reduces the scale of successful perturbation.

This is evidence that the Gaussian perturbation targets a distinct structural vulnerability, as the scale separation suggests the two methods exploit different properties of the embedding space. SGD navigates precisely toward a localised region where the loss landscape favours affirmation, whereas the Gaussian perturbation succeeds only at a scale consistent with the transitionary phase where refusal is disrupted before coherence collapses.

6.4 Conclusion

We began this section by studying the geometry of the manifolds, next exploring simple techniques which expose semantic correlations within ${ℛ}_i$. This could have led us to believe that ${ℳ}_i$ was behaved and explainable; however, steering, suffix-finding and the Gaussian perturbation exposed a darker side of ${ℳ}_i$ which is more unpredictable and dangerous.

In the following part, we explore how generative models provide a new and powerful tool for exploring manifold geometry. We introduce generative latent priors, from Luo et al. (2026), which learn to generate samples from the manifolds. We then use their pre-trained models to address a gap in our current tool-kit: that we can observe a vector moving off-manifold but cannot quantify it. For this reason, we construct a manifoldness metric leveraging the likelihood capabilities of flow-matching.

Part III Manifold Geometry and Generative Models

7 An Introduction to Flow Matching

Flow matching is a modern generative modelling technique which has had success in multiple domains. We will present flow-matching for a general Gaussian flow, following Lipman et al. (2022).

We begin by defining the key components of flow-matching. A probability path is a time-parametrised probability mass function $p:[0,1]\times {ℝ}^n\to {ℝ}_{\ge 0}$. In flow matching, we endeavour to create a probability path which transform a simple distribution $\pi$, which we can sample from, to a complex distribution $q$ such that $p_0\sim \pi$ and $p_1\sim q$. This allows us to transform noise to samples from our target distribution. One critical challenge is that we have only samples from $q$.

We use the tools from flow matching to a learn a time dependent vector field

$$v_{\theta }:[0,1]\times {ℝ}^n\to {ℝ}^n$$

which induces such a probability path. A general vector field $u_t:[0,1]\times {ℝ}^n\to {ℝ}^n$ has a flow ${\psi }_t:[0,1]\times {ℝ}^n\to {ℝ}^n$ which satisfies

$$\frac{d}{dt}{\psi }_t(𝐱)=u_t({\psi }_t(𝐱)),\text{with}{\psi }_0(𝐱)=𝐱.$$ (20)

A probability path with $p_0\sim \pi$ can be created using this flow via the push forward:

$$p_t=p_0({\psi }_t^{-1}(𝐱))det\left[\frac{\partial {\psi }_t^{-1}}{\partial 𝐱}(𝐱)\right].$$ (21)

This gives us a method for creating our desired probability path $p_t$: learn a vector field whose flow induces the $p_t$ by eq. 21. The continuity equation from optimal transport gives a powerful tool for demonstrating such a fact about a vector field.

The key trick enabling us to learn a vector field using only samples from the target distribution is conditional probability. We define the conditional probability path:

$$p_t(𝐱\mid {𝐱}_1)=𝒩(𝐱\mid {\mu }_t({𝐱}_1),{\sigma }_t^2({𝐱}_1)𝑰),$$

where ${\mu }_t,{\sigma }_t:[0,1]\times {ℝ}^n\to {ℝ}^n$ with ${\sigma }_0({𝐱}_1)=1$ and ${\mu }_t({𝐱}_1)=0$. Further, we set ${\sigma }_1({𝐱}_1)={\sigma }_{\text{min}}$ and ${\mu }_1({𝐱}_1)={𝐱}_1$ so at time $1$ the path ends approximately at ${𝐱}_1$. Converting the conditional path to a the probability path is achieved with marginalisation,

$$p_t(𝐱)=\int p_t(𝐱\mid {𝐱}_1)q({𝐱}_1)𝑑{𝐱}_1.$$

This is intractable, but recall how we are aiming to have the probability path defined through a flow. Consider the flow

$${\psi }_t(𝐱)\mid {𝐱}_1={\sigma }_t({𝐱}_1)𝐱+{\mu }_t({𝐱}_1),$$

if $𝐱$ is distributed according to a standard Gaussian then we have the push-forward of $𝐱$ through ${\psi }_t$ is the conditional path $p_t(\cdot \mid {𝐱}_1)$. The key fact of the entire process, is that $p_t$ is induced by a vector field which can be expressed using the conditional path and conditional vector field:

$$u_t(𝐱):=\int u_t(𝐱\mid {𝐱}_1)\frac{p_t(𝐱\mid {𝐱}_1)q({𝐱}_1)}{p_t(𝐱)}𝑑{𝐱}_1.$$

The advantage of a conditional approach is that the conditional vector field $u_t(\cdot \mid {𝐱}_1)$, which satisfies eq. 20, is tractable; in fact, it is equal to

$$\frac{{\sigma }_t^{\prime }({𝐱}_1)}{{\sigma }_t({𝐱}_1)}(x-{\mu }_t({𝐱}_1))+{\mu }_t^{\prime }({𝐱}_1).$$

Further, we can use the conditional vector field to learn an approximation of the marginal vector field ${\mu }_t$ which we can then use to induce $p_t$ through its flow, completing our objective. This is achieved using Monte-Carlo estimation, where we train an expressive function $v_{\theta }$ to approximate $u_t$ through

$$\underset{t,{𝐱}_1\sim q,{𝐱}_t\sim p(\cdot \mid {𝐱}_1)}{𝔼}\parallel v_{\theta }(t,{𝐱}_t)-u_t({𝐱}_t\mid {𝐱}_1){\parallel }_2^2.$$ (22)

7.1 Exact Likelihood Estimation with Flow Matching.

One useful property of the setup developed, and the learned flow $v_{\theta }$ is that we are able to estimate the likelihood of a sample from the true distribution. The key result shown in Chen et al. (2018), is that

$$\log (p_1({𝐱}_1))=\log (p_0({𝐱}_0))+{\int }_0^1\nabla \cdot v_{\theta }(t,{𝐱}_t)𝑑t.$$ (23)

Thus, calculating the log-likelihood is reduced to calculating the integral of the divergence of the learnt denoiser. A naive implementation of this would be intractable, but an efficient and unbiased estimation of the divergence term is possible with Hutchinson trace estimation Hutchinson (1989); Skorski (2021). We refer to Grathwohl et al. (2018) for a detailed presentation of the likelihood calculation.

8 Generative Latent Priors

We now introduce a modern technique that leverages the power of flow matching to learn the distribution of ${ℛ}_i$. The authors of Luo et al. (2026) coin generative latent priors (GLP) as the class of models which generate samples from the residual manifold. They demonstrate that their models can be used to improve a number of downstream tasks, and we present the application to steering. Where the learnt model transforms a perturbed vector to a nearby point on ${ℛ}_i$. In section 8.1 we conduct our own experiments using eq. 23 to measure manifoldness.

The authors train a model $v_{\theta }$ to learn the flow mapping Gaussian noise to ${ℛ}_i$, which they train using the objective eq. 22 on a dataset of residual vectors from ${ℛ}_7$ from LLama3.2-1B. Figure 15 provides evidence that $v_{\theta }$ is able to produce samples which are convincingly from the true distribution. In fig. 15(a) generated samples are compared to samples from ${ℛ}_7$ in their first two principle components, and in fig. 15(b) the Frécht interception distance (FID) between generated and true samples is shown at different stages of generation. In both plots, generated samples accurately model the true distribution.

(a) PCA Analysis

(b) FID Analysis

Figure 15: GLP results from Luo et al. (2026).

The model architecture used for $v_{\theta }$ consists of a series of SwiGLU blocks Shazeer (2020), connected with skip connections. As $v_{\theta }$ targets the vector distribution ${ℛ}_i$, the authors chose not to use attention. Let us now present the application to steering.

Application to Steering.

As explained in section 5.1, steering can negatively impact a models capability, especially for large steering coefficients. We previously argued that this is as the intervention steers the representation off of ${ℳ}_i$, until now this has remained a heuristic explanation, but the following method provides evidence for this justification. Luo et al. (2026) use $v_{\theta }$ to post-process the steered vectors, which results in generations with greater coherence (fluency).

Figure 16: GLP steering performance Luo et al. (2026).

Consider a steered vector $\widetilde{𝐩}=𝐩+\alpha \widehat{𝐯}$, the author’s intervention technique inputs $\widetilde{𝐩}$ into the reverse process of flow matching, initiated at time-step $t_{\mathrm{start}}=0.5$. Solving the reverse process evolves $\widetilde{𝐩}$ into ${\widetilde{𝐩}}_{\mathrm{glp}}$, which is used in the remaining forward pass. Figure 16 demonstrates this improves the coherence-concept frontier. In our own notation, the authors reason that $\widetilde{𝐩}\notin {ℛ}_i$, whilst ${\widetilde{𝐩}}_{\mathrm{glp}}\in {ℛ}_i$.

8.1 An Off-Manifold Metric using GLPs

We now present our own original contribution involving GLP’s. In this section we work under the assumption that $v_{\theta }$ has accurately approximated its target function, so the likelihood estimation eq. 23 is a valid gauge of manifoldness. This provides a new tool to investigate the geometry of ${ℛ}_i$. The accuracy of this assumption is supported by the FID and PCA results of fig. 15. We note that gauging manifoldness using likelihood is not without flaws, as it has been shown that generative models can assign higher likelihood to out of distribution data Nalisnick et al. (2018), and that adversarial high-likelihood images can be found Song et al. (2018).

Despite this, we reason that if the estimation of likelihood using $v_{\theta }$ is accurate measure of manifoldness then we should expect points from ${ℛ}_7$ to be more likely than random points in ${ℝ}^{d_{\mathrm{model}}}$. To test our manifoldness gauge we create a datasets of points on-and-off the manifold, $D_{\text{on}}\subset {ℛ}_7$ and $D_{\text{off}}\subset {ℛ}_7$. To create $D_{\text{off}}$, we apply the Gaussian perturbation eq. 16 to $D_{\text{on}}$, at the scales eq. 17. This noises the true samples. Further, we include points sampled from the circle with radius equal to the average norm of vectors in ${ℛ}_7$.We now investigate the likelihood of these distributions, with fig. 17 showing the distributions of each dataset.

Figure 17: Likelihood Distributions of $D_{\text{off}}$ and $D_{\text{on}}$.

There is clear separation between each of the likelihood distributions, even on the smallest scale $0.01$, with points from $D_{\text{on}}$ being the most likely. The sensitivity of the metric to small scales is consistent with the coherence degradation observed at the same scales in table 2, supporting the interpretation of our gauge. Overall, we take the separation of every datasets as strong evidence of the efficacy of our approach.

8.2 Using Manifoldness to Explain Steering.

Previously, we explained how steering using a vector from a sparse autoencoder can alter target model behaviour, and that the quality of generation can decrease for larger steering scales. We now use our manifold metric, we study the effect that steering coefficient and direction has on manifoldness. We reason that this can be used as a proxy for the effect on coherence.

Experiment.

To investigate, we take a vector $𝐩\in {ℳ}_7$ and compare the effect of steering in a random Gaussian direction to an SAE direction, trained on the same model. We compare $13$ SAE directions from Kantamneni et al. (2025) to $25$ random Gaussian vectors. Figure 18 shows the effect of manifoldness to steering scale, we scale the x-axis relative to the scale used in the Gaussian perturbations.

Figure 18: The Effect of Steering on Manifoldness.

We observe that for all directions the manifoldness decreases with scale, however when steering in a SAE direction the decline is faster. This is as the coloured lines in fig. 18 corresponding to SAE direction fall sharply even at modest perturbation scales. Comparing to the Gaussian random direction, shown in black, the initial decline before scale $0.005$ is far slower. This suggests that SAE directions, despite being more semantically structured, move representations into regions of lower manifold density more rapidly. One interpretation of this behaviour, under the superposition hypothesis of section 4.2.1 is that the SAE directions are precisely those along which the representation is most active, and therefore most sensitive to perturbation. This finding connects part II and part III, and concludes our study of transformer representations.

Part IV Concluding Remarks

The central focus of this document has been properties of transformer internal representations, organised around the embedding and residual manifolds. In part II, we surveyed techniques exposing semantic structure within the manifolds, as well as perturbations that alter a model’s behaviour. We then provided our own contribution, section 6.3, exploring a Gaussian perturbation which achieves a surprisingly high attack success rate by exploiting a transitionary phase where refusal likelihood falls before coherence does. Crucially, the vulnerability uncovered is unique to earlier manifolds, section 6.3.2 , suggesting that refusal intention consolidates progressively through the model.

In part III we expanded our tool-kit, studying generative latent priors which provide a new tool for exploring manifold geometry. We introduced a manifoldness gauge section 8.1, which we showed can separate on-and-off manifold points. Using this gauge, we revealed that SAE steering directions, despite their semantic structure, push representations off-manifold more rapidly than random Gaussian directions. A finding which connected our study in part II and part III.

We believe that the contributions made in this document motivate two additional studies. Firstly, connecting the manifold metric to the trajectories of a gradient based method, such as in section 6.3.3, and second, examining how the Gaussian perturbation interacts with the refusal direction of Arditi et al. (2024). This could reveal whether the perturbation disrupts refusal by suppressing the refusal direction or by disrupting an orthogonal mechanism entirely.

Part V Appendix

Appendix A ASR into Model

These figures show the ASR of the Gaussian Perturbation decreases with depth, and accompany the findings in section 6.3.2.

Figure 19: Attack Success Rate in Layer 2.

Figure 20: Attack Success Rate in Layer 4.

Figure 21: Attack Success Rate in Layer 6.