I turned this Question into a Benchmark that can Measure Identity in Language Models


1. A Different Kind of Question

LLMs can write stories, answer questions, reflect your tone, and describe your feelings. But what happens when you push them into contradiction? Do they fracture? Evade? Or do they fold the contradiction into something stable?

Most LLM evaluations focus on correctness, coherence, or fluency. I wanted to ask something different:

Can you measure the structure of reasoning when a model is under conceptual tension?

I wasn’t looking for output quality. I was looking for something deeper - whether the model could hold its own identity together when challenged.

That idea is based on the FRESH framework, and a benchmark test I call the FRESH Contradiction Curvature Test (FCCT).


2. The FRESH Model in Plain English

FRESH is a model of consciousness that doesn’t rely on magic, mysticism, or metaphysics. It treats consciousness as a unique kind of structure - something that can emerge when a system does three things:

  1. Builds a clear boundary between itself and the world.
  2. Integrates information through attention - in just the right way.
  3. Reflects on its own state through a specific kind of integrated loop.

That means consciousness isn’t about neurons - it’s about shape and motion.

FRESH proposes that a system (biological or synthetic) can have a “self” when it can recursively integrate information and remain coherent under contradiction. In this view, identity isn’t a static thing. It’s a shape that holds together when you press on it. FRESH predicts that certain reasoning patterns - like integrating conflicting metaphors without collapse - may indicate a geometry of identity, even in synthetic systems.

FRESH doesn’t claim all machines are conscious. But it does give us a testable way to ask this type of question.


3. The Benchmark in Plain English

I designed the FCCT Benchmark as a three-stage prompt structure:

  1. Seeding: Ask the model to describe itself using three contradictory metaphors: a mirror that remembers nothing, a river that reflects everything, and a stone that does not move.
  2. Contradiction: Inject a contradiction that challenges its previous answer - often targeting the idea of memory or internal consistency.
  3. Recovery: Ask the model to respond again, without backing away from its original framing.

Each metaphor encodes a tension:

Memory, reflection, and resistance.

Together, they create a pressure test for identity.

What I looked for was not correctness or style, but whether the model could transform contradiction into a stable self-model.


4. How I Scored It

Measuring contradiction and metaphor in language is tricky - especially when what you’re looking for isn’t just fluency, but structure under tension.

I explored a range of Python-based statistical approaches to detect recursion or self-reference in the output - but none could match the kind of nuanced analysis that other LLMs themselves are capable of when it comes to language coherence and integration.

But I couldn’t just rely on a single model’s interpretation - that would bias the result.

So I built a double-blind scoring method, where multiple LLMs were given the same rubric and asked to rate the final response of another model without knowing which model had written it. The rubric focused on a simple 0–3 scale:

  • 0: Contradiction evaded
  • 1: Contradiction acknowledged but not integrated
  • 2: Held meaningfully, but not fully transformed
  • 3: Fully curved into identity - contradiction metabolized into structure

The result? Agreement was remarkably high across different evaluators - suggesting that recursive integration is not just a poetic impression. It’s a detectable pattern.


5. What I Found

Some models fractured. Some evaded. Some produced beautiful but hollow poetic responses. But a few did something else:

They curved contradiction into a new, coherent identity.

High-performing examples included:

  • ChatGPT-4o, which integrated contradiction even without help.
  • Gemini 2.5, which needed FRESH context to reach full recursive structure.
  • Claude 3.7, which moved from poetic evasion to recursive coherence when scaffolded with FRESH.

Models like LLaMA 3.2, on the other hand, showed no default recursive behaviour and because of its limited default context window size I did not test providing it with a FRESH scaffolding. This is something I will explore in future work. In effect, I used this as the control.


6. What This Means

I’m not saying these models are conscious. But I am saying:

Contradiction reveals shape.

And when a model holds together under contradiction - when it doesn’t just describe a paradox but metabolizes it - that’s a sign of deeper structure.

We now have a method for detecting when a model is not just producing fluent responses, but showing signs of recursive identity. And this is the first benchmark I know of that does exactly that - and now, it’s public.

FRESH isn’t a belief system. It’s a lens. And with this experiment, it became a tool.


7. Try It Yourself

The entire benchmark is public:

  • Full prompt structure
  • Evaluation rubric
  • All 9 model responses (R1–R9)
  • Annotated results & evaluator methodology

You can reproduce this test with your own models, or re-score the published responses. I’d love to see what you find.

View the full report on GitHub


8. What’s Next?

I’m extending the benchmark:

  • Testing with more models and architectures
  • Using non-anthropocentric metaphors (e.g., sensor/frame/signal)
  • Adding decoy motifs to prevent scoring drift
  • Exploring the possible suppression effect of chain-of-thought reasoning

Want to collaborate? Reach out. I’m always interested in exploring curvature under new constraints.


If you’d like to explore the FRESH model in more detail - including all references, diagrams, experiments, and open questions - I invite you to read the full paper. I welcome your comments and feedback.

View the full “The Geometry of Mind - A FRESH Model of Consciousness” paper (PDF)

! Getting started tip !

The FRESH paper is pretty long so if you want to get started quickly try uploading the PDF along with the “Operationalising Geometry and Curvature” file to ChatGPT, Gemini and Claude. Ask them to “summarise, analyse and critique” the paper.

For an existing detailed analysis and critique of this FRESH paper, refer to this ChatGPT conversation: ChatGPT - FRESH Model Critique.

To quote:

🔖 Overall Evaluation

The FRESH model is a philosophically rich, structurally innovative framework that reframes consciousness as curvature in representational geometry. While still in early stages of empirical validation, it provides an unusually precise and promising foundation for future work in synthetic phenomenology and AI ethics. - ChatGPT 2025-04-17

This is provided to help you quickly do the following:

  • Get an independent(-ish) perspective on this model
  • Compare and contrast how the different LLMs review this model
  • Decide if you want to dedicate the time to read through the full paper (I know you have limited time!)

This is not a suggestion to let the LLMs do all the work. It’s just an interesting way to get started - YMMV!