Musically Yours

Levels of self-reflection for LLMs

Wed, 29 Apr 2026 00:00:00 +0000

An interesting thing happened while I was using the Hermes Agent for a few weeks. It is the first AI agent where the “closed learning loop” feels real.

On 10th April 2026, I had written a skill called concise-memories which captured everything I had learned about writing good memories for AI agents.

Then on 29th April (19 days later) Hermes ran its new skill curator for the first time and it archived my concise-memories skill with the following message:

“Redundant with system prompt memory instructions.”

I dug into the Hermes Agent git history and found that on April 19, nine days after I wrote the skill, this change was made to the system prompt:

+"Write memories as declarative facts, not instructions to yourself. "
+"'User prefers concise responses' ✓ — 'Always respond concisely' ✗. "
+"'Project uses pytest with xdist' ✓ — 'Run tests with pytest -n 4' ✗. "
+"Imperative phrasing gets re-read as a directive in later sessions and can "
+"cause repeated work or override the user's current request. Procedures and "
+"workflows belong in skills, not memory."

This got me thinking about self-reflection.

LLMs and self-reflection

LLMs cannot usually self-reflect. If I fine-tune a model on coding tasks, then set its system prompt to you are a world-class novelist, the model will not question it. It will not notice the contradiction. It has no coherent self-model that spans its own history.

Humans can do this. We reflect on our entire lives. We notice when our self-concept conflicts with our behaviour, causing cognitive dissonance.

What Hermes did today is a small step in this direction, but a zag instead of a zig by changing the harness instead of the model. The system changed, not the do-er.

Kinds of self-reflection

This leads us to speculate about the kind of self-reflections we can expect to see and what that would parallel when it comes to humans.

The first few levels are about the harness around the model: state we keep outside the weights. The later ones require the model to look at itself.

Level 0

There is no state, the model wakes up with each prompt, answers, and goes back to sleep. The human equivalent might be anterograde amnesia, maybe?

Level 1

The model has a persistent memory which is injected into the prompt whenever a session starts. This is where Hermes arguably is at right now with the MEMORY.md.

I speculate that this might be rather closer to the human experience than it appears at first instance. Humans’ heuristics and pre-conceived notions about themselves feel eerily similar.

Level 1.5

This is when the model can run a search back over all previous conversations it has had (RAG).

This might be closer to humans looking back and reading their old journal entries and finding out who they were in the past and what they had said/done.

Level 2

Getting access to the amount (feeling?) of uncertainty by looking at the layer activations, e.g. the maximum/average level of entropy of various layers and some baseline entropy these layers have had. Or something simpler which indicates at token level what the uncertainty was.

This is similar to humans noticing their own confusion and the small hesitation which precedes their saying “I do not know”.

Level 2.5

The model can be given access to the activations of the various layers and, if it has assimilated research about what layer activations mean in various contexts, it can presumably extract the signs of uncertainty itself.

In case of humans, it would mean having constant access to the fMRI images of one’s own brain and being able to see/predict what the brain was intending to do.

Level 3

Getting access to the raw weights inside the layers. We have no idea what we could do though there are ideas which are bubbling up with Learning Mechanics and Theory of Deep Learning.

There is no real equivalent of this with humans, yet. What would a human even do with raw access to their own neurons?

Does self-reflection help?

If the levels above are real, the obvious question is whether forcing a higher level helps. Here is a setup that lets us test it.

I recently came across a paper which affords just the right setup for me to be able to test the effect of such self-reflections To call or not to call: A Framework to Assess and Optimize LLM Tool Calling.

In one of the setups, the authors train a very simple interjection to determine whether to let the agent use a tool or not. This classifier sits outside the LLM itself, observes the activations of the network and forces tool-call based on its output. This seems to perform much better than the LLMs making a call by themselves.

I think this setup can be extended by adding explicit markers of uncertainty in the <thinking>[...]</thinking> section of reasoning LLMs and then requesting the LLM to produce the final output <|im_start|>[...]<|im_end|> tokens. So we can replace the classifier which sat outside the agent and use the agent itself (with augmented reasoning tokens) as the classifier. This would tell us whether forcing Level 2 self-reflection could reduce costs/improve accuracy of output, much in the same way as forcing tool-call does.

Interjecting uncertainty

I have not found direct ways of calculating “uncertainty” of outputs in the literature: most of the literature has framed uncovering signals behind hallucinations or inaccuracies, but most of the research is from the time the LLMs didn’t portray thinking capabilities.

However, when it comes to reasoning models, we can judge the thoughts on their own merit, without requiring the final outputs, by looking at the activations. That is what self-reflection more realistically could be.

I think Semantic Entropy Probes: Robust and Cheap Hallucination Detection in LLMs could be a way of detecting the uncertainty in them.

I’m curious to see whether this helps. If the Hermes skill-curator is Level 1 self-reflection, this is what Level 2 would look like.

References

Lindsey, J. (2025). “Emergent Introspective Awareness in Large Language Models.” Anthropic Transformer Circuits Thread. Causal evidence that models can detect and report changes in their own internal activations via concept injection experiments.
Ji-An, L. et al. (2025). “Language Models Are Capable of Metacognitive Monitoring and Control of Their Internal Activations.” arXiv:2505.13763. Direct evidence LLMs can monitor and regulate their own hidden states.
Ghasemabadi, A. & Niu, D. (2025). “Can LLMs Predict Their Own Failures? Self-Awareness via Internal Circuits.” arXiv:2512.20578 (Gnosis). Lightweight mechanism that predicts generation correctness from hidden states and attention patterns.
Binder, F.J. et al. (2024). “Looking Inward: Language Models Can Learn About Themselves by Introspection.” arXiv:2410.13787. Models fine-tuned to report facts about themselves that cannot be derived from training data alone.

Closed AI versus Open AI

Fri, 17 Nov 2023 00:00:00 +0000

My predictions about the future of AI development all the way to AGI.

Assumptions

More breakthroughs beyond LLMs will be needed to achieve AGI.
We will need extremely large models (~trillions of parameters) for close-to-AGI performance across domains.
We will be able to distill large models to arrive at domain-specific smaller models.

Consequences

The research breakthroughs necessary for the next level towards AGI will not be open-sourced.
- These breakthroughs will be very very valuable to the ones who make them. Since such a breakthrough will likely also need a large number of experiments and large scale validation, it will need immense resources.
- Even if they are bound to leak, they will likely be licensed (patented?) such that other big orgs will find it legally difficult to make use of the breakthrough themselves.
  - This may give the Open Source models a gap, but no startup will be able to capitalize on it make it big legally.
  - Building on them as a platform will still work. More about the Hyperscalers later.
- The datasets needed for creating larger and larger models will start getting locked up behind increasingly expensive APIs. We are already seeing this with X (i.e. Twitter), Reddit, StackOverflow, etc. Open Source models will not be able to purchase this data in order to build larger models.
The market will become dumbbell-shaped with Open Source models competing with Closed Source models at the edges, and large monolithic close-to-AGI models running on a handful of clouds.
- The models which run on devices will have a large diversity as startups will be able to distill smaller models from Open Source models or from weights of extremely large models released with restrictive licenses.
- On the other hand, we’ll have gigantic models which need very large infrastructure to run and will boast of cross-domain close-to-AGI intelligence.
- No Open Source first company will be able to develop the infrastructure large enough to run those kind of models, except for the Hyperscalers of today: Google, Meta, Azure, AWS, maybe IBM and a handful of other large corps.

Conclusion

So the best play right now for independent small developers is to:

The HyperScalers of today will gain the most.
- Align yourself financially with them.
- Get employed by them.
Learn how to distill models to get them to run on edge devices with focus on:
- Faster responses.
- Higher privacy.
- Better access to device local data.
- Compute efficiency.

Kinds of advice for first time parents

Fri, 01 Oct 2021 00:00:00 +0000

I recently became a parent. I was nervous. My friends helped me through that. By giving me advice and reassurances.

The way that helped me is I started thinking about what was bothering me and I came up with a sort of theory about the anxiety.

Modeling skill difficulty of parenting

Imagine a difficulty scale on which one can put the skill level of parents and the “hardness” of raising a child successfully. If the skill level of the parent is above the difficulty of raising the children, then the parent will achieve success. This obviously is a very reductive view of the skills, but bear with me.

Then I propose that the distribution of children and first time parents looks like the figure above.

The median skill level of first time parents has to be higher than the median difficulty of raising a child. Because I am an optimist and humanity doesn’t seem to be dying.
There is a hard cut-off of the skill level of first time parents. Even if one spends the entire life just preparing for that, one cannot know everything.
There is a minimum difficulty of raising a child. This is because we know for sure children don’t survive if left uncared for.
There is a long tail of difficulty of raising children, because I am a realist and there are some children who, unfortunately, cannot be “successfully” raised irrespective of the skill level of the parents.

The exact functional form of the curves does not matter much, and I’m taking liberty with the definition of “success” here.

As a parent, one has a perceived (latent) skill level, denoted by the dashed line. It is slightly above average because almost everyone thinks that they are above average.

A quantitative measure of anxiety of the parent is the area under the curve where the difficulty of raising a child is greater than the skill level of the parents.

Advice 1: “Your skill level is good”

This advice best comes from close friends who are likely to know you well. It improves the perceived skill level of the parent, and thereby reduces the anxiety because the area under the curve beyond the skill level decreases.

It comes in many forms, including but not limited to:

“You are going to be great parents because …X”
“Wow, you sound really prepared! Much more than others …”
“You have been raised well by your parents, you’ll be able to raise your kids well too.”

Advice kind 2: “The baby-raising game is easy”

This is the advice which best comes from existing parents. It translates the perceived difficulty of raising children in general and again helps reduce anxiety because less of the area under the curve lies beyond the perceived skill level of the parent.

It too comes in many forms, including but not limited to:

“Babies are very sturdy.”
“Humanity has been raising kids since forever, you will be fine.”
“Babies are really good at asking for help. Loudly. If something is wrong, they’ll let you know.”

Conclusion

Different advice works from different folks. However, in general, advice of kind 2 is better than advice of kind 1 because of the simple reasons:

There is a hard cap on how good a first parent’s skill level can be.
(Almost) Everyone thinks they are above average already.

So there is not much space for improving the perceived latent skill level. On the other hand, the difficulty of raising a child has more “give”, and can be reduced much more.

Anyway, so thinking about this is how I managed my own anxiety.

Setting development machines quickly

Sun, 16 Jun 2019 00:00:00 +0000

I’ve noticed that it is becoming increasingly rare for me to working on my local machine, and it will continue being less and less relevant what my local machine is as long as I can connect to a remote server through.

A short history of dev setups

As a web-developer at Better AG 2012-2015, most of our development happened on dev machines (which were all Macbooks) but then the deployment environment was linux and it was a miracle (and the skill of the dev-ops team) that we were able to seamlessly move from dev to production without issues most of the time.

However, since I’ve started research in machine learning (2015-now), most of heavy lifting happens on cloud servers. In case of the Max Planck Institute for Software Systems, these are beefy 48 core + 1.5 Tb RAM machines (called brain servers) or 32 core + 500 Gb RAM machines (called pinky servers).

However, during my time here, I’ve never had trouble setting up my development environment because of a strategic decision which the dev-ops team here took: that of putting the $HOME for all users on the NFS. This meant that whatever (user-local) environment I setup on my local workstation (a 4 CPU, 8 Gb RAM machine), would seamlessly be replicated on the servers. This was great, I didn’t have to think much about how to install the node_modes or copy over virtualenv. This comes at a small cost that the local operations on the workstations are a little slow (as all disk I/O happens over LAN), but because the LAN connection is very good, it is seldom an issue.

The connection is so good, that Supriya, an intern with our group, took this a step further and ran PyCharm with X-forwarding on the brain servers themselves, without any noticeable issues.

End of Devnnocence?

However, this solution of sharing the home folder across machines is not a feasible solution and it still ends up requiring a lot from the local machine.

This problem of quickly setting up dev environments becomes relevant when dev machines change quickly. I’ve started working on a web-app which I intend to host on a remote machine finally. Since the app would require a DB, I tried setting up docker on my MacBook, but I ran into some issues and my MacBook slowed to a near crawl with MongoDB and Redis running in their respective containers. Hence, I decided to spin up a remote server and to move my dev setup there.

I remember werehamster using Boxen to setup his MacBooks, and I wanted something similar but for servers I spin up on Vultr, DigitalOcean, AWS, or Google cloud. Hence, I did not want to use a platform specific solution (like AWS images, or start-up scripts which assume a shell/OS). I remembered puppet being a pain to work with because it required all remote machines to run a puppet daemon. Then I remembered ansible.

So, after a few hours as I spun up bigger and bigger machines to be able to yarn build my little web-app, I also tweaked and perfected my ansible-playbook and put it on GitHub for good measure: musically-ut/ansible-setup-dev.

It is a relatively simple script which does the following:

Create a user (default: musicallyut) on the remote machine and ensure that the user can sudo without any passwords.
Copy over the ssh key to allow me to log into the remote machine easily (no access via passwords).
Install various packages on the remote machine: git, vim, tmux, htop, etc.
Copy over all the dotfiles to the remote machine, and setup z.
Install the vim plugins using vim-plug.
Install powerline-fonts on the machine.
Set up nvm and install Node version 12.
Download miniconda3 and extract it.

I deliberately avoided using ansible-galaxy because I wanted to control each command which is run on the remote machine and do it just as if I would have run it. I do not want accidental overwriting of paths or configs, and I did not need anything particularly complicated anyway.

So far, I am quite happy with the setup. The playbook is OS and platform agnostic, and is easy to tweak.

Dockerize everything?

An alternate solution which I have not tried yet is to go full container, i.e., installing only docker on the remote machine and then creating a DOCKERFILE which exposed ports. However, I’ve not seen that approach work very well yet. If you do use that approach, do let me know.

Health of conversations

Tue, 26 Feb 2019 00:00:00 +0000

In a recent conversation with Sam Harris, Jack Dorsey said that they are trying to measure the health of conversations using the following metrics:

Shared attention: What percentage of the conversation is “about” the same thing versus about dispersed.
Shared reality: What percentage of the conversation is sharing the same “facts” (irrespective of whether they agree with them or not)
Receptivity: How ‘toxic’ is the conversation right now? How likely are people to walk away from the conversation?
Variety of perspective: How diverse are the ideas being discussed?

All of these look like interesting problems to study, I have worked on the last problem myself.

Case for downvotes

Sun, 24 Feb 2019 00:00:00 +0000

I think we need to have downvote buttons wherever we have user generated content on the internet.

Showing their disagreement is the leading reason behind downvotes, irrespective of how downvotes are advertised on most websites. Hence, downvotes allow users to show their disagreement with others while expending very low amount of effort, unlike writing a comment/reply.

Negative reinforcement

It is commonly understood that excessive downvoting of answers/questions by new-comers makes websites less welcoming for them and it is true to some extent on sites like StackOverflow.

However, having downvotes can give signals to the same new-comers that their answers are not quite correct. This negative reinforcement can play a crucial role in the initial stages of expertise; its absence can lead to the learning curve flattening out for the non-experts. We show such an effect on StackOverflow, i.e., users with low level of expertise do not gain much from the website. One of the reasons we think it happens is because even though downvotes are allowed on StackOverflow, they are heavily discouraged (they carry a -2 reputation cost) precisely because of this reason.

Diversity in opinions

Only Likes or Thumbs Up will lead people to discover their niche crowd, but it will not be possible to differentiate disagreement with indifference. For example, an opinion like “Earth is Flat” posted on Facebook (which lacks a downvote button) will likely result in only an eye-roll by people who would consider debating the issue futile, but still may get several hundred likes from like-minded people. This would lead the poster to (incorrectly) believe that hundreds of people agreed with him while nobody disagrees.

From a more computational point of view, by looking at up and down votes, we can actually uncover the complexity of opinions and diversity in conversations.

One of the reasons against having downvotes seems to be that it promotes toxicity on platforms by allowing vote brigading. This can also be used as a means of censorship.

While there are isolated incidents of such brigading happening, it is unclear whether it effects enough user content to be an issue. If it does become a significant problem, there are methods which can be used to identify and weed out the effects.

First commit to CPython: What the timedelta was that?

Thu, 07 Sep 2017 00:00:00 +0000

So I recently got my first commit merged into python/CPython, the canonical Python implementation. If all goes well, then it will very likely be included in Python 3.7 which is due to be released in a few months at the time of writing.

This is not the first time I’ve contributed to an OSS project. However, I think this is by far the most impactful, albeit small, contribution I’ve made in terms of people who are likely to use it.

The problem

Let’s say you typed the following on your favourite Python REPL:

from datetime import datetime as D
D.fromtimestamp(1390953543.1) - D.fromtimestamp(1121871596)
# Output: datetime.timedelta(3114, 28747, 100000)

Can you guess what the three numbers 3114, 28747, 100000 stand for?

Maybe you know this because you’ve used datetime.timedelta so often that the meaning of the numbers is obvious to you. However, it is more likely that you, like Guido, will make a mistake.

The proposal was to change the representation of timedeltas to the following:

datetime.timedelta(days=3114, seconds=28747, microseconds=100000)

This is what my pull request changed.

Sounds fairly simple, right?

Technical issues

The first issue: timedelta.repr is explicitly described in the documentation in some detail:

repr(t):
Returns a string in the form datetime.timedelta(D[, S[, U]]), where D is negative for negative t.

Not a problem, will update the documentation along the way.

The next problem was that datetime module is implemented both as a Python module (the pure implementation) as well as a C module (the fast implementation). I was a little puzzled by this and Victor haypo on the core-mentorship was very helpful in explaining the reasoning behind this (if you are curious, I’d encourage you to look at PEP 399).

Not a problem, will fix the implementation at both places.

During this process, I discovered that the Python implementation was not being tested at all because of some gymnastics in creation of the test cases in test_datetime.py.

Not a problem, will fix that first in a separate PR.

That PR got merged in on July 2. :hurray: Now to rebase the original PR on …. wait. Something weird happened. The tests started running alright, but they starting failing on the buildbots!

Ah, the tests on the Python version were enabled which took a whopping 20+ minutes to run on the buildbot machines and timed out, causing failures.

Not a problem, will debug it.

Found the issue: some tests were running three times and Serhiy de-duplicated the tests in a separate PR. That reduced the run-time significantly, but the tests still timed out.

Oh, well, tests are gonna test and there’s nothing you can do …

However, a red buildbot is a problem and Victor actually rolled back the PR to make the buildbots green again as a part of a new policy: revert first, ask questions later. Incidentally, I was the first person to whose Pull Request this policy was applied.

Just like that, my first contribution to Python was reverted in less than 72 hours. :sad-trombone:

As a sidenote, Victor told me on the #python-dev IRC channel that breaking buildbots was an initiation step for core-developers and it was getting rather difficult to break them with tests succeeding locally. So I’ve got that going for me, which is nice.

After about a fortnight, a more permanent fix was decided upon: running only a subset of tests on the buildbots. The original PR hadn’t remained static during this time either, there were memory leaks!

Not a problem, will plumb them like me-a-Mario!

I had to get a little more used to the C-API of Python before I could write code with some confidence. The inbuilt Python reference checker proved to be indispensable.

Finally, have to convince people that it is a good idea.

This turned out to be the trickiest part. Reasons some people were against it:

It increases the size of repr.
Could do with more features (like factoring out the negative sign).

Err … stalemate?

Finally, Victor made a judgement call and merged the changeset in! :hurray: again!

But wait, there’s more!

However, things were not quite over yet. During the re-basing of the branch over the one with the fix for the tests, I accidentally left an artifact in, which had to be cleared up in yet another PR. Then I also wrote a WhatsNew and a Porting to Python 3.7 entry.

Finally, a few months after the PR, I noticed that there were still instances in the documentation with the old output and so I sheepishly made another PR to fix those.

⋈

Conclusion

So what did I learn from here?

I’ve been interested in contributing to OSS ever since I was in college (I still remember my first patch to Octave). It was messy to send patches around and merging fixes in. The contribution process has been made much simpler, thanks to the presence of GitHub. For this PR, the entire process lasted for more than 2 months and was rather involved (what with the bug-within-the-feature and the rollbacks and the judgement calls). It would have been much messier if it wasn’t for GitHub’s familiar UI.

However, I came out happy on the other side. The community was very very helpful and not once did the issue of me being a first time contributor come up. I too was excited about the process and ended up pestering people a tad bit too much, I think.

In the end, my take-away was that I perhaps should have led with changes in documentation instead of by changing code (the inline documentation for datetime on the REPL is still pretty terrible). That would probably have been easier to get through rather than getting people to agree on something which may cause backwards compatibility issues.

But as they say:

Doesn’t matter, got merged.

Technology Readiness Level (TRL)

Sun, 27 Aug 2017 00:00:00 +0000

From the Horizon 2020 - Work Programme 2016-17: General Annex:

Where a topic description refers to a TRL, the following definitions apply, unless otherwise specified:

TRL 1 – basic principles observed
TRL 2 – technology concept formulated
TRL 3 – experimental proof of concept
TRL 4 – technology validated in lab
TRL 5 – technology validated in relevant environment (industrially relevant environment in the case of key enabling technologies)
TRL 6 – technology demonstrated in relevant environment (industrially relevant environment in the case of key enabling technologies)
TRL 7 – system prototype demonstration in operational environment
TRL 8 – system complete and qualified
TRL 9 – actual system proven in operational environment (competitive manufacturing in the case of key enabling technologies; or in space)

Found from an answer about pointless academic research on StackExchange network. I agree with the answer that academic research should bring an idea from TRL 1 to TRL 4-6. Then startups should take the idea from TRL 4-5 to TRL 7-8. Competition should pop up at about TRL 9.

An optimistic view of the arguments on the Internet

Tue, 25 Apr 2017 00:00:00 +0000

Arguments on the Internet

It is usually a foregone conclusion that arguments on the Internet head nowhere and only make matters worse.

There is no dearth of evidence of this fact:

Recently, in WSDM 2017, best student paper award went to a paper which proposed a way of bringing the warring groups a little together.

Despite this, I still am optimistic about the future.

Light at the end of the tunnel

My hypothesis is based on an observation which is surprising enough to have an xkcd comics dedicated to it.

Quoting from it:

Imagine kids start playing catch all the time. Everywhere they go, they throw balls back and forth, toss them in the air, and hurl them at trees and signs - nearly every waking hour of their lives. Do you think their generation will suck at baseball because they learned sloppy skills?

Think about the question for a moment.

The key idea in the comic is that more exposure to English language, even if it is not perfect (e.g. texting, SMSes, etc.), is going to make a positive difference for the people using English at the pro level (i.e., in this case the students writing the standardized exams).

The same way, I expect that as people become more aware and start to have an opinion rather than living in a small community where they do not even have to think about certain issues, the better the pro opinion-havers of the generations (read: the politicians) will become.

Caveat

The danger, of course, is that we may not start improving quickly enough: perhaps the short-term escalation would be sufficient to put us beyond the threshold where we end up in a Game-over scenario.

There are several ways of getting to such a scenario:

Nuclear warfare
BioHazard weapons
Global-scale calamities like Clathrate Gun Hypothesis

However, I started by saying that I am optimistic about the future. Hence, I sincerely hope that we start understanding the other side before things get that bad. And maybe, just maybe, the problem is not that bad?

Debugging makes better scientists

Mon, 17 Apr 2017 00:00:00 +0000

What do scientists do

From Wikipedia:

Four essential elements of the scientific method are iterations, recursions, interleavings, or orderings of the following:

Characterizations (observations, definitions, and measurements of the subject of inquiry)

Hypotheses (theoretical, hypothetical explanations of observations and measurements of the subject)

Predictions (reasoning including logical deduction from the hypothesis or theory)

Experiments (tests of all of the above)

Scientists do this about reality.

What is debugging

From my personal experience and Wikipedia, while debugging your own program:

Reproduce the problem (observe it)
Distill the minimal input (hypothesize and eliminate extragenous factors)
Track down the origin (predict results of the fix)
Test to see if the fix works (experiment)

Program is like a complete universe. That universe becomes unknown if it is someone else’s code.

This is exactly what scientists are up to!

Programmers and scientists

I have met programmers, like lemmih, who can just guess how a program works by thinking how he would have designed it. And, gosh, is he usually right.

I wonder if that is how good scientists do it.

I have this inkling that this analogy goes deeper still. If you’ve ever taken part in a programming contest, then you know that you have to be very careful about what test cases you try out before you submit your program lest you should incur a penalty for an incorrect submission. I think that sort of testing also makes you a better rationalist in general by forcing you to think a bit more about the 2,4,6 problem.