A quick recap for anyone landing here cold. I built myself a board of five AI advisors, each one with a different personality and a different set of convictions, and each one keeping its own memory of what we have discussed. It is all built through a set of markdown files. More detail about the overall file structure can be found in Part 2 (https://www.ironrungconsulting.com/posts/ai-board-of-advisors-part-2/).
When I started out building the AI Board of Advisors, one of the things I wanted to test in this tech demo is isolating memory across personas. Given that Claude stores context in-memory during a session, I wasn’t sure if memories and voice patterns would bleed over from one persona to the next with extended usage. To minimize the potential impact of this, I isolated each persona and memory into its own specific markdown files. Later I built a new version that stores all the memories from every persona in sections of a single markdown file so I would have something to compare it to.
The structure of the test was simple. I built a new instance of the multi-file structure, and a second one with the same text merged into a single file, under headings instead of in folders, checked line by line so the words were identical. Same model, same rules, just the text structured in one file instead of across many. Then I asked eight questions, once to each version of the board of advisors, and compared the behavior and overall performance.
The short version is, they came out about the same. Not identical on every measure, but nothing I found would have told me which version I was talking to if I was interacting with them blind.
The primary thing I wanted to test was whether five advisors stay recognizably five, or whether I was reading personality into five answers that were basically the same underneath. I asked one question to all five advisors individually in both versions, and I had Claude strip the names off the ten answers. Each answer was readily identifiable to the robot persona that created it, save for one where Claude made an error in presenting it and conflated two answers together. I think this was a good but unexpected lesson. When you utilize a skill or AI task beyond its original design parameters, the AI model will try its best to stretch and fit to what you are asking, but it may make some assumptions or mistakes along the way or not behave exactly as intended. In this case, the model attempted to anonymize the responses but it made a small (and easy to spot) mistake in doing so. Overall though, there was no obvious personality bleed between personas in either version of the board.
So the personalities hold, and the folders do not seem to be a primary driver of this. Put all five in one document with headings between them and they stay as distinct as they were across several files. Whatever is producing that distinctness, it is the way the advisors are written, not the structure of the files.
Another behavior I observed was that the model was not consistent in following the instructions to only read the relevant files. When I ask one advisor a question, the system is supposed to read six files, the three files shared by all board members and the three files specific to that advisor. Across various prompts it read eighteen, then eight, then thirteen, then eight, then eight, then eight. Not once did it read only what the instruction file tells it to read. This never led to any unexpected results or behavior, but it would be an interesting follow-up experiment to see if I could get the read behavior to work as designed through modifying the parent instruction file.
The other place the two versions split was speed. I assumed the single-file version would be faster, and surprisingly it was not. When I ask one advisor a question, the single-file version reads the entire document, all five advisors’ material, just to answer as one of them, while the multi-file version reads less. Saving a new memory is even more lopsided. The single-file version copies the whole document, edits the copy, and writes all of it back, where the multi-file version adds one entry to one small file. It turns out that these read/write operations only make up a very small percentage of the overall processing time. Since most of the lag time between prompt and response was the model itself generating the answer, the markdown structure did not have a noticeable effect on performance.
There are a few concrete results I can draw from this set of experiments. First, I think the most important thing to focus on during an AI build is to get the behavior guiding instructions correct. Storing the memory and context is not super dependent on the shape of how you store it, and more dependent on if the model can consistently understand how to use those memories correctly. I also think it demonstrates that you can build AI tools that have persistent memory that grows over time, and it can follow instructions to only apply certain memories to certain tasks or prompts if the instructions tell it to do so. It was interesting that this held even when the model would consistently read more files (and thus absorb more context) that it was instructed to. The final lesson was that when you stretch an AI skill to work beyond its original design parameters, you might get some unexpected results. So carefully considering the use cases and edge cases is an important part of the design process. An AI user is not restricted to specific inputs like they would be with a traditional app UI. And AI will always try to find a way to execute what the user asks, so for larger tools with multiple users it’s important to consider how it might be utilized in unexpected ways.
I also want to be clear that these tests were pretty small and conducted on a simple prototype-y build. The AI Board of Advisors was never meant to be more than a tech demo, so it would be interesting to see how using markdown files as a persistent memory store would work in an application that encompassed significantly more context. But for a tech demo and for the purposes of this experiment, the performance between the two versions was more or less identical.
For my final set of tests, I ported the exact same set of instructions to ChatGPT and Gemini. As I dig deeper into AI, I think it is important to stay model agnostic. Things are changing so rapidly in terms of cost, performance, and connectivity that I would not ever want to pigeonhole a client into being reliant on a single frontier model. I still have some more testing to do across various models, and I will be talking about that more in my next post.