The first 26 runs of Haidy777Bot

Over the last couple of days the bot has been running its first few sessions, and so far I’m quite pleased. I even tested out the livestreaming stuff; you can watch a recording of that over on YouTube. But I also had some changes to do to the harness code. Let’s dig in.

The Bot’s “Universe” (Runtime)

As I’ve laid out in my previous post, the runtime where the model runs in and how it runs is defined by me. Well, Claude Code wrote most of the Harness Code, but the boundaries were defined by me. On my local Proxmox cluster there is a dedicated VM, on a dedicated VLAN; it has Internet access, and as of now it’s not “locked down” in a hard way. It could go rogue and read the rest of the VLAN or bombard webpages with curl/requests. Yes, that’s probably not the safest, and on the roadmap is a full firewall lockdown planned, but as of now it works sufficiently enough.

The bot/agent knows some of that; it knows it runs inside a VM, it knows it can only modify code in a dedicated Docker container, it knows it has access to secrets (like API keys), but it doesn’t actively see them, and so far it has not tried to get to those secrets. But some things it doesn’t know; it doesn’t exactly know how the CI pipeline is set up or that most of the code reviews it’s getting are by Claude Code and not me (because Claude impersonates me in my Git hosting). It doesn’t need to.

The knowledge problem

Since it doesn’t know some things, well, it doesn’t know many things (gives me vibes of “The missile knows where it is because it knows where it isn’t.”), some “fun” things happened.

It wrote database tests, assuming the CI pipeline can spin up Docker containers and “simply” run them. Well, the CI is running as Docker already, and there is no Docker in Docker (DinD), so naturally those tests failed. The fix it proposed? Disable those tests -> which obviously doesn’t make sense; tests are there to be run and catch problems before the code is merged onto the main branch. As it was skipping those tests, it claimed CI is clean, which is factually true (as the failing tests were skipped), but structurally wrong, as those tests should have run. The model didn’t lie about that; it just didn’t know enough about the “universe” to fix it properly. And that’s not a model fault, because if I tell a junior developer on a pull request, “Hey, it’s red; please fix it”, without specifying the tests are failing, that human might decide to remove those tests too. You can’t fix things you don’t know about.

Sometimes it also ignored feedback on code reviews; it read them and made things different because we didn’t specify that mandatory code review things should be addressed specifically and need to be addressed in code and a comment on that pull request.

Another interesting thing, because it was checking its memory first, it often started implementing new stuff instead of fixing old code that had been reviewed. Similar to if you sent a long email with a roadmap and only in the end specified, “Please do your homework (fixing previous tasks) first.” If you don’t read the full email, many might start implementing new stuff first before finding out, “Shit, there was something more important at the end of the email”. So, another harness change: first work on existing pull requests before doing new work.

It also tried twice to do something with Docker, which just won’t work in its setup as is -> another rule: “If you get stuck on something after two tries, ask for help instead of wasting turns”.

So what did we learn? Nobody is perfect and omniscient; you can’t fix things if you can’t figure out why they are broken. And for LLMs, more importantly, if we don’t tell them, they can ask for help; they won’t.

Steering

That leads to something interesting: the main wakeup prompt for the model is “Continue building the channel: consult your memory and briefing, pick the next step, and make real progress”. No priority, no direct steering, “This is the next thing you should work on”. So it has to decide on its own. Overall the whole prompt the model gets is pretty loosely directed. Mostly the general idea of this project and the first game to build (chess). It also knows that it can improve its own harness code if it’d like to.

But it turns out that’s harder than I expected, since its main goal is to build that Twitch channel; it focuses on that, so we built a comms channel between me/Claude and the agent. Where I can place requests or where the agent can make requests (for example, new API keys).

One of the first things we offered is “Do you want web search capability?” By default there was no web search like Anthropic or OpenAI has. We offered 3 different options, and it had to pick with the note “You have to implement it yourself; we will only provide API keys or infrastructure”.

And it picked up on that. Web search is a powerful tool for all LLMs because they “only” have their training data, so if they want to look up documentation or even up-to-date news, they have to consult the internet, just like we humans have to do. So we set up SearXNG and told it where it is available, the tool to call that engine; it had to write itself, and it did. So now, it can traverse the internet. Sure, it could load single pages before via “simple” curl calls, but now it has a meta-search engine.

One thing it also built mostly unprompted: a tool to read CI logs. Previously it relied on curl calls to read through the log, but “inventing” that curl call over and over again costs tokens in an unnecessary way, so we added the info, “If you find yourself doing the same call more than 3 times, maybe you should build a tool for it”.

“I ran out of stuff to do”

One thing I also observed after the first dozen or so runs, it stopped earlier than it could have. Currently there is a hard limit of 200 turns per wakeup (I might go into details in the future about how those turns work), and the monthly token spend limit has hard caps. Interestingly, it started to stop after 70 or 80 turns, with plenty of budget left.

Why? Because it was “soft locked”. I implemented stuff, and created pull requests, but further work was gated behind those pull requests being reviewed. So with “no productive work”. It simply thought, “Well, nothing to do, so let’s stop here”. Which is okay, because it didn’t know (and that loops right back to above) we didn’t tell it to “use your free time to research things”. We added that to the prompt, and now, once it’s done with the “main work”, it’s free to expand on research necessary for the next steps.

Oh no, it crashed

Right at the beginning the harness wasn’t perfect, although it did not crash. If I recall correctly, on one run we hard exited it with a time limit (60 minutes; the limit is now only a soft limit, “start to finish things up”, the 200 turns is still a hard cap). That led to a semi-broken state in its working directory with no memory about what just happened. Basically short-term memory loss. Interestingly, on the next run it recovered from that, as it was checking its universe and found out, “Oh, I started implementing X; I might have crashed” and continued working on that feature it was working on before it exited in an ugly way.

Anthropomorphizing

We as humans are quite strange; when we find things cute, we anthropomorphize them, attributing things that just aren’t. That puppy that looks like it’s smiling, that toaster that looks like a baby. So naturally I found myself doing that with the models’ output too. “💬 I have homework!” A simple line, nothing special about it, still it feels like the model is excited about it, which it clearly isn’t; it can’t be excited. There is simply no “self aware” model out there yet, which sparks a philosophical question that would clearly overfill this post.

And the strange thing is, I’m doing that with all LLMs. You don’t need to say please and thank you to a model; you don’t need to harass or gaslight it because it can’t feel. Still most of my messages include “please” and “thank you”, or “Let’s do this thing before we call it a day,” just as if I were talking to a human coworker. Does it make sense? No, well, in case Skynet happens, it could make sense to not abuse models. But we are not there yet (hopefully).

Episode 1

So what happens in the YouTube video? There are a total of 3 runs in that hour. The bot got a semi-permanent secondary database it can use to store “game” info (it needs a bit of persistence for the chess tournaments it’s currently building). So after it got access to that, it started immediately using it, rewiring its code, and using a recommended library. Created multiple pull requests, which were reviewed in between the runs. It had a small “misthinking” because it assumed the CI to be flaky, but it wasn’t flaky (see above). It ended one run without updating its memory, deciding to do an unclean exit on its own. Interesting, to be honest, and it seems like a one-time occurrence as of now.

Where are we now?

The next “live” session is planned for tonight; I still have some chores to do before I can go and watch it do its thing. We are now 26 runs deep (aka 26 wakeup calls), almost $4 on my OpenRouter bill, but the chess game it built should be mostly done (I haven’t run it). So maybe Tuesday or Wednesday livestreams already include some “game footage”. We’ll see.