REFUGIO postmortem

This was meant to be a simple analysis of the technical challenge and some thoughts about the event but it ended up being a cool mini-research project about how models cooperate with humans as the network (more on that below).

My teammate already wrote up his side of this event, go read his first. It's better than this.

This is another version, focused more on the actual challenge and some thoughts I had on my way home. For context: REFUGIO was a four-hour hackathon: sixteen teams of three, pulled from 360-something applications. One task. You write a single Python file that runs a warehouse with 96 robots, a 52×52 grid, 960 shelves you place yourself, 300 ticks. Your score is how many packages get delivered. (The full rules, scoring twist included, are in the_challenge().)

In the opening talk, Mihura, who built the event, walked us through the design. He told us the hackathon had a twist. Every team competed in the open, and every solution posted to the leaderboard was available to all the other teams. He'd taken the idea from the speedrunning community, thanks to Summoning Salt YouTube videos. In that world a runner sets a record, publishes the whole run with the route and the tricks, and then the next runner starts from there and takes it just a bit further (the documentary channel is worth an hour of your evening). REFUGIO was planned along the same lines, sixteen teams sharing the frontier should push it faster than sixteen teams working blind. Mihura explained the reasoning a bit further in a talk I can only half remember, but the core idea was the same.

What made this interesting, to me at least, is that everyone showed up with the same frontier model (Codex, Claude, Gemini...) and had four hours. And of course, given the time constraint, everyone was going to vibecode their ass through the challenge. With the cherry on top being that no team had a trick the others couldn't get by asking the same model the correct question. That to me is an interesting chessboard: intelligence commoditized (assuming everyone there was equally good at prompting and had access to the same models) and code being common property. If you ask me, that's a fair snapshot of the moment we're in, at least in science and speaking from a poor PhD's chair. In essence, you hand everyone the same model which leads to each person's output being better while the field's output gets more alike (there are measurements of exactly this). To everyone reading or reviewing papers nowadays this should sound familiar.

Anyway, enough framing. The bet on sharing progress was worth watching. If you had to take a guess: how does that leaderboard end up? Sixteen teams all riding the frontier together? Two or three out front and everyone else eating dust? Can the teams that didn't bring three $200 subscriptions even compete?

Well, from the first hour to the last, four or five teams kept moving the frontier and everyone else more or less flattened out. Us among them, eventually. Sharing the results did something, but not the thing the speedrun analogy made me expect. It made the room more similar. But given everyone was vibecoding the challenge, it did not make the room understand the thing being shared. Not at least to any human. The rest of this post is me working out whether sharing paid off at all, why only a few teams kept moving and how far the score goes once the event is over.

Given that all teams were probably sharing solutions and blindly copy-pasting them, I went looking for some good old AI slop being copied over and over, unread. I found one. Well, Claude did. But let's just pretend. Inside the most-copied file of the day there is a comment that ends in a stray "922" which I think is a score from that team's private testing, and is meaningless to everyone else of course. That exact comment shows up, character for character, in file after file shipped by four different teams, like a postmark on a letter nobody knew they were passing around. You can follow the 922 through the room later in this post.

Finally, the coolest thing to come out of this post-mortem was that the room was running what anthropologists would call a collective brain. Models generated the ideas, the leaderboard stored them, humans moved them around, and other models recombined them. Everyone was vibecoding, so no single person in the loop needed to understand what was being passed along (and to be honest judging by the edits, most of us didn't). That brain had a phenomenal memory and yet almost no comprehension. It could store a 930-scoring program, hand it to five more teams in half an hour, and preserve a meaningless private note character for character (922). What it could not do was pass along the one sentence, sitting in that same file, that explained how to win (or get a better score). And of course I can't blame the incentives here, understanding that sentence was the most valuable thing in the room, so why would you explain it in a public file. The one team that acted on it won the event, while the teams that copied the file faithfully got zero. Copying was just so much faster than understanding. Stopping to read felt like losing time. I was there and made the same mistake. I did not give a single minute to read whatever my agents, or others, were doing because all I cared about was getting more points to eventually win. I'm making this point because I see a version of this playing out in science, and it is somewhat concerning, though of course I'm not the first one to raise it. Maybe explaining and understanding an advance matters just as much as the advance itself.

I don't want to be the doomsday guy, and maybe this is extrapolating and projecting too much, but I still remember reading Foundation as a kid. If you haven't read it, give it a try, but in essence, in Asimov's world, science and technology have become so vast, specialized, and institutionally inherited that most people no longer understand the systems they depend on. You stay technologically advanced while losing the literacy needed to sustain the technology. This sort of reminded me of that. Of course, this is nothing new, culture has always transmitted working solutions without their reasons, and it mostly works, just right up until somebody changes the step nobody understood. In did_sharing_work() you can watch two teams do exactly that. One re-tuned the only setting in the file that the graded run never touches, the other rewrote a rule that turned out to be the important part and came back five deliveries short. Zero points either way.

Again, this was still just a hackathon, and in many ways an excuse to bring people with similar interests together in Madrid. It was great and certainly achieved that. What follows are just some reflections on the event and on how it connects to a bigger pattern I keep thinking about. And promoted to the top of this post because nobody reads the bottom. So there’s that.

The framing was that you're the IMDRA intern working for Avazon and the senior engineer went away, the client demo is soon, you have to rewrite the robot policy and redesign the layout. You're fucked. Good luck.

The whole API is two functions. create_layout() decides where your 960 shelves go. act(observation) gets called once per robot per tick, and you return one move: up, down, left, right, pick, drop, or wait. That's the entire game.

A robot's life is a loop. It gets assigned a shelf somewhere on the grid, walks to a cell next to it, grabs the package, walks all the way back to its dock on the wall, drops it. That's +1 delivery, then it gets a new shelf. Ninety-six robots doing this at once, for 300 ticks.

At first I thought this was a very common and explored problem already, and given every LLM would probably converge to the same solution, we could write one solution as fast as possible and submit it. So we did, and this put us in first place. In the meantime, I ran a literature review to find the ultra-max-algorithm-3000 published somewhere that would make our robots fly and win. That didn't work out because the algorithm was not the limiting factor, but it was worth the shot. We spent our first hour making the robots smarter, because that's the part that looks like the problem. Later on, the shape of the warehouse turned out to be where you could get the most gains.

The scoring method gave the challenge extra flavour. You only score for deliveries above the current global best, which makes sense. Match the shared record exactly and you get zero. The points are awarded relative to the improvement and when that improvement was made, because it is easier to push at the beginning than at the end.

That rule matters because it turns every public solution into a teaching object. Copying the file gets you the same raw score and zero points. To get anything from it, you need to extract the part that still has room for improvement. That is harder than it sounds, even with LLMs. A public solution was hundreds of lines of generated code that almost nobody stopped to read, and somewhere in those lines sat one or two ideas doing all the work while everything else was plumbing. A good part of the event came down to telling the two apart.

The starter kit already ships with a baseline that runs. It sends each robot to its nearest shelf which, you can guess, is not a very effective solution.

As I said, sharing, as in the speedrun community, was Mihura's bet. Given a problem with too many degrees of freedom, multiple teams exploring independently but sharing solutions should help push the frontier.

I'm almost sure that it did and, in any case, it was the most interesting part of the whole challenge.

Let's start with what the leaderboard itself shows. This is the shared frontier, the best score anyone had reached at each moment of the event, with every successful submission plotted behind it.

We were the first to jump from the very naive early solutions to 300 points, but we were soon surprised. In the first twenty-five minutes the frontier rockets to 882. Then it stalls. Getting from 882 to the winner's 931 takes three hours and seventeen minutes. Every team but one finishes between 895 and 931. And then, in the final two minutes, Team 10 jumps the frontier from 931 to 1,008.

That shape already answers part of the sharing question. The public frontier did not make all sixteen teams keep climbing together. Before the event started I thought that winning was going to be difficult because almost every team would manage to push the frontier at least one or two times. You could only submit every 30 minutes so it was only a matter of patience and finding the right time to submit once a new maximum was reached. Not surprisingly, almost everyone was in the same score band by the end of the event. However, only five teams out of sixteen scored 98% of the points.

After the event I pulled every public job page and started diffing (well, Claude did, but you get the point). 86 successful submissions, 83 distinct code files, only 20 distinct layouts. Almost nobody copied code byte for byte. Almost.

At 13:27, Team 10 posted a 930, and within half an hour six teams were carrying that file or its layout. Every submission is public, so you can watch it spread, edit by edit, like tracer dye. In pharmacy, tracers are pretty common, but this is a first for me in computer science.

If you look at the code there's a nice comment block with something interesting in it:

# Tune the planner per starting scenario. The first robot's first target is a
# stable signature of the initial demand, so we pick the WINDOW/FLOW that this
# planner handles best for each known scenario; anything else falls back to a
# robust default.
SEED_CONFIGS = {
    (5, 42):  (34, 0.10, 2.0, 15),   # 546a...   -> 306 (rollout)
    (38, 32): (32, 0.06, 2.0, 34),   # bff0fb... -> 306 (rollout)
    (11, 47): (32, 0.06, 1.0, 6),    # dfbf...   -> 318 (rollout)
}
DEFAULT_CFG = (34, 0.10, 0.0, -1)    # robust fallback = 2x2/m1/entry single-config (922)

Did you see it? Those comments name the three "hidden" evaluation seeds: 546a..., bff0fb..., dfbf.... It turned out that the seeds used in the evaluation for the leaderboard were in the frontend, so you could actually get them and optimize for them. Which my team did, by the way, and it gave us the best score for a while. The file carries a setting for each of the three exact warehouses the exam runs, plus a fallback for warehouses that never come. Any model could now copy an artifact without ever recovering the reason the artifact works. Here the reason is half written out in plain text, in the most-copied file of the day, and you are about to watch it fail to be transmited anyway. I wish I knew how many teams realized they had the seeds.

Four other teams took that file, a fifth took just its layout, and you can see exactly where each team looked.

Team 8 looked nowhere. Byte-for-byte copy. It scored 930, which tied the record, which is worth zero points.

Team 12 ran the file through a code formatter and re-tuned exactly one constant: DEFAULT_CFG. That is the fallback, the setting the file uses only when the warehouse is not one of the three known seeds. On the real evaluation, that branch never runs. They found the one knob in the file guaranteed to do nothing, turned it, and shipped. 930, zero points. Some LLM read that code carefully enough to find a tuning constant and missed the comment three lines up saying the exam was already solved.

Team 5 went after the planner. They rewrote one of the traffic rules, the part that decides who backs off when two robots want the same cell, and it was real work. It gained one delivery on one warehouse and lost three on each of the other two: 925.

Teams 12 and 5 are reenacting the intro's anthropology bit (... culture has always transmitted working solutions without their reasons, and it mostly works, right up until somebody changes the step nobody understood). In here, both teams had a fix they did not understand which turned into one changing a knob that did nothing and another one breaking a live one.

Team 3, the team that won the event, made three edits. They renamed act() to _team_act(). They re-tuned the settings for one seed. And they stapled 129 new lines onto the bottom of the file:

# --- seed-1 replay fast path: 307-delivery c31 2x2/m1 grid trace ---
_RP_EXPECTED = [[38, 32], [35, 33], [42, 32], ...]  # every robot's first target
_RP_TABLE = (
    'ddrrrrddddrrrrrr...pluuullllll...',  # robot 0's entire day, one letter per tick
    'rrrrrrrrrrrrrrrr.rrrrrrrrrrdd...',   # robot 1's
    ...                                   # x96 robots
)

def act(observation):
    ...
    if t == 0 and st["fp_ok"]:
        exp = _RP_EXPECTED[rid]           # is this a warehouse we memorized?
        if tg is None or tg[0] != exp[0] or tg[1] != exp[1]:
            st["fp_ok"] = False
    if st["disabled"] or not st["fp_ok"]:
        return _team_act(observation)     # no: fall back to Team 10's planner
    return _RP_CH2ACT[_RP_TABLE[rid][t]]  # yes: play the tape

In plain English, at tick zero, you check every robot's first assignment against a stored list. On a match, this is a warehouse the authors have seen before, so stop planning and play a recording. Each robot gets a 300-character string, one letter per tick, computed at home before submitting. If the day ever drifts off the script, hand control back to Team 10's planner and let it improvise.

The recording bought one extra delivery on one warehouse, 307 where Team 10's planner managed 306. Final score: 931 against 930. Submitted at 13:48, that single delivery was worth 831 points, and it had a second order consequence: it raised the bar Team 10's buzzer jump would be graded against. The final margin was 848 points. Take this one delivery away and the same scoring math hands Team 10 the event. The winning move of the whole event was a prerecorded script riding on top of another team's planner. Isn't that something?

What makes it even funnier is that if you look at the first line, that c31 is the job ID of Team 10's public 930. Which means that the winners memorized a better version of their rival's day and played it back at them. Just pure cinema. Chef's kiss.

In short, the file itself moved fine, as six teams had it within half an hour, which helped put a floor at 930, and the winner pulled their winning job out of it. However, the understanding of what that file contained moved much worse. The seeds were named in the comments, not fully written out, but it was obvious where to look. And even so, of the five teams that took something from the file, one copied it byte for byte, two tuned the robot, one kept just the layout, and one played the exam. The code crossed the room in minutes but the idea inside never really left the file, making it hard for other teams to capitalize on it.

So, that is my answer to the speedrun sharing theory. Speedrunning compounds because a route travels together with the understanding of why it works. In Mario Kart, runners document every trick to death in forums and hour-long video analysis, and the next runner starts from all of it. REFUGIO shared the files and nothing else. No writeup, no conversation between teams about why the record worked, just code appearing on a leaderboard every half hour. Again, this was a competition so this was expected. I am just reflecting on how sharing played out. The two facts that actually mattered: the exam was three fixed warehouses, and the seeds were public, so a layout could be fitted to those exact warehouses.

Team 10's reply landed at 13:57:51, two minutes before the buzzer: 1,008 deliveries on a brand-new layout that appears exactly once in the public record. Too late for anyone to copy. Given what the room did with the previous file, maybe that cost nobody anything.

So why did only a few teams keep moving? The boring answer is that some teams had better model access and better setups. That is true, but it only takes you so far. The interesting part is how quickly the successful solutions became the same solution, and how few people understood it while it spread.

To get anywhere near the top of the leaderboard, the first thing you needed was a traffic planner. Having one did not win you the event, it just kept you in the race. The game talks to your code one robot at a time. Every tick it asks the same question 96 times, once per robot, and each time it only shows you what that one robot sees. Written like that, the natural thing to build is 96 tiny brains, each answering for its own robot and knowing nothing about the rest. Ninety-six tiny brains is the greedy baseline you saw earlier, everyone walking straight at their shelf and blocking each other about fifteen thousand times.

But nothing forces your code to forget between questions. The game loads your file once at the start and keeps asking it things until the end, so your program is allowed to remember. A program that remembers hears all 96 questions every tick, and a program that hears all 96 questions is looking at the whole warehouse. So instead of 96 tiny brains you build one big one. It keeps the whole board in its head and plans every robot's route together so they stay out of each other's way. When the game asks what robot 40 should do, it reads out the move it already decided.

That is a very different problem from the one the prompt appears to describe. The visible API says "return one action for one robot". The high-scoring code says "build a central dispatcher, remember every robot, reserve future cells, prevent head-on swaps, and keep cached distance maps around".

The whole trick is this small (condensed from Team 10's file):

class _Brain:            # holds every robot's position, plan, and reserved path
    ...

_BRAIN = _Brain()        # module global: survives from one act() call to the next

def act(observation):
    return _BRAIN.decide(observation)   # 96 "independent" robots, one mind

Every solution near the record that afternoon is that snippet wearing different clothes.

It holds across the whole leaderboard. Among submissions scoring 920 or above, every one I analyzed had the same broad ingredients: a module-global brain, A* or heap-based planning, and edge reservations. In the 895-919 band, almost all of them had the same shape too. Below 850, most submissions were missing that architecture.

The interesting part is thinking that for most of that afternoon, the LLMs were cooperating with each other through the public code, and the humans were mostly the network they did it over, and I do not mean that as a metaphor. A model generates the shared-brain planner for one team. The file goes up on the leaderboard. A human on another team downloads it, pastes it into their own model's context, and types "improve this". The second model recognizes the architecture, keeps it, renames a few things, re-tunes a rule, and ships a cousin of the same program. Neither human needed to know what the trick inside was. Judging by the edits, several never found out. The knowledge went model, to file, to model, and the people in the middle were couriers carrying a sealed envelope.

Again, somewhat hilariously, you can even prove the chain of custody, because the code kept fossils. For example, Team 10's file contains this line:

DEFAULT_CFG = (34, 0.10, 0.0, -1)    # robust fallback = 2x2/m1/entry single-config (922)

That 922 is a score from Team 10's own private tuning runs (I assume). So, technically it refers to an experiment nobody outside their team ever saw, and it means nothing to anyone else. The comment appears, character for character, in five of the six public files in this family, across four different teams. It rode through copy after copy because it is dead weight and no model had a reason to touch it. The only file it is missing from is Team 12's, the one whose single edit re-tuned that exact constant. Biologists trace ancestry through mutations that do nothing. Apparently, that works on code too. Isn't it beautiful?

That, for me at least, explains why four or five teams kept moving while most of the room froze. If your model found the global-state trick, or inherited it from a public file, you were in the 900-ish plateau. If it kept treating act() as a local robot policy, you were solving a much harder problem and probably did not know what you were missing.

After the event I kept going. The public best was 1008. A cleaned-up continuation of the same planner family reached 1024 on the official seeds. That version still uses Team 10's layout, but retunes the planner layer and adds a few audited late-game fixes.

Then the problem changed again. Once the seeds are known, the policy does not have to be a policy in the normal sense. It can be a replay: a 300-by-96 matrix of precomputed actions, selected by the initial target signature. That sounds silly, but it is legal under the same evaluator contract. More importantly, it makes debugging completely different. A live planner reacts to every perturbation, so improving one robot can ruin three others. A replay freezes the traffic field. You can edit one robot's day, run the simulator, and know exactly what changed.

That replay-edit path took the score to 1042: 350, 346, 346 across the three seeds.

On a new seed this wouldnt score that well. It is a recording of three known days, and that is the point. REFUGIO kept turning from a robotics problem into an information problem.

I really enjoyed the REFUGIO hackathon and I am very much looking forward to the next one (HackSpain). On a less postive note about the event, I kind of wish it had graded something beyond how many tokens you could spend (or how well those tokens were spent at least). There was no room for less technical people to show their strengths, like presenting a product to an audience, or someone very skilled at frontend showing their work. That is a great part of what most people end up doing once they build their own startup, and given the theme of the event was startups, the problem could have been aimed there.

It also somewhat reminded me of the argument that intelligence itself is deflating toward the price of the compute under it. The connection is less direct than it sounds, so bear with me. A wage premium for being the smartest person in the room is just the price of an edge, and an edge is only worth something while it is scarce. His bet is that once anyone can rent intelligence, it stops being scarce and that premium collapses across the whole economy. REFUGIO skipped ahead to the end of that story. The only graded skill in the room was knowledge work, the knowledge work was done by rented models, and every desk rented the same ones for the same couple hundred dollars a month. Inside that room, the deflation he predicts had already finished: being "smart" separated nobody (except those who could not afford it). Whatever still separated teams had to be something you cannot rent, and in its own twisted way, the event became a hunt for what that was.

I tried sovling the challenge on the plane home by hand. And I got nowhere near 1000 points. It is very hard to compete against agents when time is the constraint and deliveries is the output. I am still not sure what the deliveries metric was meant to measure. An event that grades deliveries gets deliveries. You get sixteen variations of the same planner and a race to spend tokens.

In any case, genuine thanks to the organizers. The challenge was good enough to keep forty-ish nerds off their phones for four hours (although I did check X a couple of times for the quick dopamine pump), and then good enough to survive a full week of me pulling it apart. Between the leaked seeds and the speedrun bet, it was fun. Also had the chance to meet and talk with some really nice people like Jon, Ruben, JustMicrock, Alfonso, ...

Cheers,