Progress feels very slow. It’s not for a lack of trying. It’s been a while since my last post, and while writing this I found that I forgot to send the previous one. Oops. So now you get a special double-length edition.

Here are some highlights from the last few months:

• Did my taxes and am now deeply dreading how complicated they will be now that I own an LLC
• Revived my SVN server.
• Starting to get competent at Rust
• Implemented more netcode. I did the first cross-continental playtest and lag was tolerable. It felt great to see it work.
• Added some primitive visuals to the game
• Implemented the game’s “vertical slice”
• Total LOC: 29,708; 100% rust; not vibe coded, I’ve been busy

Screenshot of the little spaceship that could. The force shields are rippling from a fender bender with a star. The star in the lower right is slightly distorted due to a lack of “Panini Projection”.

SVN Server

I set up an SVN server before joining google and it ran unattended for eight years. One day it didn’t turn back on automatically after a power outage and I checked up on it. It still worked, but the auto-update scripts I had on it never upgraded Ubuntu from the 2018 LTS to a modern one. It had been running an un-patched OS for years and didn’t seem to get hacked. Go linux. I re-imaged it and got it running again.

Let me know if you want my notes and setup scripts to run your own SVN server.

Why host my own server? Games are huge and dedicated hosting gets expensive. Source control providers assume that if you have GB’s of stuff checked in then you’re a big company and can pay big bucks. My basic game already has several GB of raw assets, and I don’t have big bucks to waste.

I’ll probably want to switch to GIT if I hire folks though because it has a lot more CI tools. But that feels a long way off, and SVN is a lot simpler and handles large binary files (like images and sounds) better than GIT.

Rust

There’s a meme about how Rust developers can’t help themselves but talk about how great Rust is. I can barely tolerate it. I think I would have liked it in college when I was more of an architectural purist, but professional experience has taught me the virtues of slapping stuff together at first and then focusing design effort on the stuff that matters. I feel like I only started to make headway when I stopped trying to take shortcuts and design the industrial-strength solution from the beginning, which is great for building nuclear reactors but terrible for game dev.

Rust is a weird language. It forces you to think in its terms.

It doesn’t have polymorphism or classes

It has “structs”, “enums”, and “traits”.

Rust structs are similar to classes in other languages. They can hold data and have methods.

Rust enums are essentially polymorphic classes instead of enums. Each enum instance can hold entire structs or they can be simple integers like normal. They’re similar to proto’s oneof.

Rust traits are basically interfaces in Java/C#. One big difference is that you can add them to classes that you don’t own (with restrictions). This is because they’re mostly syntax sugar for static methods.

As a game dev, I really miss polymorphism. A basic usecase is to support multiple types of entities in the game with some shared properties like health or movement speed, or even something as mundane as an ID.

A normal game might have classes like this:

abstract class Entity {
    EntityId id;
    int health;
    pub EntityId getId(){return this.id;}
    // ...
}

class Spaceship extends Entity {
    // ...
}


class Asteroid extends Entity {
    // ...
}

A basic rust port of this pattern would look like this:

pub struct Entity {
    id: EntityId,
    data: EntityType,
}

pub enum EntityType {
    // "Spaceship" enum instances contain a "SpaceshipStruct"
    Spaceship(SpaceshipStruct),
    Asteroid(AsteroidStruct),
}

pub struct SpaceshipStruct {}
pub struct AsteroidStruct {}

But this basic pattern often offends Rust’s Borrow Checker in real usage. I’ve found flipping polymorphism inside-out to be a little more forgiving but it creates a lot of boilerplate (not shown because it’s several pages long).

pub enum Entity {
    Spaceship(SpaceshipEntity),
    Asteroid(AsteroidEntity),
}

pub struct EntityBase {
    id: EntityId,
}

pub struct SpaceshipEntity {
    base: EntityBase,
    // my spaceship fields
}

pub struct AsteroidEntity {
    base: EntityBase,
    // my asteroid fields
}

Here’s a third form:

pub struct Entity {
    base: EntityBase,
    data: EntityType,
}

pub struct EntityBase {
    id: EntityId,
}

pub enum EntityType {
    Spaceship(SpaceshipStruct),
    Asteroid(AsteroidStruct),
}

pub struct SpaceshipStruct {}
pub struct AsteroidStruct {}

I haven’t built up an intuition for which flavor to use, and I often rewrite class hierarchies 3-4 times to try them out. They interact differently with the Borrow Checker, and it’s hard to guess what data access patterns work best in a situation.

The Bleeping Borrow Checker

I cannot emphasize enough how much I hate this thing, and it is an inescapable cornerstone of the language.

Theoretically, the borrow checker makes it impossible to have seg-faults, null-pointer exceptions, and many types of multithreading bugs, and it prevents this at compile time.

Here are some of the things it protects against:

// Returns a pointer to a heap-allocated string
// The string would be de-allocated at the end of the function and the pointer would be invalid
fn foo() -> &String {
    &"hello_world".to_string()
}

// Creates a pointer to an array element and then resizes the array
// Causing the pointer to reference invalid memory
fn foo() {
    // Create an vector (list in C#/java) with a one-element backing array
    let mut vec = Vec::create_with_capacity(1);
    // Add something to it
    vec.push("hello_world".to_string());
    // Store a pointer to the first element
    let hello_world = &vec[0];
    // Adding another element forces it to re-allocate the array, making the pointer invalid
    vec.push("goodbye_world".to_string());
}

The compiler says that you cannot modify the underlying array while you have a pointer to one of its elements, which is an easy-to-forget edge case.

But it also blocks mutably accessing multiple array elements simultaneously, even if it’s safe to do so:

fn foo() {
    let mut vec = Vec::create(2);
    // Add something to it
    vec.push("hello".to_string());
    vec.push("goodbye".to_string());
    
    // Store pointers with write access to the first & second element
    let mut hello = &vec[0];
    let mut world = &vec[1];
    
    
    hello.append(" world");
    goodbye.append(" world");
}

Here is a basic nested-for-loop that calculates the force of gravity between particles. It works fine in every language but Rust.

struct Particle {
    position: Vector2,
    force: Vector2,
    mass: f32,
}

fn calc_gravity(particles: &mut Vec<Particle>, gravity: f32, delta_time: f32) {
    // for-loop, iterate over all integers between 0 and # of particles
    for a_index in 0..particles.len() {
        // set 'a' to be a pointer to the 'a_index' particle
        let a = &mut particles[a_index];
        // for-loop, iterate over all the other particles
        // The borrow checker forbids the next line and everything that follows
        for b_index in a_index + 1..particles.len() {
            // set 'b' to be a pointer to the 'b_index' particle
            let b = &mut particles[b_index];
            let delta_pos = a.position - b.position;
            let dist = delta_pos.length();
            let force =
                delta_pos * (gravity * delta_time * a.mass * b.mass
                  / (dist * dist * dist));
            a.force += force;
            b.force -= force;
        }
        a.position += a.force * (delta_time / a.mass);
    }
}

What’s going on…

This creates a write lock on the whole array for as long as “a” is in scope.

let a = &mut particles[a_index];

Even checking “particles.len()” again is illegal for some reason. Maybe those pointers are causing the issue? So if we replace them with basic array access it will work…

fn calc_gravity2(particles: &mut Vec<Particle>, gravity: f32, delta_time: f32) {
    for a_index in 0..particles.len() {
        for b_index in a_index + 1..particles.len() {
            let delta_pos = particles[a_index].position - particles[b_index].position;
            let dist = delta_pos.length();
            let force = delta_pos
                * (gravity * delta_time * particles[a_index].mass * particles[b_index].mass
                    / (dist * dist * dist));
            particles[a_index].force += force;
            particles[b_index].force -= force;
        }
        // Now this line is illegal
        particles[a_index].position +=
        particles[a_index].force * (delta_time / particles[a_index].mass);
    }
}

The error message is “cannot borrow *particles as immutable because it is also borrowed as mutable”. Splitting the reads and writes into separate lines gets it to compile:

let delta_pos = particles[a_index].force * (delta_time / particles[a_index].mass);
particles[a_index] += delta_pos;

This isn’t much of an issue when making standard business applications. I probably only needed double-mutable-access to an array about once a year at Google because business applications are more functional and input-output focused. The borrow checker was legitimately helpful when I made the backend server for my game. But it is a huge pain when making the actual game because everything is interconnected in video games.

Say you have an array of game entities, and one of them shoots a missile, which creates another entity.

fn step_game(entities: &mut Vec<Entity>, input:&Input) {
    for i in 0..entities.len() {
        let mut e = &mut entities[i];
        e.update(entities, input.get_input_for_player(e.owner));
    }
}

impl Entity{
    fn update(&mut self, entities: &mut Vec<Entity>, input:&PlayerInput) {
        if self.can_shoot() && input.is_shoot_button_down() {
            entities.push(Entity{..});
        }
    }
}

This doesn’t pass the borrow checker for a lot of reasons.

Network Architecture

Implemented the core game logic using a “Deterministic Rollback” architecture to allow for fast-paced action while allowing for global effects like gravity. With deterministic rollback, all players share an identical simulation of the game world and only transmit inputs to each other. They also keep a second copy of the game world that is executed slightly ahead of the server using anticipated inputs. A “Rollback” happens when it gets authoritative inputs from the server. It reverts to the last good state and re-simulates the subsequent frames with new data. It’s undoubtedly overkill for such a simple game, but I want to use this architecture in the future for more ambitious projects so I’m thinking of this as a “hello world” for the technique.

In order for this architecture to work, it’s critical to build a deterministic “headless” kernel of the game that you have full control over. This is almost impossible to build directly using a fully-featured game engine like Unity or Unreal because the engine doesn’t let you have full control. With this technique, the game engine is merely a renderer and draws the current state of the game instead of driving it.

This approach comes with some benefits:

• Supports complex, simulation-heavy games
• Easier to unit test core game logic
• Difficult for players to cheat
• Relatively easy to switch game engines later if I want to
• (hopefully) Lower server costs because they only need to transmit inputs

The main downsides are:

• Cannot use as many built-in features of game engines
• Must build it this way from the beginning
• Player input can feel laggy. Alternate methods that trust the player’s machine allow you to send messages like “I turned left and dealt Bob 500 damage”, which feel instantaneous but are easy to cheat.
• Need to reconcile differences when re-simulating frames. Ex: You get a packet from the server dated ~0.2 seconds ago which causes something to explode 0.1 seconds later, which is still in the past from your perspective. How do you render the explosion? What if instead, it prevents an explosion, and you need to delete special effects you already started?

I made an end-to-end implementation of this over the last couple of months, and it actually works. I had a playtest with four people, one in California with 80ms ping, and they didn’t notice lag or have desyncs. I was shocked.

Photo of the first working cross-continent test

The networking is pretty complicated, and I have the engineer’s curse of being so in-the-weeds that I can’t describe it anymore, but I’ll try anyway.

Lag is most noticeable when it’s inconsistent. If all of your inputs are exactly 500ms behind, you’ll get used to it. If sometimes they’re 10ms and sometimes 500ms, you won’t. So I schedule all player inputs a little bit into the future.

The game server keeps track of the current realtime tick and the “locked input” tick. When the server “locks” ticks it means that it’s done accepting input for them and the game can advance. This way players know what the real state of the game is. The locked tick can be in the future or even in the past. Set it too far in the future and players will feel the input delay. Set it in the past, and laggy players will teleport as their input causes the game to be recalculated. The server side is relatively simple because it decides what time it is and what events actually happened.

Meanwhile, the players’ computers are trying to guess what tick the server is on and stay in-sync with it. They are running 2-4 copies of the game simultaneously (2-4 depending on how you count).

1. The server-synced universe that only executes locked ticks. It exactly matches the server at any given timestep. It is almost always slightly behind realtime.
2. The realtime universe that executes the local player’s inputs before they’ve been acknowledged by the server and executes another player’s inputs before the ticks get locked.
3. The game that people actually see. The one with spaceships flying around your screen. This isn’t a real copy of the game, it’s the render layer. But it has to stay (mostly) in-sync with the realtime one and update when things happen retroactively.
4. A “no-op” game for when we haven’t gotten a “lock tick” command from the server in a while, like if no one clicks for a few seconds. This is an optimization and used situationally. I can’t blindly advance the server-synced universe because it’s possible that the lock tick command is delayed.

I think I would need another 10+ pages to describe the system in depth, so I’ll save it for future editions. Getting this to work is a huge accomplishment because it means I’m done with netcode for a while. The huge advantage of this is that each client only sends inputs, like “the joystick is aimed at 22 degrees and they are pushing the shoot button down”, and then everything stays in sync after that. So I can add features without messing with the network stack. My next big networking-related hurdle is keeping the render layer in-sync with everything and handling spawning and despawning historical explosions as packets come in.

I think I’m done for a while with the little Starcraft map that I talked about in the first edition. Now all the remaining work is polishing it, balancing it, and actually making it public. But that’s actually a ton to do, and I have my hands full.

Factorio is about to release their last big update, and all mods need to be updated to it. I have 30 mods and I really don’t want to. I’ve gotten random “offers” from folks to take over maintaining them for me, but they basically want to take ownership instead, and I’m not into that.