Game Design, Programming and running a one-man games business…

Category: programming

Pak files

Posted on June 30, 2020

I just added pak file support to Democracy 4. Its something I already had coded for an earlier game, but I had to do a bit of fussing to get it to work properly with democracy 4.

Pak files are basically big phat files that contain other files inside them. If you are a new developer, you probably have no idea they exist, because you probably use unity and AFAIK they handle it for you. Pak files are pretty old school, as I recall Doom and Quake used them (maybe called wad files), anyway the principle is pretty simple:

A pak file contains two sections, an index that tells you where all the other files are inside the main section, and a big phat list of data that is the contents of those files. All of this gets stuck in a single flat blob of binary data. A class exists that lets you grab the memory address of the file you want if you pass in the name of the file, so hopefully to anybody who didnt write the pak file code, using it is easy. You can read the contents of a file just like its on disk, except you have to use functions that read from memory, not ones that explicitly read disk files.

In my case, that meant stepping into the engine code, and the opengl stuff that reads in graphics files (we are only using a pak right now for dds and pngs), and just changing the contents of one function. Instead of using this code to load the data ready for creating a dds:

fread(filedata, filesize, 1, fp);

I now use this

memcpy(filedata, GetPakFiles()->GetData(pentry->StartOffset), filesize);

Big deal :D. Similar changes happen for pngs. To keep things super-simple, the old code for loading a file will run if the pak file reports it cant find that file, so you can stick a ‘loose’ png or dds file into the bitmaps folder structure and it will still get found, and the rest of the code doesnt even know the difference, which is perfect for mod support.

Most of the hassle in getting this to work was just writing some code to enumerate (that means list really…) the contents of a folder from within the pak file. I had to support this for stuff like minister profile pics, because the code previously would ask ‘what files are in this folder, I need to know so I can select a random one’, and now that code gets handled by the index at the start of the pak file instead.

So why bother with this?

Basically speed. Do you know wwhat the read speed of your hard drive is? Checking a random new one on amazon.com shows 6Gb/s. I assume thats bits not bytes, so thats 750MB per second. My hard drive is a little bit older, so lets say 500MB/second. I’ll just copy a big chunk of the source and obj files from D3 to another disk, brb…

Ok…copy speed is between a low of 1MB/second up to 100 MB/second. Wow. Thats so much slower. Why?

A HUGE amount of bullshit happens on a PC when you access files. To simplify it, it goes something like this: *deep breath* You ask to read in a file. The OS then looks up the file table to check that file exists. it then asks the security system if the current user has permission to access that file. When it gets a yes, it then opens that file, and sets attrributes so other processes will know that file is in use. The antivirus software then kicks in, and hooks into the file read so that it can check to see if that file is excluded from scans or not, and gets ready to analyze its contents. The O/S then has to use the file-table to work out where all the various scattered chunks of that file are, and start reading in each block. This means talking to the driver, and ultimately to the hardware, which may also have to check its cache to see what blocks have been cached and whether or not it has to start the glacial process of spinning an old physical drive or not (faster with SSDs obviously).

THEN! when the file read is complete, we can close that file again, notify the system that its not in use by our process any more. We can then start the process of opening the next file in our list..

You do that bullshit for EVERY DARNED FILE. But the good news is… if you have a big phat pak file…you do it once. Just once. The rest is free.

So Democracy 4 goes the extra mile, because our pak file is small (only a few hundred MB). We dont just open the file on startup, we stream all 200MB into RAM. That should take way under a second. We then have the entire file system of dds and png files in memory already, and able to be loaded almost-instantly into our engine. (RAM->VRAM is mega fast)

99% of players will not notice the speed difference. But if you have especially shit anti-virus running on a laptop, in low-battery mode, with an extremely fragmented hard drive, running democracy 4 on a train, you will be glad I bothered. Its really easy to code. My PakFile code has 263 lines in it. Many of them are whitespace or comments.

Democracy 4: The fixed income rewrite

Posted on June 23, 2020

About a week ago I had this mad idea that it would be cool to plot every single Democracy 4 voter’s wealth on a graph, so that you could see where they were clustering in a nice easy-to-understand way. Within an hour it worked, within a day, a new rewritten version that looked much nicer with blue dots on it was done, and I was tweeting, and people were saying ‘yay’, and then everything went fucking mad.

By hovering the mouse over one of those blue dots, you could see a breakdown of how that persons income was affected by every government policy or situation, indirectly, through their membership of voter groups. This data already existed in Democracy 3, it was easy, it was just GUI code, and done really quick… and that showed me what an absolute mess the simulation was…

Democracy 1,2,3 and 4 are all coded as a homebrew neural network. Every neuron has a value either capped from 0->1 or -1->+1. Everything in the game is a neuron, a voter, a voter group, a policy, a minister, an event…everything. Voters also have an ‘income’ neuron which tracks how much money those voters have. So…in supersimplistic terms if you want to know why Bobs income is 0.78, you look at the 21 weighted inputs from all the voter group income neurons, and you see all the +0.2, -0.1,+0.32 etc, that adds up to 0.78. If you want to go one stage further up the hierarchy you can track the origins of those effects to policies etc.

Thats worked for 3 games perfectly. But its a crap system.

The trouble is, peoples incomes are on a 0->1 scale. And all effects are percentages. So for example if free bus passess give retired people a 0.05 income boost, that increases the income of all retired people by 5%. Fine?

NO

Because working class ex-street sweeper mavis just got a bus pass worth $500, but retired hedge fund manager Boris just got a buss pass worth $15,000. WTF? why does democracy hate poor people? The problem is that we have only ever been able to use effects to apply percentages to incomes. That means EVERY benefit, or tax, or effect is proprotional to your income. That means lambourghini drivers pay more in car tax than skoda owners (maybe intentional), but means the state pension depends on how wealthy you already are.

Its fixed. it was hard.

Basically I have had to code an entirely new shadow system of fixed-income neurons that can cope with values beyond 1, and then (this is the hard bit) written code that stitches it all back together internally so that we can still use the same mechansims to move people between middle-income and poor etc, and still display everything in the UI as though nothing has changed.

This was hellish, because it also means restitching together lists of totally different UI items on the fly with different calculation methods to come up with a result that looks the same as it used to. Its taken a week of fixing edge cases, checking, altering UI, and writing lots and lots and lots of code which mostly will go underappreciated :D.

But thankfully it now works, and it means we can have effects in the game which are +10% income of retired people, and also effects that are +$10,000 income of retired people, as the designer or modder sees fit. This means helicopter money can actually be fixed for everyone, free school meals no longer serve foie gras to rich students, and so-on.

You wont notice it immediately but its a massive improvement in the underlying simulation code in the game. It stressed me and tired me out so much to do it that im forcing myself not to code today so I can recover.

Coding vs Software Engineering

Posted on May 16, 2020

This is a topic I feel strongly about, but at the same time I am very aware that its very difficult to get across in text, because its not something you can really illustrate with a single line of code, or a witty cartoon or a small diagram, so I may go on a bit here…

I have been looking at code *not written by me*, and also talking to friends learning some new stuff who are also working with other peoples code, and have been reading a book on this topic, so my head is full of opinions on the topic of coding versus software engineering. let me first explain the difference.

‘Coding’ is the skill of understanding syntax and principles of how programming works, and slapping together a bunch of code that makes something happen. This is not *that hard*, and in fact yes, you can buy totally serious books that claim to teach you C or C++ in 21 days or less, which is laughable…but yes it does allow you to write code that compiles without errors and does the thing you want it to do.

‘Software Engineering’ is like coding, but much much HARDER. Mostly its about the scalability and long term usability of what you code. Code may ‘work’ in the same way that replacing a key component of an old car with a coat-hanger or a piece of string may *work*, but its likely going to go wrong at some point, nobody else will understand what it is or how it works, and when you try to scale it up, everything may completely fall to bits.

Software engineering is a pain because the best way to really get good at it is probably just experience of writing very large programs again and again and again, with different people, on different platforms, with different requirements, and having people criticize your code, or finding bugs in it, or having to revisit it five or ten years later to fix stuff.

The problem is that to 99% of people, and even 95% of coders, the difference between coding and software engineering is actually REALLY hard to spot. because many coders are managed (especially in the games industry) by non-coders, they aren’t even encouraged to get good at software engineering, because frankly the boss doesn’t know what it is.

When you are working as a coder, in crunch, at a game studio with deadlines, generally speaking the boss wants result X by date Y. The big problem is that result X is really shoddily defined. If ‘compiles and runs and the QA team couldn’t make it crash’ is the criteria, then LOL, yeah done easily mate. Unfortunately anything beyond that level of skill goes unrewarded, because its REALLY hard to spot.

Luckily I’ve worked for some very clever coders. My first coding boss (at elixir) was Dave Silver, who is now a mega-celeb in the world of AI at DeepMind. My second coding boss was James Brown (Lionhead), who now spends his time replicating conways game of life using lego for some reason. Both of them were very clever, and I’m a better coder for working under them. I learned a lot from them, not about *code* (which you can get from a book) but about software engineering.

If you haven’t already read ‘Code Complete‘ I really recommend that you do. Its excellent and is probably step one on the path to this stuff. The next things you should do are to work on a BIG project with other coders, and also work on the complete ‘project lifecycle’. This means, you start off with nothing, and finish when the project has shipped, and gone through multiple updates, ports and patches. Only then do you really know if the architecture choices you made at the start are correct.

A fairly simple blog-post style tip on this stuff concerns feature/syntax use and what I call the ‘gunslinger’ attitude. Take this line of C++:

X = X +1

Pretty much anyone (coder or not) can tell you that this adds 1 to the value of X. You can also write this

X += 1

Which does the same thing actually, and theoretically is very very very slightly faster because X is only evaluated once. However, its dark times indeed if in 2020 we cant expect a compiler to realize this and do that sort of thing for us.. Lets get a bit more vague…

float fInitA = InitA > 0 ? ( float )InitA : 1.f;

WTF? Now I am a C++ coder, so I can understand this… but I have to actually engage my brain to do so, which slows me down. Its not immediately intuitive to my half-asleep brain exactly whats going on here and… It really does not have to be written this way. You can just do this:

float fInitA = (float)InitA;
if(InitA < 0)
{
  fInitA = 1.f;
}

And OH MY GOD THE HORROR, its 5 lines of code instead of one. My god. What a n00b. Obviously this idiot doesn’t know about the C++ ternary operator and its syntax. The fool!

And yet its actually readable, and much easier to debug because its multiple lines allowing for breakpoints. The longer simpler version here is much BETTER code. And thats generally IMHO a principle that you can stick with. The trouble is, some coders adopt a ‘gunslinger’ attitude where they are presumably living out dreams of alpha-male dominance through writing the most complex obfuscated mess imaginable. Believe it or not your job as a coder is to write CLEAR and MAINTAINABLE code. You do not get fined for every line you use, and you do not earn points for confusing the people working with you.

There is a very ‘macho’ culture in programming, built around showing off, and using obscure stuff that you just learned. This is nuts. Just because you learn how to use a certain feature/function/syntax does not mean you HAVE to use it everywhere. I’ve worked with coders like this. Its a nightmare.

Its a worthy goal to write code that someone who isn’t even a programmer can look at and go “errr… I think I can see what you are doing here.”. This is because really GOOD code is code that can be understood by someone you have never met, five years later when a bug has been found and they need to work out if its in that function or not. If you are writing a tiny program thats only 1,000 lines of code and nobody else will ever see it, and you will never edit it then…ok maybe you can hack it together, but a proper software engineer always writes code that can be maintained.

Programming is a HUGE topic, and to get good at it, to get REALLY good at it takes an entire lifetime. I started coding aged 11, which is 39 years ago. I think I’m pretty good at C++ now, but not an expert, and its the only language I’m comfortable with. The internet and its many youtube vids and forums have spawned an attitude that you can learn to code one summer, or during lockdown, and…yeah not really. You can learn to hack stuff together by copying and pasting from stackoverflow…but thats really not proper software engineering.

Its worth saying I’m not exactly at the end of the journey myself yet either. The code for Democracy 4 is *not perfect* by any means. Some bits are hacky, there were some fundamental design decisions I made about 15 years ago with the basis of my GUI library that are embarrassing but still there (of COURSE buttons should be a subclass of window you idiot!), but overall my code gets better with each game.

I coded about 5 games before I realized that having a decent separation and naming convention to keep GUI and Simulation code entirely separate was a worthy thing! I probably coded 8 games before I had a rock-solid translation-management system that meant not a single line of text exists in code. It took me maybe 10 games to get threading to work safely, and maybe another 2 until I had a rock-solid and highly-optimized multi-threading system. I didn’t really start to use the power of macros for about 10 games. I’ve only just (in the last 2 games) really got my code for setting up configurable color palates to be usable.

I had most of the technical knowledge to do all of that stuff about 15 years ago, but to do it *well* and to know how to arrange things, and to set them up to be re-usable, optimized, stable, and readable… thats what those extra fifteen years were spent doing.

The VAST majority of comments you read online about programming, especially games programming are written by coders, not software engineers. They suffer a lot from the delusion that they have mastered code, because (as is natural) they don’t know what they don’t know. Its REALLY hard from a distance to spot the software engineers from the coders, but in my experience the amount of time they have been in the industry, and the number of large completed projects is a really good sign.

A final way of spotting the difference: If a lot of someones code has been copied and pasted from stackoverflow or pastebin then… yeah. Thats not a software engineer.

Rethinking the game dev productivity gap

Posted on February 12, 2020

Its really only in the last six months I’ve realized this, and I’ve been an indie for twenty+ years and coding for 39 years, so yeah…this took a while to sink in.

I am frustrated on a CONSTANT basis by the lack of productivity of almost everybody in the universe. I am especially irritated by the low productivity of most people in game development, and most indie devs. I almost never read about the development schedule of a game, (mostly through post-mortems, interviews or chatting to actual humans), without being shocked at how long it took to do stuff.

For most of the time, I have attributed this to an attitude. I work pretty much every day, and for most of the day, although my schedule these days is deliberately lighter than the early years. I’m prone to going out for lunch or to coffee shops, but then I’m prone to working all day Saturday and Sunday, so YMMV. I also often reply to forum posts, youtube posts, blog posts and emails in the evenings from my laptop. I’m often thinking about code when I’m not writing it.

Because of this, I find talking to people with a less work-centric attitude to be infuriating. It boggles my mind how long it takes most devs to add what seem like easy and simple features to games. I am constantly told that I am woefully inefficient because I don’t use unity, but still seem easily capable of working faster in terms of adding features & content than the very people who berate me for not using such productive tools.

So yup, I often think such people are just lazy. Or do not have the same attitude as me, or do not realize just HOW HARD it is to compete in this industry. In other words I think that their mindset is less focused, and its a personal weakness on their part, because yup…i’m a bit obsessed.

But now..I’m thinking there are two other things that explain the disparity better.

First thing: Lack of distractions. I have 3 cats, and live with my wife and these 3 relatively-low-maintenance pets, but no kids. I have a hobby of playing the guitar, which I make myself do a bit each day, but thats it. I am not having to take time out to walk the dog, pick kids up from school, drop kids at school, answer questions from kids, sort out other stuff for kids, walk the dog again, and so on. My wife is a writer, so has the same introverted ‘happy to be alone with a keyboard’ daytime work schedule as me.

Nobody ever phones me, unless its an elderly relative. I have a call screener device to prevent phone spam, and we live in the middle of nowhere. Nobody knocks on our door trying to sell us anything. There is very little noise. Its the perfect set up for zero distractions. If you possibly can do ANYTHING to reduce the distractions in your day, do it.

The second thing: experience.

This is the big one. I’ve been coding for 39 years. Thats an AGE. When I first started learning computer programming, this person was US president:

Image result for jimmy carter

Yup, exactly.

That means any silly mistake you can make when designing code…I’ve done in thirty times. 95% of my conversations with fellow devs when I’ve hit a bug go like this:

“Could be a memory-bounds issue…?”

Me: “Nope”

“Could it be that you deleted the object?”

Me: “Nope”

“…Maybe its a multi-threaded synch issue?”

me: “Nope”

…and so on.

Now that sounds super arrogant, like I think I’m the bees knees at C++. Actually I am not. I am not that good an all-round programmer *at all*. I am VERY good at learning in excruciating detail about the elements of C++ that I use, and nothing else. Because I work for myself I have no marketable need to be an all-rounder. I don’t need to learn ‘agile’ or ‘scrum’ or ‘.NET’ or RubyOnRails or whatever the hell jobs ask for this week/month/year. Its irrelevant to me, so I can be VERY good at VERY few things. This is hugely efficient.

Plus… again, trying to put my arrogance in context here… language proficiency is language proficiency, whether its English or C++. C++ is way less forgiving than English, but still…how good at English were you when you had been speaking it for just five years…versus thirty years? Hardly an exact comparison I know, but I think its a good mental exercise. I get better at C++ every year, but in a way that is not exactly how you would think:

I do NOT know more ‘clever tricks’ than a newcomer to C++. I do NOT have a better memory of the syntax of C++ than a newcomer. I do NOT type *that* much faster. I do not make use of a wider range of the standard C library than anybody else. I don’t do any of those things. What I *do* better, is that I have just learned from my mistakes.

A lot of mistakes.

I used to take the odd coding test in job interviews back in the day. These tests are good for one reason: to see if the candidate has any clue about syntax. Thats pretty much it. The amount of code required otherwise renders the test pretty much useless.

The trouble with C++ is that it attracts hotshot coders. These are people who think a super-complex algorithm, or the algorithm that uses the most clever combination of features will somehow get them more sex/money. This is predictable and sad, but not useful in terms of real productivity.

The best code, is the combination of three things:

Simplicity, Performance, Readability.

A lot of really, really good code looks fairly boring, because boring is often simple, fast and readable. The worst possible insult you can get from a senior/lead programmer with experience is this:

“That looks a bit over-engineered”

Its truly a damning insult, but you only really realize how insulting it is after about thirty years of writing crappy code. I wish I knew of an easy way to help people fast-forward those thirty years and develop the skills you have at the end of it, without those thirty years but I don’t think I can. The only advice I can offer is this:

  1. Write as much code as you can. Not over-engineered nonsense, but just code a lot. Put the hours in. At least the thousand obviously, but likely way, way more.
  2. Get a job with a really experienced coder and ask for criticism of your code. Only someone who works with you all the time will read enough of your code to really give you structural, high level advice on why your code sucks.
  3. Read code-complete at least twice, if you have not done so already.
  4. Get cats not dogs. Cats don’t need a walk.

Hope that helps someone :D

Website optimization in 2020

Posted on January 2, 2020

Sooo… in a random moment of surfing a few months ago I encountered an article of the webp format and how it was faster, and how it was a Google thing, and they therefore wanted you to use it. I knew I had a server move coming up (long story) so delayed worrying about it until now…

Basically webp is like a super-amazing improved replacement for PNG that is MUCH more efficient. Full details here, but for example one of the files I converted to webp went from 942k to 189k which is not to be sneezed at. I still cannot tell ANY difference when I look at both images. Sadly wordpress is too useless to upload webp, but here is one embedded:

…and here is the png:

…exactly.

So with this in mind, I replaced some of the larger images on the Production Line webpage with webp equivalents to speed up the loading. This IS WORTH DOING, but its also worth remembering that some Luddites may be using stupidly old browsers that cannot cope with webp, and you need to also have the option of a png for these people. You can do this with some magic modern html like so:

<picture>
	<source type="image/webp" srcset="images/thumb.webp">
	<img src="images/thumb.png" width = "1000" height="563" >
</picture>

That basically says ‘show this webp image, unless you don’t have any idea WTF that is, in which case here is an old fashioned png. All of the attributes for your image still go in the src bit.

That got me a nice speed bump, but some test done both with googles site checker and also the popular web speed test showed I was mainly slowed down by third party stuff, specifically humble bundle widget and youtube embeds. (I embed 2 large youtube videos on that page). This is annoying, but after a lot of fiddling I found a reliable way to get around the slow youtube stuff.

What I did was have 2 identical sized elements on the page for each video. One a ‘panel’ and the other a ‘vid preview’, which was basically a big thumbnail made by me (webp obviously) with a fake play button to simulate youtube. The code in the actual page body looks like this:

<div id="panel">
<table width="100%" align="center" cellpadding="0"cellspacing="0">
	<tr>
	<td align="center" width="1000" height="563">
	<iframe id="trailer_youtube" width="1000" height="563" src="" frameborder="0" allowfullscreen></iframe>
	</td>
	<tr>
</table>	
</div>
				
<div id="vidprev">
<table width="100%" align="center" cellpadding="0" cellspacing="0" onclick="myFunction()">
	<tr>
	<td align="center" width="1000" height="563">				
		<picture>
		<source type="image/webp" srcset="images/thumb.webp">
		<img src="images/thumb.png" width = "1000" height="563" >
		</picture>						
	</td>
	<tr>
</table>	
</div>

In practice what this does is say ‘here is an embedded iframe called ‘trailer_youtube’ with NO source. And here in the same place is a big phat image. BTW if we get clicked call myFunction()’.

Then at the top of the page in the header we add some code:

<style>
#panel, .flip {
  font-size: 16px;
  text-align: center;
  color: white;
  margin: auto;
  z-index:1;
}
.vidprev
{
z-index:2;
}
#panel {
  display: none;
}
</style>

…which sets the z index (bottom to top stacking) of the two panels, and then we need some actual code for when the thumbnail is clicked on, also in the header:

<script>
function myFunction() {
  document.getElementById("trailer_youtube").src = "https://www.youtube.com/embed/IhGTKBAC94c";
  document.getElementById("panel").style.display = "block";
  document.getElementById("vidprev").style.display = "none";
}
</script>

…that code basically grabs the youtube panel, sets it visible, and assigns it a proper valid youtube link, handily deferring any connecting to youtube.com until we need to. it also hides the thumbnail. The result is a MUCH faster page load (roughly half the time).

In addition, I used some javascript called ‘lazy sizes’, to make the loading of some items lower down the page asynchronous, so they wont even get loaded until the visitor scrolls down. source:

<picture>
 <source type="image/webp" data-srcset="images/resources.webp" class="lazyload">
<img data-src="images/resources.png"  class="lazyload">
</picture>

and that requires an extra include:

<script src="./js/lazysizes.min.js" async></script>

The result is pretty good, and raises Google’s estimation of my site speed quite a chunk. That will be good for SEO with Google, and they are basically the only search engine that counts so…yay :D. here is the full waterfall chart: