Category: programming

I am well aware that my game Democracy 4 is not exactly slow with huge framerate issues. However, optimization is fun! or at least it should be, but in practice, getting profiling to work on remote PCs is not exactly easy. I have basically used every profiling software imaginable and still have not got one that I think really does the job well…

I have basically wasted about an hour today trying to work out why I couldn’t get the intel vtune amplifier stuff to work with event based profiling and get rid of this pesky error that was clearly nonsense about ‘not able to recognize processor… until I finally realized that I actually have an AMD chip in my (relatively) new PC so…yeah… That drove me to try out the AMD uProf profiler, which is something I had not used before.

It took me a moment to realize that this software, good though it is, does not suggest to you ‘hey if you run me in administrator mode I will show you 50x more config options’ but luckily I worked that out. My first act was a brief run of Democracy 4, starting a new game then immediately going to the next turn. In the list of functions taking up all the time (and ignoring windows system functions) I get this list:

Which is about what I would expect. The game is implemented as custom-coded neural network structure, hence the terminology. Mostly everything is a neuron, and most of the processing is where each neural effect (the links between neurons) processes its equation, and then neurons do some math on their inputs and outputs.

The inner machinery of the neural network ultimately comes down to that top item there:

SIM_EquationProcessor::Interpret Value.

This is code that basically takes those equations in the game’s csv files like this:

StateHealthService,0-(0.4*x),2

And actually calculates a value from that. There are 2,000 voters with about 10 connections each, pre-processed on a new game 32 times, so thats 640,000 equations right there, plus all of the actual simulation stuff layered before that. In other words, that equation processer probably runs a million times on a new game, and the equation might have 5 values in it, so max case is 5 million values get interpreted when you click on ‘new game’.

Can I speed it up?

First step is to see how stable these values are any way, so I’ll do an identical profile run and check that the +/- errors on different profiling runs are small…

I think thats pretty close. I definitely have numbers here that are in the same ballpark. So now lets try some optimisations to speed this puppy up. Looking at the top function with a double-click gives me a whole bunch more data:

The function is much longer than this, but thats mostly catering to relatively rare edge cases. Looking at the bits that actually have numbers on it show pretty clearly that its pretty much all about the pesky strcmp() call. A separate piece of code has already parsed the full equation of 0-(0.4*x), so I have a bunch of char buffers for each variable, declared like this:

char Vals[MAX_VARIABLES][32];

The thing is, do I need the overhead of calling strcmp() when I am only really checking for whether the first letter is x? Sadly I cannot JUST check that, because that would prevent we having a named variable starting with an x. Lets imagine this equation:

0-(0.4*xylophone)

Obviously not very likely, but theoretically possible. If the length of the buffer was 1, and the first letter is x, then thats a hit, but the question is, will inlining 2 manual checks be faster than a strcmp function call? Lets replace that code with

if(Vals[valindex][0] == 'x' && Vals[valindex][1] == '\0')

And check out the results:

Hmmm. Actually WORSE as far as I can tell. So it looks like whether we strcmp or not, just checking the value of two bytes at that point is slow. probably because its not immediately available memory? Its notable that the code at line 232 is super fast by comparison, as its just checking a bool value we cached earlier. Maybe I should try that? When I parse the function, just keep a bool for each Value, saying if its ‘x’ or not?

Whoahh. This looks like a pretty major speedup. 326 cycles versus 1,143. What the hell? why didn’t I do this earlier? Lets look at the line by line…

This is awesome. I then tried to make this code inline, but it seemed to not make things any better. I haven’t fully explored uProf yet, but it does do cool flame graphs:

Profiling UIs are great fun :D

There is a service I use that occasionally means I have to upload some files somewhere (who it is does not matter, as frankly they are all the same). This is basically a simple case of pointing at a folder on my hard drive and copying the contents onto a remote server, where they probably do some database related stuff to assign that bunch of files a name, and verify who downloads it.

Its a big company, so they have big processes, and probably get hacked lot, so there is some security that is required, and also some verification that the files are not tampered with between me uploading and them receiving them. I get that.

…but basically we are talking about enumerating some files, reading them, uploading them, and then closing the connection with a log file saying if it worked, and if not what went wrong. This is not rocket science, and in fact I’ve written code like this from absolute scratch myself, using the wininet API and php on a server talking to a MySQL database. My stuff was probably not quite that robust compared to enterprise level stuff, but it did support hundreds of thousands of uploaded files (GSB challenge data), and verification and download and logging of them. It was one coder working maybe for 2 or 3 weeks?

The special upload tool I had to use today was a total of 230MB of client files, and involved 2,700 different files to manage this process.

You might think thats an embarrassing typo, so I’ll be clear. TWO THOUSAND SEVEN HUNDRED FILEs and 237MB of executables and supporting crap, to copy some files from a client to a server. This is beyond bloatware, this is beyond over-engineering, this is absolutely totally and utterly, provably, obviously, demonstrably ridiculous and insane.

The thing is… I suspect this uploader is no different to any other such software these days from any other large company. Oh and BTW it gives error messages and right now, it doesn’t work. sigh.

I’ve seen coders do this. I know how this happens. It happens because not only are the coders not doing low-level,. efficient code to achieve their goal, they have never even SEEN low level, efficient, well written code. How can we expect them to do anything better when they do not even understand that it is possible?

You can write a program that uploads files securely, rapidly, and safely to a server in less than a twentieth of that amount of code. It can be a SINGLE file, just a single little exe. It does not need hundred and hundreds of DLLS. Its not only possible, its easy, and its more reliable, and more efficient, and easier to debug, and…let me labor this point a bit… it will actually work.

Code bloat sounds like something that grumpy old programmers in their fifties (like me) make a big deal out of, because we are grumpy and old and also grumpy. I get that. But us being old and grumpy means complaining when code runs 50% slower than it should, or is 50% too big. This is way, way, way beyond that. We are at the point where I honestly do believe that 99.9% of the code in files on your PC is absolutely useless and is never even fucking executed. Its just there, in a suite of 65 DLLS, all because some coder wanted to do something trivial, like save out a bitmap and had *no idea how easy that is*, so they just imported an entire bucketful of bloatware crap to achieve it.

Like I say, I really should not be annoyed at young programmers doing this. Its what they learned. They have no idea what high performance or constraint-based development is. When you tell them the original game Elite had a sprawling galaxy, space combat in 3D, a career progression system, trading and thousands of planets to explore, and it was 64k, I guess they HEAR you, but they don’t REALLY understand the gap between that, and what we have now.

Why do I care?

I care for a ton of reasons, not least being the fact that if you need two thousand times as much code as usual to achieve a thing, it should work. But more importantly, I am aware of the fact that 99.9% of my processor time on this huge stonking PC is utterly useless. Its carrying out billions of operations per second just to sit still. My PC should be in super-ultra low power mode right now, with all the fans off, in utter silence because all thats happening is some spellchecking as I type in wordpress.

Ha. WordPress.

Computers are so fast these days that you should be able to consider them absolute magic. Everything that you could possibly imagine should happen between the 60ths of a second of the refresh rate. And yet, when I click the volume icon on my microsoft surface laptop (pretty new), there is a VISIBLE DELAY as the machine gradually builds up a new user interface element, and eventually works out what icons to draw and has them pop-in and they go live. It takes ACTUAL TIME. I suspect a half second, which in CPU time, is like a billion fucking years.

If I’m right and (conservatively), we have 99% wastage on our PCS, we are wasting 99% of the computer energy consumption too. This is beyond criminal. And to do what? I have no idea, but a quick look at task manager on my PC shows a metric fuckton of bloated crap doing god knows what. All I’m doing is typing this blog post. Windows has 102 background processes running. My nvidia graphics card currently has 6 of them, and some of those have sub tasks. To do what? I’m not running a game right now, I’m using about the same feature set from a video card driver as I would have done TWENTY years ago, but 6 processes are required.

Microsoft edge web view has 6 processes too, as does Microsoft edge too. I don’t even use Microsoft edge. I think I opened an SVG file in it yesterday, and here we are, another 12 useless pieces of code wasting memory, and probably polling the cpu as well.

This is utter, utter madness. Its why nothing seems to work, why everything is slow, why you need a new phone every year, and a new TV to load those bloated streaming apps, that also must be running code this bad.

I honestly think its only going to get worse, because the big dumb, useless tech companies like facebook, twitter, reddit, etc are the worst possible examples of this trend. Soon every one of the inexplicable thousands of ‘programmers’ employed at these places will just be using machine-learning to copy-paste bloated, buggy, sprawling crap from github into their code as they type. A simple attempt to add two numbers together will eventually involve 32 DLLS, 16 windows services and a billion lines of code.

Twitter has two thousand developers. Tweetdeck randomly just fails to load a user column. Its done it for four years now. I bet none of the coders have any idea why it happens, and the code behind it is just a pile of bloated, copy-pasted bullshit.

Reddit, when suggesting a topic title from a link, cannot cope with an ampersand or a semi colon or a pound symbol. Its 2022. They probably have 2,000 developers too. None of them can make a text parser work, clearly. Why are all these people getting paid?

There was a golden age of programming, back when you had actual limitations on memory and CPU. Now we just live in an ultra-wasteful pit of inefficiency. Its just sad.

I don’t blog much about the nuts and bolts of programming, because I know enough to know that others know more than me, but increasingly I’m aware of how other people code, and feel that I probably do have *some* worthwhile advice on the topic. Not many people have been programming for 41 years, especially given how coding seems to skew young, so here we go…

First some background, I started coding in sinclair BASIC on the ZX81 aged 11. I took some time away from code to try and be a musician, and then worked in IT support, but I have worked as a full time programmer for the last 25 or so years. After a while its hard to keep track! I basically code video games in C++. Thats it. I know some php, but not much. I can probably remember some BASIC, but not much. Yes, I’ve been coding 25+ years in a single language.

Lesson #1 Learn a single thing well.

I sometimes do some stuff that would probably be a bit easier in C#, or Rust, or Java, or probably some other language I have not even heard of. I don’t care. I am a massive believer in knowing how to do one thing super well. This is very unusual, because these days, coders are hired as a shopping list of buzzwords. Adverts for coding jobs amuse me no end. They always want someone with experience of 10 different languages. Did the lead programmer have a problem deciding which to use? When you want a coder who knows 10 languages, you get someone who is mediocre in lots of different ways. Thats not ideal.

The ‘familiar with 10 languages’ trope is common because non programmers understand it. A coder with 10 languages must be more skilled than one with just 1 right? To a non coder, its all unintelligible gibberish anyway, they have no way to tell a really GREAT c++ coder from someone who just knows a bit of syntax. Recruitment consultants and idiotic managers without coding experience have created a world where everyone is truly rubbish at 100+ different programming languages.

We do not expect this in other fields.”Doctor wanted. Must be a throat expert / hearing expert / heart specialist / Neurosurgeon”. We also manage to get by in our lives speaking only a single language. Sure, french is more romantic, German more precise, Spanish more passionate, English more subtle, but we don’t change languages constantly in our day to day lives! Tip: If you work for yourself, learn ONE language super well. Only learn others if they are absolutely essential for the task in hand. You can do almost anything in C++.

Also note that being really good at one language does not mean learning every possible feature (see next lesson), but it means using that language so MUCH, in so many different circumstances, that you know the best way to get things done using that language. (by best I mean reliably, efficiently and legibly).

Lesson #2 Language features are optional.

C++ lets you override operators. It also supports templates and the stringizing thingy (## i think?). I don’t use any of them. I use maybe a quarter of C++. Most of the newer stuff seems like solutions looking for a problem. Its FINE to just stick with the feature set you find to be most usable for you. Nobody is going to point at you at parties and mock you in front of the opposite sex because you don’t use the whole range of C++ features in every program.

There are multiple ways to handle files in C++. fopen, iostream, CreateFile() etc. I use the old fashioned fopen() stuff. I know the syntax super well. I can type it as fast as I can speak. This is fine. I’ve never had someone leave me at the altar for not using iostream.

I am constantly told by people I am reducing my productivity by sticking with old, outdated and cumbersome systems when new-fangled ones have apparently done the same but better. Without exception I am more productive than everyone who tells me this.

Lesson #3 Legibility is the goal.

Coders start off as n00bs with no confidence or idea how to do things. They then go through a ‘gunslinger’ phase, filled with the coders equivalent of testosterone, where they write ‘impressive’ code, that uses cool obscure language features or fancy techniques to do whizz-bang stuff that impresses other coders at parties. This phase lasts about a decade, or maybe longer if you are in a big company with bosses that need impressing.

The ultimate stage for a coder is to realize that ‘impressive code’ phase is just so much adolescent bullshit. The goal of code is to be reliable, efficient and legible. That last one really matters. If there is a way to perform a task that is simple, clear, and a way that I have used 100x in the past, then thats the way I do it. There may be a ‘cool’ way to do that task that involves callbacks and threading and whatever the heck microservices are… but why over-complicate things when you can just write simple code that you will understand instantly on reading it in 10 years time. 99.99% of the impact of your code is on the users, not the other coders who might glance at it. My code probably looks simplistic, very literal, very accessible for even non coders to read. I’m happy about that.

Lesson #4 Consistent Formatting

No 2 coders ever agree on how to format code. I have my own opinions which I shall not bore you with, but here are two thoughts: Firstly, be aware that a lot of ‘standard’ ways to format code were dreamed up before we had intellisense and visual assist. If you are still relying on a naming convention that shows you what is a function, what is an enum and what is a variable… then its kind of pointless. We have syntax-coloring now. You don’t need to do that, and can get back some much needed legibility. No more LpSZ_mem_Name.

Secondly…and this is even more important, the goal of code formatting is to minimize the cognitive load on the person reading it. Most programs do complex things, and to debug or add features, you need to maintain a HUGE network of complex stuff in your head as you work. If you are having to mentally translate a lot of different variable-naming conventions and code layouts in your head as you read then you will be amazed at how much

HARDER

to

_read

This

cAn

then

BE.

Our brains seem very good and building up mental translation systems. if you have text in ALL CAPITALS then we adjust to it ok. Pretty much *any* coding standard is fine… as long as you have only one. This is another reason why I hate middleware, because obviously its all formatted differently. Having to chop and change mentally between different ways of doing the same thing adds huge mental overload.

If I ran a studio, the question ‘will you agree to 100% without question always follow the code style guide of the company or face immediate dismissal’ would be the first interview question for new hires.

Lesson #5 Be aware of the overhead.

Almost all modern code is absolute trash, total garbage, an embarrassingly badly constructed towering pile of crud that should bring deep shame on everybody who wrote it. People with a lot of coding experience tend to agree on this topic. I think its inevitable, given that the demand for coders vastly outstripped that number of people with experience. Maybe over time it will be redressed, but I worry that these days coders are just used to code being crap and do not understand its even a problem. You don’t need 20 years experience to post a useless reply on stackoverflow. Sadly.

Its REALLY instructive for you to step through every single line of code in the execution of the stuff you do. If you are an app developer relying on middleware that is closed source… i pity you. A simple step through one instruction of yours probably means running 5,000 lines of bullshit layers of code that you will never see. This is why we all have incredible supercomputers, and yet we stare at progress bars…

If you are lucky enough to have full source access, or have written it all yourself (yay!), then do the ‘step-through-everything’ test regularly. It will AMAZE you how much code seems to be run without actually being needed. The inefficiency of most code is mind-boggling, and its because hardware has outrun software so much that we can write code that runs at 1% efficiency and hardly anyone complains.

Learn how to use a profiler, and go through your code looking at where all the time is spent. Its actually REALLY interesting, and kinda fun. I enjoy spending time in stuff like aqtime, or vtune. If you have not profiled your code, then you have not finished it. You are not a software engineer if you have not done the post-coding analysis of what the code actually *does*.

Lesson #6: Avoid massive functions, and massive source files.

There are technical reasons for this, in terms of compile times etc, but it really comes down to the overall structure of code. If you have a function that is 100 lines long or more… then you have probably fucked up. If a single source file for a class is 1,500 lines long…then you have probably fucked up. (yes, I know about function call overhead. Are you working on spacecraft control systems that have to run at 5,000 fps with minimal power? I am guessing no, so don’t worry about that trivial overhead here)

The most common mistake that leads to buggy, hard-to-work-with C++ code is that classes are too big, and functions too long. A function does one thing. The clue is in the name. What tends to happen is people start off with a perfectly reasonable class and functions, then they add functionality (ha!) and the class sprawls, and the functions sprawl. What was once a perfectly reasonable Update(), becomes 100 lines long. What was once a reasonable Entity class suddenly becomes a behemoth.

This is normal. You now need to work out the cleanest, most sensible way to break that class into multiple objects or derived classes, and change the layout so that the code is done in more single-focused functions. This is not some bothersome task that wastes time… this is literally software engineering. This is the difference between a coder and a software engineer. This vastly increases the legibility and debugability of your code.

Getting code to work is just the easy bit. The hard bit is knowing exactly how to layout the relationships between objects and functions so that everything seems sensible, and organised. Knowing how to do this takes a lot of experience, and is normally the outcome of many bleary-eyed 3AM debugging nightmares where you stare at the big giga-function when you are nested 16 deep into curly brackets wondering what the fuck its supposed to do.

Call stacks are your friend! Call stacks are amazing. I have the call stack always open at the bottom of my screen. I offload my mental model of ‘how we got to be here’ into the callstack window. I find this incredibly helpful. Making sense of the flow of code through the callstack window is 100x easier than keeping mental callstacks in mind as you navigate giant functions.

Conclusion

These are my broad thoughts on stuff I wish I knew earlier. I’m sure lots of people disagree, and thats fine, I’m not starting a political movement here, just sharing my experience. Sadly, the awful state of modern coding is largely outside coder’s control, due to incompetent managers, feature-centric marketing and tech-ignorant recruitment practices. I hope its of some interest to solo coders or lead coders with complete control of how a codebase is made.