Game Design, Programming and running a one-man games business…

Category: programming

Fixing the latest Production Line Code Bottleneck

Its always worth occasionally grabbing a full run-time-profile of your game to see whats taking up all that processor power. Today I dug into it a bit with aqtime and wanted to see what was chewing up the PC the most on a (fairly) big map.

If I ignore child threads (that take a lot of the load-off in terms of route-finding and other things) and just concentrate on the main thread, I can see that 73% of a frame is in the drawing code, and 22% is processing. I suspect on crazy big maps that processing chunk goes higher, as the drawing code scales to some extent with the screen size, and a big map on a small laptop is still my target for improvement.

If I break open that 22% I get this:

SlotManager::Process is obviously the biggest problem here. Its also something that scales quite badly because on really jam-packed maps, there is a lot of slots, and for my sins, EVERY item placed down is a slot, even if its just a conveyor with one entrance and one exit… so whats going on in there…? 95% of the time happens in ProductionSlots (the rest is facilities or supply stockpiles…

…so quite a mixture here, which is a good sign really, as it shows no real obvious glaring screw ups! Because I’m really worried about maps with just a lot of conveyors on them I guess the thing I really need to look at is Processvehicle(), as the other top offenders don’t seem to be related to those theoretically simple slots…

Hmmm. so CheckExitSlotFree() is taking up a lot of time, and some digging around shows that in my short sample run I called that on average of 1,248 times per frame. Ouch. Thats a lot, but to be expected with so many slots, and them all wanting to know their immediate status. The sad thing is, this is likely to be 99.5% of the time, a lot of slots going *is the slot in front of me empty* and the answer coming back *no, just like it was the last 100 times you asked*.

There is a fundamental algorithmic screw-up here, in that polling a value, rather than being notified when it changes, is hugely inefficient. Its also incredibly reliable and fault tolerant, which is why its such a popular method… What exactly takes so long anyway in that function?

Aha… so it looks like a big chunk of the time is actually spent just notifying the popup GUI above a vehicle or a slot, what message to display which is almost *all* cases is none. This seems crazy, and clearly a target for optimization. On its own, this function is taking up a tiny part of total run-time, but a larger part of the main (blocking) thread, and certainly seems to be inefficient. What takes up most of the code? actually its the code that works out the width and height of any of the strings based on the UI font, and thus resizes that notice. Yikes, am I *really* calling that every frame?

Out of 1,992,097 calls to SetType() only 55,280 update it, so only 2.7%, but its still a lot. Inside that function we set the size and the text without even checking if its onscreen, or if we are zoomed in far enough to see it anyway. I can simply set that to be something thats done ‘lazily’ once I know we are onscreen and zoomed in…

Ok done that and…. Hahaha. That function now averages 0.0001ms vs 0.0007ms before the change. Thats a decent hours work :D

That reduces the overall time for a slot processing a vehicle from 0.0012ms to 0.0009ms. For big factories, thats a welcome change.

Creating a real working environment for game development

It drives me mad when I talk to some younger indie devs how little actual *work* they do. They are hardcore serious game devs, into game jams, and going to games conferences, and maintaining twitter, instagram and facebook pages about their game, and they often talk at shows, or attend talks, or tweet about talks, and watch tons of past talks, and play fellow indies games, and meet up at game dev meetups and try out the hot games and compare them to other games and…

…very rarely they sit in front of a keyboard and code a game.

This is a big problem if you actually want to make a living from games rather than just enjoy the ‘indie lifestyle’. FWIW, the indie lifestyle is easy. Dye your hair (or for extra points just part of your hair) bright blue, get an apple laptop, and cover it in stickers from games shows. Buy a GDC T-shirt (or for extra points, one from a smaller show), and spend at least three hours a day on social media. Bonus points for every 100 posts on gamedev subreddits talking about tech and marketing and design issues. Super bonus points for getting into heated twitter arguments about whether or not games are art or inclusive.

Oh obviously, you need to have a name and genre (maybe even a game jam concept?) for your game, so you have something to talk about.

This is all fine, and reminds me a LOT of the guy who persuaded me to take up learning the guitar when I was at college. He had been playing the guitar for about 3 years and was very cool. He showed me guitar tab one day and I bought a guitar the next month. Within 2 months I was a better guitarist than him. I ended up playing in 2 bands, and working briefly as a session guitarist, as well as teaching probably 100 people to play. AFAIK he never played a gig.

The difference between us was that I wanted to play the guitar and learn how to play well, whereas he wanted to be a guitar player. This was probably related to a desire to be cool, or get women to sleep with him, or both.

My advice is to know which you are. Are you an indie developer because you love the indie scene, and the people? Or are you an indie developer because you want to make games, ideally full-time? I also suggest that if its the latter, you need to lock down and optimize WHERE you do it.

Briefly, when I quit my job to go indie the first time, and my wife was at work full-time, I experimented with coding in coffee shops, because thats what they show people doing on TV and in magazines. It was crap. They are full of mothers with screaming kids, expensive (but average) coffee, no stability, no room, no peace, an environment 100% NOT conducive to C++.

this is not work

You MAY be one of the 1% of people who can program and design games and do real serious *deep work* while in a noisy environment you do not 100% control and surrounded by other people who often interrupt you. You really may be… but you probably are not. Almost everyone can concentrate better when things are quiet. Programming especially requires *deep* concentration, that is easily shattered and hard to rebuild.

In short, unless your environment is quiet, free from clutter, dedicated to one thing (work) and set up to convey that this is a WORK location, not a chill-out zone, then you are not going to get much done.

My tips?

Get a dedicated room in the house/flat if you can. If you cannot, then set dedicated work times, when you are not to be disturbed for any reason except a literal burning building…

Get an office chair, I recommend an aeron but cheaper alternatives are available. Set it up to be perfect for typing and reading, not slouching. You are *not* going to work all day on a beanbag or a sofa. You just are not.

Only people insecure about creativity think beanbags will save them

If its not quiet enough, get noise cancelling headphones and wear them.

Do not fill your desk and office with lots of fun toys and other distractions. Yes, you are making a fun game, but 95% of your time is work and implementation. Don’t confuse your subconscious. Are all those desk toys REALLY making you more creative or just distracting you from actual work.

Most of us are pretty shallow. We really care about what other people think of us, and when we are young, especially if we are single, we obsess about seeming cool. Work is not cool, work is for serious grown-ups who are boring. Thus we spend a lot of time trying to look cool, rather than be effective. If you saw me sat here right now, unless you noticed the framed prints of past games on my office walls, you would assume I’m working in fintech or IT. Its a very un-gamey environment, and it keeps me focused.

When you have shipped game #10 and sold game copy 1,000,000, feel free to fuck around. I *do* indeed have a child-size Tesla model S propped up in my office, and a toy car with toy robots on my desk, and am happy to be interrupted by cats and visitors all during the day. I can afford to be slack, but it was not always the case.

Set your environment up to be worklike, and you *will* get more done. We are simple animals and highly influenced by our surroundings. Stop trying to be cool.

Cliffs guide to optimisation tools for C++ games in 2019

Hey. I’m a games programmer who has been coding for 38 years (yes really) and making indie games for twenty. Thus… I’ve done a lot of it, and learned from a lot of mistakes. I still do not consider myself an expert (who does?) but I’ve stuck primarily with one language (C++) and have used a lot of optimization tools over the years, so I’m simply passing on what I know about making your game run faster in the year 2019.

Disclaimer: if you are making games in unity then… you are on your own. I have no idea what tools are available in unity (although talking to friends makes me suspect they are pretty…lacking). Frankly if you are relying on someone else’s engine code that you can’t change then you are probably fairly screwed anyway, performance wise. For everyone else…read on :D

First of all, its worth getting some perspective. The first modern games I tried to write were on the intel 386dx2 processor. The speed of my current chip (intel i7 6700 @ 3.40 GHZ) compared to that is…vast. I can’t even find stats for such an old chip, so zapping forwards a lot I guess I can compare my current chip to an intel Pentium 4 1500…

So frankly my current chip is about 80 times faster than one from 2009. And lets not kid ourselves that 2009 was the stone age. 2009 games looked PRETTY GOOD. So in the year 2019 (let alone the 2015 of my current chip), everything should be running silky smooth at 60 FPS (minimum) and with a UI that is as responsive as lightning right?

We all know this is not the case. Games still have performance problems. My point is that we should only be seeing performance problems in extreme cases now, when we are really pushing a machine to its limits to process incredible amounts of data, or render insane amounts of pixels, or applying insane effects to them. As an indie game developer, you are unlikely to be making Battlefield V style graphics, so your performance problems should be easy to solve right?

The biggest problem is that many developers just have NO IDEA what options there are out there to work out WHERE your performance problems are. I intend to show you some of them. (Click to enlarge any screens.)

Option #1 The visual studio profiler.

Obviously this is built in, so free with the IDE. I use a slightly older version of Visual C++ to the current one, and the profiler seems…ok, but a little basic. I get the impression its similar to the unity tools. It seems to want to tell you what to do, identifying specific functions that *it thinks* are the problem, rather than just giving you data and letting you investigate. Its very limited when it comes to actual visualization.

(default after-run screen of the profiler. Not much help tbh)

Now sure… this sort of thing is 100x more useful than no profiling at all (or horror of horrors: trying to measure your own code using hacked timers…), but frankly its pretty badly put together compared to dedicated 3rd party tools. Still…its built in I guess? I hardly bother using it.

(one of many uninspiring views in the visual c++ profiler)

Option#2 The visual studio concurrency visualizer.

In my version this is an ‘optional add-in’ which frankly is essential if you do multithreaded code (and if you don’t…omg why?) Its one of the best tools imaginable for visualizing where one thread is blocking another in a specific frame. It has truly excellent support for you adding in your own named sections, markers and events that let you build up really detailed and helpful pictures of why one thread is waiting for another. Given the ‘free’ price, its highly recommended, and probably the simplest tool for visualizing how your game is handling multi-threading.

It also has a baffling but I guess kinda cool view where you can see the relationship between actual threads and physical cores which shows them bouncing all over the place. Frankly I think this tool is made redundant by vtune, but like I say, its free and integrates nicely into visual studio. If you place custom code markers into it, it gets very helpful indeed.

Option#3 NVidia Nsight.

A visual debugging system thats free for nvidia card owners. This is great for GPU debugging, as it lets you freeze your game in real time, and then generate a ‘frame-scrubber’ view, where you can step through each individual draw call in a specific frame and watch every mesh get rendered, one at a time, and see the textures that were used, and what was changed on screen. This is *great fun*. if you ever have a bug where something is being drawn in the wrong order, this is how you spot it. This also allows you to spot those cases where you do far too many draw calls that could be easily batched. I’m doing too many draw calls here:

(nsight scrubber half way through a frame render showing current draw call texture atlas)

Nsight also has a system to let it run on a remote PC (which is cool), and has some nice little visual tools like the ability to visualize overdraw or to set all textures blank to see if its texture memory that is acting as a frame rate blocker. A real-time histogram shows you how many polys are being rendered by each draw call, which lets you spot parts of your games where you need more batching. Its very useful for games that are graphically complex, and where you need to visualize which items are taking up too many draw calls.

(nsight real time view showing GPU busy chart and draw call histogram)

Option #4 AQTime (by SmartBear) (approx $650)

This is commercial profiling software that I’ve used for years. it does an extremely good job of working out exactly what line of what function is slowing you down, which it does over the entire lifecycle of running the program. You can programatically (or manually) turn data collection on or off if you only want to capture a specific part of the run, but you have to do that at run time, not after the event, which is less than ideal if hunting for a specific slow frame.

AQtime has exceptionally good stability, and is very good at visually allowing you to ‘drill down’ into code and see where a bottleneck is. Its absolutely rubbish at concurrency analysis (although you can view the results of each thread individually, you cannot detect blocking). It gives some really cool charts, and will even let you switch to see the source code alongside profiling data, and even the assembly code if you really want to.

I’ve found that owning a copy of this is annoying (it has some serious low level service-based DRM), and its very much aimed at trying to get you to buy a new update every year, but if you can ignore that and deal with the company, and the price, its actually very good. I’ve used it for hundreds and hundreds of hours. Running aqtime WILL slow down your code a lot (especially in line-analysis mode) but its worth it to get the data. Its also quite good at analyzing memory usage.

Option#5 Intel VTune Amplifier.

Ok this one has a free trial that seems to never end so…I dunno. I don’t understand either. In the past I’ve paid the full £800 or so for this, and it was worth it.This is a full-on serious profiling tool that integrates into visual studio but can also be launched externally. Its a very low level beast, and produces gigabytes of profiling data. By default it limits its capture to 2GB of data, and thats *not a lot*. You will not leave it running for hours. Like many profilers, it has multiple modes and methods and rules. I tend to use its concurrency analysis mode which provides incredibly complex data like this:

Frankly the UI for it is as stable as a canoe carrying heavy industrial equipment, but its numerous crashes aside, its a seriously amazing bit of software. Treat it gently and it rewards you with incredible details, and also makes you strongly aware of how your code is just part of the story, with directx, drivers, and the O/S also having to do quite a bit of stuff while your game runs.

Like the concurrency visualizer, you can place your own markers in the code to label each bit of data and you can stack them a seemingly endless amount of times to give you amazing drill-down. For multi-threaded apps, this is a goldmine of information that just is *not* there with a normal, simple profiler.

I also find that its very good at letting you view the bottlenecks in code very well when you only want to select a single, very specific frame where stuff goes wrong. Honestly you could probably spend years just looking at the data of a single run of your game and still be learning new stuff.

Conclusion

Learning to use these tools takes months, and learning to apply the knowledge from them takes years. I’m only part way into the lifelong process of understanding what makes code slow, and how to fix that, but the important thing I want to get across is you need to know how to MEASURE this stuff.

Just running a function a thousand times with timegettime() at the start and end tells you NOTHING, other than that you really need to get a profiler. Increasingly code is running on a machine with multiple cores sat idle and without a really good way to analyze and visualize the inter-relationship between code running on different cores, you are basically trying to optimize with one hand tied behind your back and a patch over one eye.

There are some great free profilers, and even the commercial ones will pay for themselves. Unless your game already runs perfectly at 60FPS even on 5 year old hardware, you absolutely need to learn how to use one.

(if you found this post helpful, check out my games)

Long numbers and dumb coding in Production Line

I have a few reports of a bug in production line where people have got VERY good at the game and amassed tens of billions of dollars. Nothing actually crashes, but financial reports start filling up with gibberish like $-1,-4,-2,-0,-1,-3,-5 and so on. The cause is simple, the fix difficult(ish) and the wider implications worthy of comment.

Firstly, as most coders can already guess, its because I’ve gone outside the max limits on the data type I’m using. I use a ‘float’ for financial figures in the game (mainly because some items get percentage reductions due to global events and competition, so I need decimals) and because *somewhere* I must be casting it to an int (no idea where yet) the numbers I can represent are limited to -2,147,483,648 to 2,147,483,647 because an int is 4 bytes. If I was using only a float, things would be super different. More details here.

Anyway, although generally speaking numbers like that are big enough, in a financial sim, restricting any possible number to a max value of $2billion is not ideal. My game is not ‘realistic’ in costs, and its rare that players finances will exceed $100 million, let alone 2 billion, but obviously it happens now and then and the game kinda screws up.

The fix is easy. I just need to replace that datatype, but the actual implementation is a bit more messy, because that means old save games need converting or handling as I load those numbers in (right now the game only knows how to load ints, floats and strings and likely casts to ints somewhere). I also have to go through every place where those numbers get locally stored by any GUI code and ensure I’m not using ints there either, and of course I need to change my display code so it handles floats properly and converts them to strings properly.

TBH its likely only a days work, but assuming another day to test it, and then a few days of ‘unstable build’ roll-out to the hardcore before folding it into the main code branch, all this means that this will be a fix coming after release on Thursday, and NOT today :D

Perhaps the more interesting topic is how do you deal with edge cases and marginal cases with a sim/strategy game. I know from experience that people WILL push these games to the extreme and WILL break them. I’ve had negative reviews from people with >100 hours playtime because their extreme playtime has led to discovering exploits, and edge cases that allow them to ‘break’ the game.

Anyway, this is a real issue because making a balanced, playable, fun, reliable game that works for 99% of the playerbase takes 99% of the effort, and making it work perfectly in EVERY edge case is…another 99% of effort. Generally speaking, its way better to concentrate on fixes, features and changes which make the game better for the 99% than to fixate on the extreme edge cases, especially as a time-limited indie who can never do everything.

I’ll fix the numbers thing post-release, but I have to admit that are likely some other real edge cases that I never will. This is also true of any of my games, like Democracy 3, or Gratuitous Space Battles. There are probably obscure edge cases and tactics that you can discover in all my games that ‘break’ them, after 100+ hours of gameplay. Thats not shocking. I’m always going to make the majority of players who play in the ‘usual’ way my primary focus. Edge case fixes are something I like to do, and want to do, but you can never get them all, or fix them all. You will go mad trying to do so.

Do not forget that there are companies like EA/DICE where thousands of players will scream that ‘X’ is unbalanced or ‘Y’ breaks the game and has not been fixed!, and yet presumably those companies have 50x the manpower I do. The likelihood is that they have data that shows this is only true in 0.1% of cases, and their dev time is better spent elsewhere. I have no illusions that players will accept that as an answer, but that will not stop it being the case!

Technical debt… or investment?

Coders talk a lot about technical debt. here is a rough definition:

“Technical debt” (also known as design debt or code debt) is a concept in software development that reflects the implied cost of additional rework caused by choosing an easy solution now instead of using a better approach that would take longer.”

It’s often used in the context of justifying redoing some work. As in… “we have a lot of technical debt. The best thing to do is rewrite all this properly from scratch”. To some extent that can be justified. The engine for your games is effectively your house foundations. Don’t start building a house without ensuring you have decent foundations, and all that sort of thing. The problem with this mindset is that people (or rather…computer programmers) often get into the situation where they want to re-code everything from scratch again, and again, and again.

The thing is, the more experienced you get, the more you realize that the big, messy, patched, complicated looking smorgasbord of code that you are working with probably HAS to look like that. it probably looks like that for a reason. Its not pretty, its not clean, it doesn’t make for nice diagrams but it WORKS and thats important.

If you think thats not true I point you towards the apache web server (literally a pun on ‘a patchy web server’) and Microsoft windows, an absolute towering pile of code mess that is the very definition of technical debt. How can I possibly view these as good things…?

The simple answer is: Practicality and commerce. Apache is not the most beuatiful work of art in terms of highly structured, elegant, perfectly designed code, and windows is literally the state of the art in horrible hacks. (Like this one, to ensure sim city still runs). To quote:

They reported this to the Windows developers, who disassembled SimCity, stepped through it in a debugger, found the bug, and added special code that checked if SimCity was running, and if it did, ran the memory allocator in a special mode in which you could still use memory after freeing it.

And it you are someone who considers code to be beautiful, who likes to describe themselves as ‘a software architect’ instead of ‘hacker’ (or code monkey), then stories like this will fill you with rage but…

…Apache sure is installed on a lot of servers, and a huge number of PCs still run windows. Why? Because accepting that your patchy, confusing, held-together-with-string piece of code is actually MORE reliable than new stuff you could code today is actually a sign of coding maturity.

I do not open source my engine, and TBH nobody would likely use it if they did, but part of my reasoning is that it would be embarrassing. I couldn’t even decide on a naming convention. At one point I didn’t care, and I had classes called things like IniLoader. Then I thought it would be cool to have a ‘G’ prefix to indicate game engine, so I have GArc GFile, GHashTable. At some crazy point I had my classes in all caps, hence DEBUGENGINE.h. Some of the code separates functions with /************/ some of it with //////////////////////////////. Pretty much the only coding standard was a member variable naming convention with FirstLetterCapped.

By far the two biggest embarrassing pieces of dodgy code in the ‘more or less part of the engine’ GUI code are the following. Please try not to laugh.

Stupidity one: I have a GUI_ButtonBase class, and a GUI_WindowBase
class . They are DIFFERENT things, with no connection, despite a button obviously being a derived class of a window in any sensible system. Thus my GUI_WindowBase class contains separate lists of buttons, and child windows. *sigh*.

Stupidity two: All my windows have a virtual function call CheckClick(int x,int y), which they process, then call on any children. You would think that this took the x,y of the current mouse position and operated on it, but for some reason I gave up bothering and use helper functions like IsMouseInside() to check such stuff, which ignores the passed-in x and y. I still go to all the bother of passing x,y, down the hierarchy, but its ignored. I’m just paranoid about it.

Sure I could fix this, I could rewrite my button code, junk that dumb x,y, thing and rename all my classes sensibly while I’m about it. I would then need to spend several days changing all the code in Production Line, AND checking it, and I *ASSURE* you, I would miss something. There would be a new crash bug. I wouldn’t spot it, and maybe after a dozen players encountered it, someone would tell me about it. I would have achieved nothing, but frustration for my players, and a smug feeling that my code was less embarrassing.

I’m not giving into that. My current code WORKS, it has less bugs than it *ever* has. My current game (Production Line) is more stable than ever, and more stable than any game I have ever shipped. Going back and changing parts of my engine because they look messy, or because modern code fashions have changed or because there is a new API or code pattern would be MADNESS.

Don’t be too hard on your old code. Sometimes age brings wisdom.