Game Design, Programming and running a one-man games business…

Cliffs guide to optimisation tools for C++ games in 2019

Hey. I’m a games programmer who has been coding for 38 years (yes really) and making indie games for twenty. Thus… I’ve done a lot of it, and learned from a lot of mistakes. I still do not consider myself an expert (who does?) but I’ve stuck primarily with one language (C++) and have used a lot of optimization tools over the years, so I’m simply passing on what I know about making your game run faster in the year 2019.

Disclaimer: if you are making games in unity then… you are on your own. I have no idea what tools are available in unity (although talking to friends makes me suspect they are pretty…lacking). Frankly if you are relying on someone else’s engine code that you can’t change then you are probably fairly screwed anyway, performance wise. For everyone else…read on :D

First of all, its worth getting some perspective. The first modern games I tried to write were on the intel 386dx2 processor. The speed of my current chip (intel i7 6700 @ 3.40 GHZ) compared to that is…vast. I can’t even find stats for such an old chip, so zapping forwards a lot I guess I can compare my current chip to an intel Pentium 4 1500…

So frankly my current chip is about 80 times faster than one from 2009. And lets not kid ourselves that 2009 was the stone age. 2009 games looked PRETTY GOOD. So in the year 2019 (let alone the 2015 of my current chip), everything should be running silky smooth at 60 FPS (minimum) and with a UI that is as responsive as lightning right?

We all know this is not the case. Games still have performance problems. My point is that we should only be seeing performance problems in extreme cases now, when we are really pushing a machine to its limits to process incredible amounts of data, or render insane amounts of pixels, or applying insane effects to them. As an indie game developer, you are unlikely to be making Battlefield V style graphics, so your performance problems should be easy to solve right?

The biggest problem is that many developers just have NO IDEA what options there are out there to work out WHERE your performance problems are. I intend to show you some of them. (Click to enlarge any screens.)

Option #1 The visual studio profiler.

Obviously this is built in, so free with the IDE. I use a slightly older version of Visual C++ to the current one, and the profiler seems…ok, but a little basic. I get the impression its similar to the unity tools. It seems to want to tell you what to do, identifying specific functions that *it thinks* are the problem, rather than just giving you data and letting you investigate. Its very limited when it comes to actual visualization.

(default after-run screen of the profiler. Not much help tbh)

Now sure… this sort of thing is 100x more useful than no profiling at all (or horror of horrors: trying to measure your own code using hacked timers…), but frankly its pretty badly put together compared to dedicated 3rd party tools. Still…its built in I guess? I hardly bother using it.

(one of many uninspiring views in the visual c++ profiler)

Option#2 The visual studio concurrency visualizer.

In my version this is an ‘optional add-in’ which frankly is essential if you do multithreaded code (and if you don’t…omg why?) Its one of the best tools imaginable for visualizing where one thread is blocking another in a specific frame. It has truly excellent support for you adding in your own named sections, markers and events that let you build up really detailed and helpful pictures of why one thread is waiting for another. Given the ‘free’ price, its highly recommended, and probably the simplest tool for visualizing how your game is handling multi-threading.

It also has a baffling but I guess kinda cool view where you can see the relationship between actual threads and physical cores which shows them bouncing all over the place. Frankly I think this tool is made redundant by vtune, but like I say, its free and integrates nicely into visual studio. If you place custom code markers into it, it gets very helpful indeed.

Option#3 NVidia Nsight.

A visual debugging system thats free for nvidia card owners. This is great for GPU debugging, as it lets you freeze your game in real time, and then generate a ‘frame-scrubber’ view, where you can step through each individual draw call in a specific frame and watch every mesh get rendered, one at a time, and see the textures that were used, and what was changed on screen. This is *great fun*. if you ever have a bug where something is being drawn in the wrong order, this is how you spot it. This also allows you to spot those cases where you do far too many draw calls that could be easily batched. I’m doing too many draw calls here:

(nsight scrubber half way through a frame render showing current draw call texture atlas)

Nsight also has a system to let it run on a remote PC (which is cool), and has some nice little visual tools like the ability to visualize overdraw or to set all textures blank to see if its texture memory that is acting as a frame rate blocker. A real-time histogram shows you how many polys are being rendered by each draw call, which lets you spot parts of your games where you need more batching. Its very useful for games that are graphically complex, and where you need to visualize which items are taking up too many draw calls.

(nsight real time view showing GPU busy chart and draw call histogram)

Option #4 AQTime (by SmartBear) (approx $650)

This is commercial profiling software that I’ve used for years. it does an extremely good job of working out exactly what line of what function is slowing you down, which it does over the entire lifecycle of running the program. You can programatically (or manually) turn data collection on or off if you only want to capture a specific part of the run, but you have to do that at run time, not after the event, which is less than ideal if hunting for a specific slow frame.

AQtime has exceptionally good stability, and is very good at visually allowing you to ‘drill down’ into code and see where a bottleneck is. Its absolutely rubbish at concurrency analysis (although you can view the results of each thread individually, you cannot detect blocking). It gives some really cool charts, and will even let you switch to see the source code alongside profiling data, and even the assembly code if you really want to.

I’ve found that owning a copy of this is annoying (it has some serious low level service-based DRM), and its very much aimed at trying to get you to buy a new update every year, but if you can ignore that and deal with the company, and the price, its actually very good. I’ve used it for hundreds and hundreds of hours. Running aqtime WILL slow down your code a lot (especially in line-analysis mode) but its worth it to get the data. Its also quite good at analyzing memory usage.

Option#5 Intel VTune Amplifier.

Ok this one has a free trial that seems to never end so…I dunno. I don’t understand either. In the past I’ve paid the full £800 or so for this, and it was worth it.This is a full-on serious profiling tool that integrates into visual studio but can also be launched externally. Its a very low level beast, and produces gigabytes of profiling data. By default it limits its capture to 2GB of data, and thats *not a lot*. You will not leave it running for hours. Like many profilers, it has multiple modes and methods and rules. I tend to use its concurrency analysis mode which provides incredibly complex data like this:

Frankly the UI for it is as stable as a canoe carrying heavy industrial equipment, but its numerous crashes aside, its a seriously amazing bit of software. Treat it gently and it rewards you with incredible details, and also makes you strongly aware of how your code is just part of the story, with directx, drivers, and the O/S also having to do quite a bit of stuff while your game runs.

Like the concurrency visualizer, you can place your own markers in the code to label each bit of data and you can stack them a seemingly endless amount of times to give you amazing drill-down. For multi-threaded apps, this is a goldmine of information that just is *not* there with a normal, simple profiler.

I also find that its very good at letting you view the bottlenecks in code very well when you only want to select a single, very specific frame where stuff goes wrong. Honestly you could probably spend years just looking at the data of a single run of your game and still be learning new stuff.

Conclusion

Learning to use these tools takes months, and learning to apply the knowledge from them takes years. I’m only part way into the lifelong process of understanding what makes code slow, and how to fix that, but the important thing I want to get across is you need to know how to MEASURE this stuff.

Just running a function a thousand times with timegettime() at the start and end tells you NOTHING, other than that you really need to get a profiler. Increasingly code is running on a machine with multiple cores sat idle and without a really good way to analyze and visualize the inter-relationship between code running on different cores, you are basically trying to optimize with one hand tied behind your back and a patch over one eye.

There are some great free profilers, and even the commercial ones will pay for themselves. Unless your game already runs perfectly at 60FPS even on 5 year old hardware, you absolutely need to learn how to use one.

(if you found this post helpful, check out my games)

Positech school #2 Cameroon update

We just got sent new pics of the start of foundations for the second school we are building in Cameroon. the site is actually different to the one that was originally started, because all sorts of internal problems in the country meant that our funds had to be redirected to a different part of the country. (There is some political unrest in Cameroon).

The location is described as:

“Ecole Catholique St Barthelemy de Djeugo is located in Bahouan village, in Bamendjou Subdivision, Haut Plateau Division of the West Region of Cameroon. It is approximately 3km from the Centre of Bahouan on un-tarred but motor able roads.”

Which looks like its about here:

Anyway, the good news is that things are back on track, and we have pics to prove it :D

Why I skipped GDC in 2019

So… the interwebs are awash with the happy high-fiving post-GDC backslapping and cheering and ‘omg this was what my GDC experience was like!’, so in typical cynical British fashion I thought I would put fingertips to keys for my alternative take.

Some background: I’ve been an indie dev for 20 years and coding for 38 years (not a typo). I’ve released over a dozen games, including Kudos, Gratuitous Space Battles, Democracy and production Line. I’ve also published some games. I work full time doing this as my job, and live in rural UK. I’m 49 (bloody hell!) and married.

I went a LONG time as an indie before finally giving in and going to GDC a few years ago. I remember my first experience being one of nervousness at not knowing ANYBODY, so I followed Jake Birkett of Grey Alien games, (who I’ve known for years) like a little lost and bewildered puppy hoping nobody would notice I was a total n00b to it all. The next year I went back, and the next year etc. This year was the first time I missed it since I started going.

As an alternative to GDC this year I did pop along to a way smaller London event, and spoke a bit about Unity and making your own engine, and met up with some UK indies I’ve known a long time to chat about stuff, which I’m really glad I did. Its a 2-3 hour trip from my house to London, so I stayed over in a local hotel. Drink was drunk, Lamb was eaten, jokes were made.

My decision to avoid GDC this year was very deliberate, but not a complete rejection of the whole concept. I may well return there next year, but my motivations for doing so will be almost entirely opposite to the motivations of people who attend their first GDC, so I thought it worth talking through how I feel about it. In that spirit, lets start by being negative (hey…British!) by listing what people do NOT tell you about GDC.

Negative Item #1: San Francisco. I actually got married in the US, and we stayed briefly in SF on our honeymoon. It was fab. The golden gate bridge! the trolleybusses! the huge pancakes! it was lovely. A great tourist town. Really cool. Almost 20 years later and… Oh…my…god. I’m not sure whats worse, the fact that there are *so many* homeless people or the fact that local residents have got used to totally blanking them. I’ve occasionally given cash to them, sometimes when I’m in a good mood, a fair bit of cash. One guy shouted ‘are you serious man?’ at me once. I guess they are used to being ignored, an inconvenience. an embarrassment. As a visitor, its totally shocking. And many of them seem to have untreated mental health issues.

I know all big cities have a problem with homelessness, but having just got back from London I know its TEN times worse, (at least to the casual visitor) in SF. Bad as the homelessness is, its not the only problem. There are parts of SF that you are very clearly warned DO NOT GO THERE. The really scary bit? they are maybe *one* street away from five star hotels. Its like some dystopian sci-fi future.

In a very small way, I’m not going this year to protest that San Fran will not deal with its problems. This is not a poor city. And frankly, I don’t go abroad much. I don’t want to spend half my time worrying about being stabbed, or getting depressed about homelessness. Other really nice US cities are available. Also, WTF is wrong with Las Vegas? come on guys… Vegas!

Negative item #2: Money. Luckily, the gods of market forces have been good to me (also I work like a fucking maniac and have 20 years indie experience and no kids), so my company does very well. I admit it, I fly business class when I visit the show. I can afford to buy a complete GDC show pass if I wanted to ( I do not). The cost of that pass?

$2,499

This is for a conference pass. Not a new laptop, or a new laptop + apple iphone X plus 3 course meal. Its just a pass that lets you actually go to everything at the show. They have to be absolutely kidding right? By the way, that just gets you into the show. Your food & drinks are extra, your hotel extra, your flights extra, your transfers from hotel, extra. Is this serious? and that brings me onto my next item:

Negative item#3: AFAIK hardly anyone is getting paid. The speakers? they get a free pass (OMG at $2,499 value!!!), but fuck-all else. You think they get free flights too? free hotel rooms? nope. Nothing, at least not the last time I checked. The CONTENT at the show (the talks) are provided by volunteers. At least everyone else got paid. But no, most of the ‘helpers’ at the show are volunteers too, they aren’t getting paid either.

People want to start a discussion of unionization in game dev? FFS lets start here. You give a talk at a show where the tickets are two thousand dollars, you need to get PAID for your time. I’ve spoken at GDC twice (one indie rant, one talk with 3 or 4 other devs). I’m never doing it again. My time IS money, especially if I have to spend my own money to get there and back. FWIW, other shows sometimes DO pay, and sometimes even for flights & hotels.

The thing is, you may consider all this to be worth it if the actual content of the show really improves your business right? I totally agree with you but that means item 4:

Negative item#4: The overwhelming majority of the talks are of zero use to you. This is not a dig at the speakers, many of whom are excellent and put a lot of (unpaid) work into them (including the work done by everyone who ‘submits’ a talk, but gets rejected). The fact is… game dev is a vast topic and a lot of platforms, genres and technologies are in play. The vast majority of it is NOT helpful to you. For example, are you a mobile dev using java and opengl to make an MMO? Awesome, but me giving a talk on optimising C++ and directx for PC strategy games is probably fucking useless for you.

Unless you are bizdev + marketing + finance + coder (all languages) + artist + animator + designer… the overwhelming majority of talks are not in your area. And guess what… multiple talks happen at once, so the chances are you will miss some of the ones you wanted to go to anyway. And oh… sometimes there isn’t enough room, so you will not get to attend a talk anyway.

Of course that applies to any big show, but that doesn’t mean its not a factor. Also applicable to every other big show is…

Negative item#5: You may well get ill. Or suffer in other ways. I used to laugh at people who used hand sanitizer and fistbumping. What feeble office-jockeys are they? whats the worst that can happen? Then it happened to me. I was ill after the show. very ill. Horribly ill. Embarrassingly ill. Get me drunk and ask for details one day. Its actually quite funny, but at the time: No. It was BAD. You are shaking hands with hot sweaty geeks from all over the planet. You *will* get ill at some point, and lose productivity.

Also, even if you don’t get ill, if you are a shy introvert coder like me, GDC is NOT DESIGNED FOR YOU. There are a lot of very confident, loud, assertive, extroverted, friendly, upbeat Americans who will talk extremely loudly in very loud bars absolutely packed with people who all seem to already know each other. You think you will enjoy trying to close a publishing deal over cocktails in a loud dark nightclub where people are yelling across you? You will not enjoy that. Nobody does, and yet its the same, every year.

So given all this… why the fuck did I ever go back?

The Good Stuff. GDC, like any big game show allows you to meet probably at least a hundred people who *do what you do*, and are as keen to talk about it as you are. You will hear a lot of business insider stuff. You will be exposed to lots of ideas, and get insight into the way the industry is heading. We are all still apes in T shirts, so physically meeting and sharing coffee/beer with someone means you are WAY more likely to work with them in the future. Networking is a REAL thing, and GDC is the heart of games industry networking. Despite everything I’ve typed here, you should all go to it once (if you can afford it).

And I admit, that even as I type this extended long-form rant, I do regret the fact that so many people I’ve met before at GDC, like Ron, Tommy, Brad, Chet, Ichiro, John & so many others… I’m just not going to see this year and thats kinda sad. TBH, I go there more for the meals and the drinks and the banter and jokes than any actual *biz* need, but I miss all you people, and wish we could hang out.

Oh and BTW I missed out my ‘kinda preachy’ (*but if you know me well, you will know its my primary reason*) argument for NOT going to GDC. I’m an environmentalist, and if you are too, you either need to not keep jumping on long distance airplanes that pump out serious CO2, or you need to offset the fuck out of it when you do go. I *did* consider flying & offsetting but decided against it.

Anyway… hope this isn’t too depressing. Its my honest take. Be aware of survivor bias, and peoples desire to always appear happy on social media. People are not going to tweet ‘Went to GDC, was expensive and crowded and probably a waste of time’. Nobody does that, but some people likely did think it. Of course, YMMV :D.

Healthy game launch. Yay!

So you may have missed it, but we launched Production Line out of Early Access almost exactly a week ago. It looks like we have had a pretty good launch. We were in various charts in various categories, sold a lot of copies, got some good word of mouth coverage, and a fairly minimal amount of bug reports. I have remained relatively calm, and relatively sane, and am still motivated to improve the game and continue to do (some) work on it for the next few months. Woohoo.

TBH this is the smoothest game launch is Positech’s history. This is the first time I’ve done early access, so its the first time I’ve had literally tens of thousands of people hammering the game code *before* I officially declare the game *done*. Frankly, these days most indie games get a way bigger EA launch than a final product launch, so its not as gentle a ramp up as it sounds, but it still resulted in a pretty bug free departure from Early Access

Something I was very happy with was that I could set the ‘final’ build live for all the EA players, not touch the game *at all* for a few days, and when I was absolutely sure everything was fine, just literally click the ‘release out of early access’ button knowing that things were pretty stable. I highly recommend this!

So far we have had ONE patch since release, which fixed a short list of things, and there is another one on the way in maybe a week or so, which will be the accumulation of a bunch of bug fixes (even some pretty rare crashes) and some UI features people have asked for like a camera speed slider, autosave interval slider, and some extra stats. It feel so good to be in a position where people are saying ‘the game needs a better UI for feature X’ instead of ‘the game needs feature X’.

I definitely have plans to do some paid content for the game (DLC) alongside regular updates. I’m obviously not in a position to even tease anything like that yet, but as I prefer content-heavy DLC to code-heavy DLC (its just optimal given that I am the only coder), such things do not take *much* time to do (although there is of course a big art budget cost).

Right now I’m pretty happy with how Production Line has gone. Even if the game makes NO MORE MONEY AT ALL, I’ll be happy (but amazed!), and I think over the long tail of the next 2+ years there is a good chance of it making at least 50% of its current earnings again, which I’d be very happy with. I’m currently minded to *not* go mad with sales and discounts, as I think this is getting a bit out of control and games are being devalued, but I may think differently about that in a years time.

Anyway, this is nothing but good news, which is a change from the usual indie ‘I sold no games and have eaten my pets’ stories, but I’m not going to pretend things went wrong when they didn’t. I always blog openly about my screw-ups (2 recent games still in the RED for me :(), so I may as well be honest when things go well.

Thanks to everyone who bought the game so far!

Production Line leaves Early Access tomorrow! (new trailer)

My super-complex spreadsheet which I use to track the spending and income for my latest game (Production Line) informs me that I have been working on it for 3.12 years, and have spent a (very roughly estimated) total of 9,115 dev hours on it. The game is leaving early access tomorrow!

In truth I have probably spent a lot more hours than that, as I tend to overwork, and spend a lot of time in evenings checking forums and reddit/facebook etc to reply to people, but anyway you look at it, 3.12 years seems to be quite a long time to work alone as the only coder and designer on a game.

Bizarrely, I am still very much enjoying the games development, and have a list of extra things I would like to tweak and improve after release. In many ways the decision that the game is ‘released’ is a purely arbitrary one. In marketing terms it encourages people who dislike the state of most EA games to try the game out, and it also signals a potential slowing down in the addition of new features.

I still have a lot of ideas for stuff that could be added to the game, and I suspect we will have some paid DLC once the dust has settled. I wont rehash my pro-DLC arguments here, but I’m in believer in it both as a developer and a gamer. Why will DICE not sell me panther tank DLC for Battlefield V? TAKE MY MONEY. (also new hats please!)

Speaking from a personal point of view.. I am TIRED. I’ve felt like it for a while, and I think I do need a brief period where I scale down my work slightly. We are currently working on Democracy 4 (I’m not coding on it), and helping to manage that is an imminent concern. I currently have NO PLANS for any other game after Production Line (as a coder), so I’m pretty free (assuming it sells ok) to relax for a bit.

While I relax (Maybe for an HOUR!), here is the new launch trailer for Production Line. I really enjoyed getting this made. I hope you like it :D

Oh and if you have suddenly decided to buy the game, you get a steam key and we get 95% of the money if you buy using the widget below!