Game Design, Programming and running a one-man games business…

Category: programming

Getting threads to rapidly schedule tasks in C++, Windows

very techy post…

My latest game uses the same multi-threading code I’ve used before. This is C++ code under windows, on windows 10, using Visual Studio 2013 update 5. My multithreading code basically does this:

Create 8 threads (if an 8 core system). Create 8 events which are tracked by the 8 threads. At the start of the app all of the threads are running, all of the events are unset. As I get to a point where I wwant to doi some multithreaded code, i add tasks to my thread manager, and when I add them, I see which of my threads does not have a task scheduled, and I set that event:

PCurrentTasks[t] = pitem;
SetEvent(StartEvent[t]);

Thats done inside a CriticalSection, which I initialized with a spincount of 0 (although a 2,000 spiun count makes no difference).

Each of the threads has a threadprocess thats essentially this:

WaitForSingleObject(SIM_GetThreadManager()->StartEvent[thread], INFINITE);

When I get past that, I enter the critical section, grab the task assigned to me, do the actual thread work, then I immediately check to see if there is another task (again, this happens in a critical section). if so…I process that task, if not, I’m back in my loop of WaitForSingleObject().

Everything works just fine… BUT!

If I have a big queue of things to do, my concurrency charts from vtune show stuff like this:

Seriously, WTF is happening there? Those gaps are huge, according to the scale they are maybe 0.2ms long. What the hell is taking so long? Is there some fundamental problem with how I am multithreading using events? Does SetEvent() take a billion years to actually process? If feels like my code should be way more bunched up than it is.

Major code speedup. Big factories ahoy!

I recently got sent a HUGE factory in a savegame for Production Line. It has over 600 production slots and 24 resource importers, and its MASSIVE. Here are some screenshots:

Despite its awesomeness it was MASSIVELY slow. Not only did loading it take forever, but whenever you changed any of the resource conveyors the game would hang for about 20 seconds. hardly ideal, especially given that I have an 8core i7 PC. So I set to work trying to optimize the code. I did some fairly small optimisations first, which boosted processing by 12% but I needed more fundamental re-design.  After chatting to a fellow indie coder, I agreed that my currents system of always calculating the optimum route to everywhere, for everyone, when something changed, was clearly not viable. I switched over to a new system of ‘lazy’ computation. Basically when I change a route somewhere, it now sets a flag on every production slot saying ‘you should recalculate your nearest slot soon’. That flag gets checked every frame, and sometime in the next 120 frames 92 seconds) each slot will calculate the route from its location to all the importers, and store the nearest two. It needs this so slots seem to import from a sensible location, as opposed to an importer that is miles away.

This was good, and sped things up a LOT. Now instead of hanging, the game would stutter for about 10 seconds. better but nowhere near good enough. I then realised I was doing something seriously dumb. I was going through all those routes I calculated, and picking the nearest, THEN doing it again, to pick the second nearest. doh! This sped things up, but in retrospect, it was trivial, it just alerted me (alongside my profiler) to the fact that when I tried to get (for example0 15,000 routes, I actually calculated 30,000. How come?

It turns out that the code that multithreaded all of the ‘calc the routes’ code, was not storing any of the results, so the code that came after it which then went through the (not saved) routes to pick the nearest ones was having to recalculate them anyway. I was essentially doing everything twice.l Worse…this second bit of code was single threaded, meaning all my multithreaded time was wasted and all the work happened in a single thread. What a dork.

A nice simple change to the code to make sure those multithreaded route calcs are actually stored *doh* means that not only did the processing time half, it now gets split over potentially 8 cores instead of 1, so on my PC its now running 16x faster. Combine that with the earlier speed-ups and its likely 18x faster, and because of the frame-spanning over 120 frames, it *feels* fluid as hell, even on insane maps.

Here is the concurrency visualiser showing the loading of the insanely big map.

And here it is zoomed in showing the multithreaded bits.

Those gaps are because each slot makes a single-threaded call to evaluate all of the import bays (each colored block is the code to get a route from one import bay to one production slot), and that call (in the main thread), then spreads it over the worker threads. Ideally I’d find time to eliminate that single thread blocking. Also I have some gaps in there because the end of each threads Calc() has to use a critical section lock to deposit its new routes safely in the main thread. Ideally I’d bunch them up inside a thread structure and dump them at the end.

Anyway, enough tech bollocks, the upshot is that massive factories will now be uber-fast :D.

Anatomy of a bug fix

I thought it might be interesting to just brain dump my thoughts as I do them for a Production Line bug.

The bug is an error message which I get triggering thus:

GERROR(“No room to TrashLowPriorityResource from stockpile”);

Which is handy, as I know exactly WHAT is happening, and where in the code. Thats 95% of the work done already. Amen to asserts! I know that inside the function SIM_ResourceStockpile::TrashLowPriorityResource(), I am not able to actually do what the function needs to do. So firstly what does this function do anyway? I can’t remember… Thankfully there is a comment:

//if we find a resource not needed by current task, destroy it (only one)

Which is pretty handy. So this is a stockpile at a production task (working on a vehicle) and it needs room for a new resource presumably. The code goes through all of the objects currently in the stockpile to check we really need them. here is the code:

    iter it;
    for (it = Objects.begin(); it != Objects.end(); ++it)
    {
        SIM_ResourceObject* pobject = *it;

        //is this object needed?
        bool bneeded = false;
        SIM_ProductionTask::iter rit;
        for (int c = 0; c < SIM_ProductionSlot::MAX_RESOURCE_QUANTITIES; c++)
        {
            SIM_ResourceQuantity req = PSlot->CurrentResources[c];
            if (req.Quantity <= 0)
            {
                continue;
            }
            if (req.PResource == pobject->GetResource())
            {
                if (GetQuantity(req.PResource) > req.Quantity)
                {
                    bneeded = false;
                }
                else
                {
                    bneeded = true;
                }
            }
        }
        if (!bneeded)
        {
            RemoveObject(pobject->GetResource());
            return;
        }
    }

Thats the first half of the function, it then tries to do a simialr thing with requests, rather thanm objects. However I think I see an issue already. That GetQuantity() is checking the total quantity, not for this single object. Nope…thjat makes sense. I guess I really need to trigger the error and step through it, so time to fire up a save game from a player and run it in *slooooow* debug mode. Ok… an immediate crash from a save game..where we have 16 requests and no objects. So there seems to be a bug in the latter code:

    //ok we need to destroy a request instead of a current object.
    reqit qit;

    for (qit = Requests.begin(); qit != Requests.end(); qit++)
    {
        SIM_ResourceRequest* preq = *qit;
        //is this object needed?
        bool bneeded = false;
        for (int c = 0; c < SIM_ProductionSlot::MAX_RESOURCE_QUANTITIES; c++)
        {
            SIM_ResourceQuantity req = PSlot->CurrentResources[c];
            if (req.Quantity <= 0)
            {
                continue;
            }
            if (req.PResource == preq->PObject->GetResource())
            {
                //sure we ordered it, but do we actually have enough for this task, when com,bining requests with objects?
                int stock_and_ordered = GetStockAndOrdered(req.PResource);
                if (stock_and_ordered >= req.Quantity)
                {
                    Requests.remove(preq);
                    preq->PObject->Trash();
                    return;
                }
                bneeded = true;
            }
        }
        if (!bneeded)
        {
            Requests.remove(preq);
            preq->PObject->Trash();
            return;
        }
    }

So what could be wrong here? Ok…well hang on a darned minute. This code is just saying ‘do we really need resources of this type right now???’ rather than ‘do we need ALL of these right now?’  which is more of a problem. We are clearly somehow ‘over-ordering’. The slot in question is ‘fit trunk’. presumably it only needs trunks… They do all indeed appear to be the same type. 4 are ‘intransit’ the rest are queued up at some resource importer or a supply stockpile or component stockpile somewhere.

So the ‘solution’ is easy. First kill off (delete) a request that is not in transit yet, to make room, or if that is not possible, kill one that actually is in transit. This then frees up the required place in our stockpile. But hold on? this is just patching over the wound? how on earth did this happen? The calling code is requesting new resources, when clearly 16 are already on the way! in fact its requesting a servo…

AHAHAHAHA!

what a beautiful bug. The player obviously upgraded to a powered tailgate, which means a servo is needed for every trunk, and yet 16 trunks were already on order. Error! error! I have special code that handles when an upgrade makes an existing resource requirement obsolete (climate control replaces aircon), but not code to handle this case, where additional stuff is needed. This is FINE, I can implement the above mentioned code without worrying how it happened.

That actually wasn’t too bad at all :D

Time for the ‘teenage coding god’ meme to die

What is it with TV executives and non-tech savvy journalists? Can they not do the vaguest bit of research and kill of this myth about ‘gifted child genius programmers and hackers’? its so off-base its laughable, especially to anyone my age who works as a software engineer. If it isn’t immediately obvious what I’m talking about, its characters like this from silicon valley:

And also like this (also from silicon valley)*

And like this from ‘halt and catch fire’

And any number of media stories about ‘teenage bedroom genius hackers’. I guess it all goes back to a single film, in the early days of computers (and the threat of hacking)…war games, starring Matthew Broderick aged 19 released in 1983. The myth of the young genius computer whizz was born, and nobody has seemingly challenged it since.

Firstly…lets get something straight., The ‘cleverest’ programmers are not usually ‘hackers’. Firstly, its much easier to break something than build it. You build software with 100,000 lines of code and 1 line has a potential exploit? you did a good job 99,999 times, versus a hacker who finds that one exploit. are the finest minds in programming really working for the Russian mafia? I suspect they are more likely to be working for Apple or Deep Mind or some tech start-up with fifty million dollars worth of stock options. They get better pay and no threats of violence, which would you choose?

Secondly, computers were invented a while ago now. We have people with a LOT of experience in the field out there now. Amazingly, C++ is still perfectly usable, and very efficient, and given the choice between someone who has written tens of millions of lines of C++ over twenty or thirty years, versus some ‘bright’ kid…I’m going with the old guy/girl thanks.

Learning to code takes TIME, yet because bookshops hawk crappy ‘learn C++ in 21 days (or less)’ bullshit, some non-coders actually believe it. There is a BIG difference between ‘knowing some C++’ and being a C++ software engineer. Writing code that works is fucking easy. Writing reliable bug-free efficient, legible and flexible and safe code is fucking hard. Why do we think that surgeons with 20 years experience are the best choice for our brain operation, yet want software coded by a fourteen year old? Is there some reality-distortion field that turns programming into a Benjamin button style alternate reality?

No.

So ideally, any movie or TV series that features the ‘ace’ coder would have them aged about 30-40, maybe even older. At the very least they would be in the darned twenties. Enough with the school-age hacker god bullshit. Here is a recent picture of John Carmack. I bet he is a better coder than you, or me. He has even more grey hair than me.

While we are on the topic, the best coders are not arrogant, mouthy, uber-confident  types on skateboards wearing hip t-shirts with confrontational activist slogans on them, and flying into a rage whenever people talk to them. Nor do they always blast out heavy metal or rap music on headphones whilst coding on the floor cross-legged, and nor do they ‘do all their best work’ when on drugs, or at 3AM, or after a fifteen hour coding blitz.

These are myths that make TV characters ‘more exciting’. Except they also make them unbelievable and stupid. I’m quite unusual in being a fairly extravert (in short bursts) programmer. Put it down to being a lead guitarist in a metal band 27 years ago. Most really *good* coders I know are actually pretty quiet. They will not draw attention to themselves. they are not arrogant, they know enough to know that they know very little. Really good coders tend not to brag. I brag a bit, its PR but would I claim to be a C++ *expert*. Nope, I know what I need to know. I also only really know C++, a little bit of PHP, and some HTML, CSS, but not enough to do anything but the few things I need. When I meet coders who brag that they know 10 languages, I get that they know the syntax, but how to use them effectively? enough to write mission critical code that a company is built on? I find it hard to believe.

Most coders look pretty boring. Most of us are pretty boring. Most of us are not arrogant shouty attention seekers. The experienced ones know to stop coding by 9PM at the very latest, and to take regular breaks. We also aren’t stupid enough to store backup disks next to hi-fi speakers in the same room (an actual plot point in halt and catch fire). We make shit TV, but good code.  I suspect our portrayal will never change.

 

*All the SV cast are young, but carla seems to be portrayed as younger, cooler, more confident than the rest.

Production Line Blog Video #12 Lot Expansion, Zoom-out GUI & Rendering tech

A mixed bag of stuff this week. I’m crunching away boih at bugs, and balance issues, which make for a poor video, but also luckily we now have lot-expansion plus some zoom-out UI shininess to look at, plus I fire up nvidia nsight to give you a quick preview of how a frame gets rendered in the game. (FWIW I use my own engine, based on directx9 which means I need to worry about draw calls when changing textures. One day I’ll switch to a newer directx version, but I feel no pressing urge to do so yet :D.

Enjoy: