So…I am 99% sure I’ve fixed this, but it was a bug hell nightmare and worthy of a blog post. It also kept me coding till midnight last night, so I did probably 14 hours work yesterday. arrrghhh. no wonder I slept late this morning. Anyway…
Democracy 4 releases into steam early access on Tuesday, but we already have 3-4,000 players, and I had no showstopper bugs., Most noticeable the game was 32 bit, and only in English. Very recently we switched to 64 bit, and I guess some combination of that, and some people playing in different countries started me on a journey of bug tracking hell.
I got one bug report “game crashes on next turn, or if you change sound volume”. I tested my copy, worked fine. Tested on my laptop too, worked fine. Asked them to try and repro on another PC, arrogantly assuming user error. Then another customer reported the same thing, then a third. This is NOT GOOD. I asked for debug logs, which handily my engine does do, and are fairly comprehensive.
Nothing in the logs explained the actual crashes. Everything looked good. nothing was especially shocking except an error about failing to find a specific neuron (no name supplied). This could be just a typo, or a badly written mod. This *shouldnt* happen, but the game should recover. its not critical… but then why didnt MY copy give me the same error… weird?
I put together a new build of the game that outputs the NAME of the neuron it can’t find (should have done this anyway tbh), and I get back a log from a user where the missing neuron is called 0,00.
WUT?
That sounds like a bug with a comma somewhere, as the equation Socialist,0.00+(2*x) for example must have a typo, so instead of the name ‘Socialist’ we are getting the value. Ha! a simple typo. But why am *I* not getting the bug? I do a windows search of the whole data folder looking for 0,00. I get two hits… neither file actually contains the string. Windows search is WORSE than useless. Weird…
So I change the code again to output the name of EVERY attempt to get a neuron by name so I can compare mine with the users, and see where in the code they were just before this goes wrong. This turns out to be no help, as its inside the Voter processing, which makes no sense… there is simply no way any string like that is being passed at this point in the code. this makes ZERO sense.
And then I notice it. I’d normally missit…but it was weird. REALLY weird. In the debug logs of the players:
PreCalcCoreSimulation time: 0,70 seconds
No big deal, this value will vary based on CPU, but on my PC it said this
PreCalcCoreSimulation time: 0.92 seconds
Yup. Mine has a period/full stop, theirs has a comma. No big deal, its debug logging, who cares. Maybe its even an artifact of copy/pasting the log to my forums… I check this…nope. I ask the player to verify thats really a comma in that log file, not a full stop, they confirm its a comma. How? HOW is this possible? I check the code:
char temp[256];
sprintf_s(temp, 256, "\n\nPreCalcCoreSimulation time: %.2f seconds\n\n", elapsed);
GetDebug()->DebugOut(temp);
This is simple stuff. I just used the very low level, very basic, very simple good old fashioned sprintf function to convert a value into a decimal number in a string. I do some googling and… what the actual fuck? apparently even good old c++, ignoring anything higher level, is looking up the regional settings, and deciding whether or not commas or periods are used in decimal strings. In english its full-stops (.). It is VERY VERY important to the structure of my code and engine that its full stops. When the majority of your data uses comma-separated values, you cant go messing around with the definition of a comma for fucks sake…
So theoretically easy fix, find the function that hacks in the ‘locale’ to be English for this app…and boom! No! it doesn’t work. Sure, I can use
setlocale(LC_NUMERIC, "en-US")
to force the number format to be full-stops, and when I check that the locale really has been set…it has! and everything is fine but… once the game loads a new level, everything has gone haywire, all the values are wrong, all the policies are set to zero, and checking the locale returns what the PC has set it to (I temporarily changed mine to commas as a test). In other words….my setlocale is being overrriden.
Arggh. Now its late, I am tired, I am thirsty, but not enough to leave my desk and lose my concentration. I am very tired now. I spam the code with multiple checks to read the locale at every possible location to see where the hell it goes wrong… And eventually I find it. Its the text renderer, which does use some middleware, to convert fonts into vector graphcis, render them into memory and then paste them as strips to the screen, dynamically. Its all very clever and supports any language, any font size, its super l33t. It also has its own opinions about locales.
So now the game is fixed, by me setting the locale to the US the millisecond the game starts, and then the minute its rendered its first bit of text (and thus the middleware has overrriden me, it sets it AGAIN, to ensure no funny business. Everything now seems to work. Hurrah. I hate computers. I need sleep. Why the hell cant humans agree on what a decimal point looks like in 2020?