Game Design, Programming and running a one-man games business…

An experienced coder’s approach to fixing business mistakes

I was the victim of an email today that was basically an invoice that was wrong. A company that has sent me a wrong invoice in the past has sent me it again another 3 times now, with increasing levels of angry demands and warnings about credit ratings. This raised my heart rate and anger level to extremely high distressing levels, but this is not a rant about how some companies are just awful, that would be too simple. This is more of an investigation into why stuff like that happens, and how software engineering can help you learn how to not fuck up like that.

I have been coding for 44 years now, since age 11. In all that time I’ve only coded in BASIC, C, C++ and some PHP, but 90% of it has been C++, so that’s the one I’m really good at. I don’t even use all of C++, so lets just say when it comes to the bits I DO use, I’m very experienced. This is why I’m using the term software engineering, and not coding or hacking, which implies a sort of amateurish copy-pasting-from-the-internet style of development. I’ve both worked on large game projects, and coded relatively large projects entirely from scratch, including patching, re-factoring and a lot of debugging. I coded my own neural networks from scratch before they were really a popular thing. Anyway…

Take the simple example of my experience today. An invoice is wrong, and sent to a customer anyway. This has happened three times now, despite the customer replying in detail, with sources and annotations and other people copied in, AND including replies from other people at the offending company who openly admit its wrong and will be fixed. Clearly this is a smorgasbord of incompetence, and irritating. But why do companies do this? what actually goes wrong? I think I know the answer: Once a problem like this is finally flagged up (which often involves threats from someone like me to visit the CEOs home address at 3AM to shout at them), the company ‘fixes’ the mistake, maybe apologizes, and everyone gets on with their life…

This is the mistake.

Companies want to ‘fix’ the mistake in the quickest, easiest and cheapest way possible. If an invoice was accidentally doubled, then they ‘go into the system’ (they just mean database) and they halve it. Problem solved, customer happy. Now I don’t give a damn about the sorry state of this company’s awful internal processes, but I am aware of the fact that companies like this try to fix things in this way, rather than the true software engineering way. So how SHOULD it be done?

Before I was a full time computer programmer I worked in IT. I had a bunch of jobs, gradually going up in seniority. I saw everything from ‘patch it and who cares’ mentality (consumer PC sales) to ‘The absolute core problem must be fixed, verified, documented and reported on with regards to how it could have happened’ (Stock trading floor software). I can imagine that most managers think the first approach is faster, cheaper, better, and the second is only needed for financial systems or healthcare or weapons systems, but this is just flat out wrong.

When a mistake happens, fixing it forever, at a fundamental level, will save way, way more time in the lifetime of the company than patching over a bug.

Spending a lot of time in IT support taught me that there are many more stages to actually finding the cause of a problem than you might think. Here is a breakdown of how I ended up thinking about this stuff:

Stage 1: Work out what has actually happened. This is where almost everyone gets it wrong. For example, in my case I got an email for a £1k bill. This was not correct. Thats the error. Thats ALL WE KNOW RIGHT NOW. You might want to leap immediately to ‘the customer was charged too much’ or ‘the customers invoice was doubled’ but we do not know. Right now we only know the contents of the email are wrong. Is that a problem with the email generation code? Does the email match the amount in the credit control database? Until you check, you are just guessing. You might spend hours trying to fix a database calculation error you cannot find until finally realizing the email writing code is borked…

Stage 2: Draw a box around the problem. This was so helpful in IT. Even when you know what you know, you need to be able to define the scope of the problem. Again, right now all we know is the customer ‘cliff’ got a wrong invoice. Is that the problem? Or is it that every invoice we send out is wrong? or was it only invoices on a certain date? or for a certain product? Until we check a bunch of other emails we do not even know the scope of the problem. If its systemic, then fixing it for cliff is pointless.

Stage 3: Where did things go wrong. You need to find the last moment when everything worked. Was the correct amount applied to the product in the database? was the correct quantity selected when entering the customer’s order? was the amount correct right up until the invoice was triggered? Unless you know WHERE something went wrong, you cannot work out WHAT or WHY.

Stage 4: Developer a theory that fits all available data. You find a line of code that seems to explain the incorrect data. This needs to not only explain why it screwed things up for cliff, but also explain why the invoice for dave was actually ok.

Stage 5: Test a fix: You can now fix the problem. Hurrah. Change the code or the process and run it again. Is everything ok? If so you may be preparing a victory lap but this is laughably premature.

Stage 6: Reset back to the failed state: Now undo your fix and run it again. Seems redundant doesn’t it? Experience has taught me this is vital. If you REALLY have the fix, then undoing your fix should restore the problem. Maybe 1 in a 100 times it will not. The ‘random’ problem just fixed itself for some indeterminate period. Your fix was a red herring. A REAL fix is like a switch. Turn it on and its fixed, turn it off and its broken. Verify this.

Stage 7: Post Mortem: This is where you work out HOW this could ever have happened. This is actually by far the most important stage of the whole process. If you just fix some bad code, then you have just fixed one instance in one program. The coder who made that mistake will make it again, and again and again. The REAL fix is to make it IMPOSSIBLE to ever have that error again. This takes time, and analysis. The solution may be better training, or it may mean changing an API to make it impossible to process bad data. It might mean firing someone incompetent. It might mean adding a QA layer. Whatever the mistake was, if you do not go through this stage, you failed at the task of fixing the problem.

Coders and organizations that work like this have fewer bugs, fewer mistakes. They need smaller QA teams, and less time devoted to fixing mistakes and implementing patches. Their complaints department is tiny, and yet still provides great communication, because there are hardly any complaints. In short, companies that run like this are awesome, popular, profitable, and great places to work. Why is everywhere not like this? Because it requires some things:

  1. A culture of giving people fixing problems free reign to follow the problem everywhere. If only the department that sends the emails is allowed to get involved in complaints about email, database errors or coding errors will never get fixed because they CAN never be fixed. The phrase ‘not my department’ needs to be banned.
  2. A culture of accepting that someone is STILL working on a proper fix, even after the complaint has been handled and the customer is happy. When there is a bug, there are TWO problems to fix: the customers bad experience AND the company process failure that allowed this. Managers who pull people away from bug-fix post-mortems to fix the next thing right now are a curse.

Of course your mileage may vary. Be aware I am a hypomanic workaholic who works for himself so my ‘advice’ may not be applicable to everybody, but I hope this was interesting and thought provoking anyway. Too many coders are just copy-pasting from stack overflow or asking chatgpt to quickly patch their bad code. Working out how to fix things properly is a worthwhile pursuit.

First full solar month stats

Ok, the sun is still in the sky, but its late enough that I am going to declare November over now, and blog about my generation from my solar farm. First here is the exciting main chart thing:

So thats 34.2 MWh, or to put it in rooftop-solar or home-energy terms, 34,248kwh of lovely renewable green solar power! Not bad I think for a pretty wet, cloudy November in the UK. June in the Sahara desert would be different…

A crude extrapolation to 12 month would give 410MWh which is obviously way too low. I am expecting more like 1,000-1,200 MWh for the whole year. If I can get 1,200 that would be nice, and I would not worry too much about the business case for the whole thing. However that depends on the ongoing costs. I am still comparing quotes and discussing ways to cut that. The actual annual sums for each individual component always seem reasonable, until you multiply them by 25 years…

I do have some other software that analyses the output and gives an estimate of how good or bad you generation has been. That software suggests that November output for me has been noticeably better than expected, but you shouldn’t get too excited extrapolating from a single data point. Lets not forget the panels are new(ish) and clean(ish), and that will not always be the case. The real reckoning will come after a full years generation and operations.

In other news, I got paid! So its only for part of October (we had some downtime, and also did not switch on until the 4th) but the money turned up in my account on time and the right amount, which is never something a small business can take for granted. If you are a UK residential electricity customer of OVO, then I am providing some of your power :D. I get paid for the power plus VAT (sales tax in UK of 20%) so I have to pay that tax to the government (useless busywork!). It definitely feels good to have even just this ONE entry in the company accounts that is green instead of red!

Global investment and the coming chaos

When I am not making games or building a solar farm, I am quite into investing on the stock market. I am NOT a day trader, but over the years I have built up enough to invest that it does require me to constantly keep an eye on stuff. I also like reading about the world, and technology, and politics, so it fits in nicely. As a result I spend a lot of time thinking about where to invest.

Investing is super hard, because it involves keeping emotions in check (hard), being objective (hard) and for best results, being happy for absolutely everyone to tell you that you are wrong (I find that quite easy tbh). It also involves a lot of risk, and you have to be ok with all of that. Plus it involves a lot of checking numbers, and being objective, and not making panicked decisions.

Because I think that short term swing/momentum/day trading is more likely to result in losses than gains, I focus on picking shares that will give me a decent return over the next 1-5 years. I have found this is the approach that works best for me. Its certainly not as EXCITING as day trading, but I actually want results, not a sugar rush. So I find myself thinking hard about what the future may bring. Ultimately I am a value investor: I am buying stocks where I think the underlying company will be very profitable in the future. I DO hold some dividend stocks and bonds, but most of my picks are for stock-price-growth, where I assess that a company will have rising profits in the next few years, and I will sell once the stock price catches up to my point of view.

Individual companies can be very good investments, regardless what the people selling funds and generic investment advice tell you. If its obvious that 50 of the stocks in the FTSE100 are rubbish, why would you want to own ANY of them? If your funds are limited, indexes and funds might make sense, but when you are able to pick 50-100 stocks, its worth taking the time to pick winners.

So anyway, with all that in mind, what am I currently thinking about when it comes to upcoming events and themes that might influence investment? I have a bunch of ideas:

  1. The decline of the USA : This is quite a big one, and probably very triggering for people who live there, but I think USA in 2024 is the UK in 1930s-1940s. A big global superpower that has not yet realized that it is screwed, despite all the signs being there. The US has incredibly tribal politics, huge social-division, bad levels of skills/education, bad infrastructure, a colossal debt problem, and a bad global image. Sure, the $ is a popular global currency for now, but in 5,10,15 years? I think China is clearly overtaking the US in everything that matters. I also think the US is so focused on its own awesomeness that they will not manage the decline well. I am hugely over-invested in US stocks but will be diversifying out of there.
  2. Clean Tech Revolution: The entire planet will embrace electric cars, solar & wind power, battery storage and heat pumps. Nuclear Fusion is too late, too pricey and too concentrated. There is no stopping this transition now. Along the same lines we will likely see a global transition away from meat consumption towards vegetarian or vegan diets. Thats a movement that is just too big and especially too popular with the young for it to be stopped.
  3. Rise of Asia: I mostly focus on China. The predominant view in western media is filtered through frankly xenophobic and tribal hatred of China in the US media. But the Chinese are starting to lead in tech, science, infrastructure, and geopolitical influence as well as manufacturing. They even have their ow space station, something the USA cannot afford. I do not expect war with Taiwan, but I expect savvy Chinese leaders to rattle just enough sabers to keep bankrupting the USA with its ridiculous military budget
  4. New Space Race: Not just spacex, but a whole range of new space startups is revolutionizing our capabilities in space. Starlink is the first tangible benefit but there will be more. Space Tourism will be one part, but zero-g manufacturing may well become a thing for some specialist pieces of technology. I fully expect all major cabled telecoms links to be replaced by satellite networks soon. Its just so much simpler.
  5. Fall of Russia / Chaos in Europe: Russia will have a messy transition when Putin dies. The country is not in a good state, but cannot be allowed to collapse into chaos because it has nukes. I can imagine a world where Europe steps in to handle the transition, in a similar way to the handling of eastern-european ex-soviet countries after the USSR collapsed. It will be messy and awful. I do not expect the UK to rejoin the EU, so we may escape some of the costs. It will be tough
  6. Climate Chaos: Insurance companies are likely screwed. We have been lucky so far but one day a big hurricane or flood or other climate event will wreck a big famous city. Might be New York, might be London. Who knows. Impact will be extreme. I will not buy insurance company stocks. Also worsening weather will destroy so much food production. I sometimes speculate on commodity prices to play this, and may do more. Global food price rises may destroy fast food chains, when people can barely afford the ingredients and cut back on dining out or takeaway food out of necessity.

Thats my list for now. I also have a speculative investment in quantum computing, just in case it really becomes a thing. I also expect AI and robotics to be big, so hold a lot of AI/Chip/Robotics stocks. Those are quite trendy though, so those investment ideas are less contrarian.

Solar Farm month#1: First ever earnings! How to earn £669 an hour. Maybe. (kinda)

OMG its actually happened, I have been sent an invoice (well a statement really I guess) for the sale of my solar power for the first time. The spoiler is that its about what I expected, but that doesn’t stop me breaking it down in extreme detail! If you are googling for the details of what a power purchase agreement (PPA) for a solar farm is, and how its calculated and how much you actually get paid etc, then you have probably found the right blog :D.

So firstly, lets have some disclaimers. The farm was only energized on October 4th, and this payment is for October, so we already lost 4 days (we didn’t switch on at 8am :D). We also had to turn everything off for two days later in the month so that we could fill in some earthworks around some high voltage cabling, and they wouldn’t do that while they were live. TBH they were only 400v I think, and this seemed a bit overkill to me, but then I don’t drive a digging machine :D.

Also be aware that this was OCTOBER, so not exactly peak sunshine roughly half way up the west side of the UK, so do not extrapolate this in a linear sense to be a typical month because it was not. So how much is the total paid?

Roughly £5,300

To break it down in more detail, the amount is calculated like this:

  • Electricity output £4,400
  • REGO Certificates £0
  • DUoS benefits £945
  • DUoS costs -£12
  • Triad benefits £0
  • Admin fees -£25

All figures are slightly approximated as I’m not sure what I can / cannot be explicit about and don’t want to be told off :D. Anyway to break it down…

Electricity output is the amount I can see that many MWH of power for. More on that later.

REGO certificates are things I do not have yet due to stupid insane *are you kidding me* levels of bureaucratic BS, but I WILL get them, and they will be backdated. I need to collect a ton of paperwork and then sit and wait for months for the awarding government body to try their hardest to find a missing comma or i without a dot. I wish I was joking.

DUoS. Well this is a whole world of insane complexity. The acronym stands for ‘Distribution Use of System’. Its the bewildering network of charges and fees and credits that are calculated depending where you are in the national network of distribution power lines. Basically if you generating power next door to a substation, and the re is an aluminum smelter next door to that, then all of the power you generate will stay local, and not flow out onto the wider ‘national grid’, so you are not heaping pressure on the big pylon networks. Thats the ideal situation. Amazingly, you can get both charges and benefits from the exact same system. My solar farm is fairly remote, but small enough that our energy will only ever go to the nearest town. So we got charged £12 for being so remote and using some local pylons, but credited £945 because we are remote enough that us powering that town a bit more means LESS pressure on the big power lines heading from elsewhere to that town.

Think of it as toll road charges.

Triad benefits: This is something very weird and arcane that makes DUoS look simple. Here is the googled explanation.

The Distribution Network Operators (DNO) pay what you might call a standing charge to National Grid (NG). This charge is set according to the amount of electricity being drawn by the DNO when the UK as a whole is using the most. More specifically, it is the average MW being drawn during the three highest half-hour periods (subject to being a minimum of 10 days apart) of UK electricity consumption between 1st November and 28th February inclusive. The cost incurred by the DNO is then passed on to energy suppliers who in turn charge it to their customers. (from:https://www.nowthenenergy.co.uk/news/triads2021 )

If you think ‘WTF? why do they choose three? and why those months? and why half hour? Yup : welcome to the UK energy market. You thought C++ was complex? Ha! Anyway, looks like we are unaffected which is good?

Now actually I will get paid MORE than this, because I get paid VAT (sales tax) but I then have to pay that to the government anyway, so its just a bureaucratic waste of time and doesn’t figure into the calculations.

But Wait! There is more! The ‘Electricity output’ figure is more involved. My actual PPA did not start until November 1st. From that point I get a fixed price per MWH for the next year, but up until then, the price was ‘floating’. I got paid the ‘market ‘system sell price’ price each half hour. (The whole UK commercial energy sector works on half hour billing). They send you a handy spreadsheet showing what the price was every half hour for the whole month. Do you think it was relatively stable? Ha. Behold:

Well I guess two things stand out right? Firstly what the hell was that spike? and secondly WTF does it go negative? To answer the second part first, YES, at times I was being charged actual money to generate power. When its super-windy and super-sunny, and mild, and the Uk has too much energy and we cannot cope with any more, the wholesale energy price goes negative. It happened a few times that month, but as you can see its very short lived. With a solar plant, you can’t easily switch it off (and you wouldnt want to keep turning it off and on again), but if you are burning physical fuel like gas, coal or oil, or even wood, then that price signal tells you to stop imemdiately, which is why I think it recover.s

But hey-ho check out settlement half hour #564, when I was paid a whopping £669 per MWH. Oh yes. Sadly there were only a few brief periods of this.It seems to have happened on the 11th October about 75% through the day (so 6pm) Lets look at country wide stats. Seemingly no explanation here:

But in a sense… it doesn’t matter, because annoyingly its October and solar output at 6PM on 11th October was likely pitiful, if not zero. I checked…, and it was 4.89kw at that time. So the super high price only earned me £3.27 per hour. Actually less, because the power trailed off and the price dropped off too.

FWIW this is why in some cases lithium-ion grid-scale batteries can make sense. If I had one (I ran the numbers and couldn’t justify it for such a small site), I could have stored 1 MWH in it from earlier in the day, then let it flow out at 6PM and pocketed a handy £669. The trouble is I would need about £200,000 to install that battery, and you need to be sure you are going to get enough of a return on those occasional spikes to make it worthwhile. Plus other people are doing the same thing, so as more grid scale storage is rolled out, that chart is going to look a lot less spiky. Especially over the lifetime of the farm (25+ years). Note you can also do ‘peak shaving’ which improves the economics. I blogged about that previously.

Anyway, all very exciting to see actual payments heading my way. 10 days until the money is transferred. It will be WEIRD to see money flow INTO this company. Its been an absolute money pit until now :D.

Solar Farm data-collection begins!

You might think that collecting data on how much energy your solar farm has generated would be a simple matter, where you have an app on your phone, or a website, with a single figure that was just easy to rely on and so on… Ha no. Obviously not. Everything is a billion times more complex when you go from the rooftop kilowatt-hour level to the grid-scale megawatt level.

There are THREE systems (oh yes) that are reporting the output data from the farm. We can call them the Solis, Orsis and Meter systems. Right now, I have data from two of them (although patchy) and the third one remains a mystery that I think I get access to through a third party, or maybe they just show up in the final invoices I get when I get the payment for the energy. Why are there three systems? surely its all one level right?

There are ten inverters on the site, each with up to 20 strings of solar panels. They are all made by Solis, but are different capacities, ranging from 80kw to 110kw in output. All of these inverters report their data, which as I recall is connected by actual physical cabling, running alongside the actual power cables, and all of that then gets connected to the big old wide interwebs using a router thats in our switchgear building. This means I can log into a portal either on my phone or a webpage on my desktop and read off the output of the whole site, or each inverter. The basic view looks like this:

This is the data so far today, which is actually looking pretty good. Total output today is 1.154MWH. For those not familiar with the terms, that’s 1,154kwh. For perspective, a high-end electric car has a battery size of 80 kwh. You probably use between 10 and 20 kwh a day in a house this time of year in the UK, assuming no EV or heat pump. Late October in the UK is bad weather, and very far below peak output, so seeing these numbers is encouraging. We have been connected for about 3 weeks, but have had 2 days complete downtime, and a lot of rain, so 40.7MWH in that time is fairly good I think. You can look at stuff in much more technical detail:

The yield figures differ for each inverter mostly because their capacities are different. I know people imagine a nice flat square field full of panels with a nice round 200 panels per inverter, all of which are identical…and oh how I wish this was the case for my farm too, but pretty much everything is slightly unusual about this site. (I was told by the developer that this was the hardest site they had done in ten years…just my luck!). Anyway, if you want to waste even more hours of your life, you can stare at stats for each inverter, which is pretty much there for fault-finding purposes but also kinda cool:

This is the screen where my geeky interest runs into the brick wall of me not being an electrician, but luckily other people also check this stuff, and the system will trigger alarms if something unusual happens, which would represent a cable breaking or a connector failing, or worst case, an inverter actually failing. Obviously right now it all works great :D.

Anyway…all of that is the inverter-level reporting, but thats not the ‘real’ output figure. There is also a site reporting system through a different portal (orsis) that is as far as I know, oblivious to the individual inverters and just looks at the combined power flowing out of the system. Thats also got a web portal. It might have an app too, I have not searched for that yet. Already most of my phone home screen is various solar/battery/farm/ev reporting apps :D. Here is the basic orsis view:

In comparison this reporting seems kinda rubbish, and it seems to not show the current day at all. It relies on a modem (WTF is a modem these days?) to send out data. TBH the whole UI for this system has a sort of ‘coded in 1990’ vibe to it. Maybe they need to hire a game designer with extensive experience of flashy data-visualization to do a new front end :D.

But hold on, I mentioned three systems right? Well there is a meter on the site that is remotely read by a special company, which is called a meter operator. It might seem ridiculous to need yet another service to just read a simple string of digits, but its worse. You also need to have a data collector and a data aggregator. These three different names for basically the same thing (reading about 1k of data a day) are how the insane fees that get charged are justified. In a sane world, some big tech company like Microsoft or Google would handle this sort of thing for a flat fee of $9.99 a year. Unfortunately this is not the case…

So anyway, as I understand it, that meter gets read remotely and the data from it passed by the Meter Operator (MO)/Data Collector(DC)/Data Aggregator(DA) to the buyer set out in the Power Purchase Agreement (PPA), and then I get sent some money.

Eventually.

Hopefully.

If I had £1 for every energy company acronym I have encountered I wouldn’t need a solar farm. There is then all the excitement of applying for, and getting accredited for the REGOs (Renewable Energy Guarantee of Origin), which always takes MONTHS and has a level of bureaucracy that would make a Tax collector blush. I hope to have all the information I need to finalize my application for that within a few weeks at most. Obviously you cannot pre-apply, because that would be efficient.

Anyway, I am currently in solar-stats-heaven. I was musing about buying some nano-leaf programmable RGB light blocks for my office wall to connect to the solar output from the farm. It seems like an SDK for the blocks is all very easy, but I would need to learn how to code some system to log into the web portals and periodically scrape the data. I probably cannot be bothered :D.