I just added pak file support to Democracy 4. Its something I already had coded for an earlier game, but I had to do a bit of fussing to get it to work properly with democracy 4.
Pak files are basically big phat files that contain other files inside them. If you are a new developer, you probably have no idea they exist, because you probably use unity and AFAIK they handle it for you. Pak files are pretty old school, as I recall Doom and Quake used them (maybe called wad files), anyway the principle is pretty simple:
A pak file contains two sections, an index that tells you where all the other files are inside the main section, and a big phat list of data that is the contents of those files. All of this gets stuck in a single flat blob of binary data. A class exists that lets you grab the memory address of the file you want if you pass in the name of the file, so hopefully to anybody who didnt write the pak file code, using it is easy. You can read the contents of a file just like its on disk, except you have to use functions that read from memory, not ones that explicitly read disk files.
In my case, that meant stepping into the engine code, and the opengl stuff that reads in graphics files (we are only using a pak right now for dds and pngs), and just changing the contents of one function. Instead of using this code to load the data ready for creating a dds:
fread(filedata, filesize, 1, fp);
I now use this
memcpy(filedata, GetPakFiles()->GetData(pentry->StartOffset), filesize);
Big deal :D. Similar changes happen for pngs. To keep things super-simple, the old code for loading a file will run if the pak file reports it cant find that file, so you can stick a ‘loose’ png or dds file into the bitmaps folder structure and it will still get found, and the rest of the code doesnt even know the difference, which is perfect for mod support.
Most of the hassle in getting this to work was just writing some code to enumerate (that means list really…) the contents of a folder from within the pak file. I had to support this for stuff like minister profile pics, because the code previously would ask ‘what files are in this folder, I need to know so I can select a random one’, and now that code gets handled by the index at the start of the pak file instead.
So why bother with this?
Basically speed. Do you know wwhat the read speed of your hard drive is? Checking a random new one on amazon.com shows 6Gb/s. I assume thats bits not bytes, so thats 750MB per second. My hard drive is a little bit older, so lets say 500MB/second. I’ll just copy a big chunk of the source and obj files from D3 to another disk, brb…
Ok…copy speed is between a low of 1MB/second up to 100 MB/second. Wow. Thats so much slower. Why?
A HUGE amount of bullshit happens on a PC when you access files. To simplify it, it goes something like this: *deep breath* You ask to read in a file. The OS then looks up the file table to check that file exists. it then asks the security system if the current user has permission to access that file. When it gets a yes, it then opens that file, and sets attrributes so other processes will know that file is in use. The antivirus software then kicks in, and hooks into the file read so that it can check to see if that file is excluded from scans or not, and gets ready to analyze its contents. The O/S then has to use the file-table to work out where all the various scattered chunks of that file are, and start reading in each block. This means talking to the driver, and ultimately to the hardware, which may also have to check its cache to see what blocks have been cached and whether or not it has to start the glacial process of spinning an old physical drive or not (faster with SSDs obviously).
THEN! when the file read is complete, we can close that file again, notify the system that its not in use by our process any more. We can then start the process of opening the next file in our list..
You do that bullshit for EVERY DARNED FILE. But the good news is… if you have a big phat pak file…you do it once. Just once. The rest is free.
So Democracy 4 goes the extra mile, because our pak file is small (only a few hundred MB). We dont just open the file on startup, we stream all 200MB into RAM. That should take way under a second. We then have the entire file system of dds and png files in memory already, and able to be loaded almost-instantly into our engine. (RAM->VRAM is mega fast)
99% of players will not notice the speed difference. But if you have especially shit anti-virus running on a laptop, in low-battery mode, with an extremely fragmented hard drive, running democracy 4 on a train, you will be glad I bothered. Its really easy to code. My PakFile code has 263 lines in it. Many of them are whitespace or comments.