i can't go to hell - i'm all out of vacation days. i watch space rocks and yell at computers for my day job. probably too old for any of this

 

i think i might be burned out on internet social. it's hard to keep doing it. it's hard to even maintain the amount of attention i'm already giving it

 

i am the cause of most of my own problems

 

furthermore, capitalism must be destroyed

 

birdsona: ?????

 

🌎 Ontario, Canada


webbed site
egrets.ca/

eniko
@eniko

hello, little gamedev pro tip from me to you: always put a version number in any file your game saves externally. this includes save files but it also comes in real handy in config files since if you change the way your game's configuration work you can use it to invalidate or upgrade a user's previous config!


mrhands
@mrhands

If there's one neat trick I've picked up from the Amazon© Lumberyard™ engine, a horrible product that has thankfully been discontinued, it's the idea of an upgrade() method for your serializable data. As the extremely experienced Ms. Fox rightfully points out, you should always put a version number in your data. With this version number in hand, you can (theoretically) upgrade all user data to the latest version without ever losing data!

For example, my game had a TaskDataModel struct that needed to be loaded from the save file. In version 1, the task had an "alignment" stored as an integer value. But for my own sanity, I wanted to change this to a string value in the save file. So, I wrote an upgrade function for the struct:

template <>
inline bool upgrade(TaskDataModel& to, QJsonObject& data, int32_t dataVersion, Mode mode)
{
	// ChangedAlignmentIdentifier

	if (dataVersion < TaskDataModel::Version::ChangedAlignmentIdentifier)
	{
		if (data.find("alignment") != data.end())
		{
			int32_t alignment = data["alignment"].toInt();
			if (alignment >= 0 &&
				alignment <= 2)
			{
				const char* alignment_ids[] = { "nerd", "cool", "jock" };
				data["alignment"] = alignment_ids[alignment];
			}
			else
			{
				data["alignment"] = "";
			}
		}
		else
		{
			data["alignment"] = "";
		}
	}

	return true;
}

This function reads the old data, extracts the alignment as an integer, and then writes the new alignment as a string value. The new serialization code can then always assume that the alignment is read as a string.

If the incoming struct already has the latest version, we don't need to call this method at all, and this is handled at a higher level of my serialization system. But crucially, we can upgrade from version 1 to version 24 by running each step in sequence! The real version of this function has these steps:

template <>
inline bool upgrade(TaskDataModel& to, QJsonObject& data, int32_t dataVersion, Mode mode)
{
	// ChangedAlignmentIdentifier

	// AddedPosition

	// ChangedSuccessToDifficulty

	// RebalancedDifficulty

	// ChangedLocationIdentifierToEnum

	// RenamedPhaseToShift

	// RenamedPhaseToShift

	// MovedIdentifierToSheet

	// ChangedDiceSlotsToDataModelList

	return true;
}

If I ever publish my game (ha!), this system means that as long as I keep incrementing the version number for the structs, I can continuously make patches while keeping my players' save files intact. 😊


SomeEgrets
@SomeEgrets

this applies when you are inventing any kind of data format you're saving to disk that doesn't already exist as a well defined format (and when you are using a flexible format that lets you define your own schema like JSON, always include a version in your schema)

we have so much scientific data that's a nightmare to work with because as far as any of us know there is no standard format for "raw (i,q) signal data from a radar return" so the grad students had to invent their own in like 2003

the metadata they stored in the raw binary header of each file changed over time and it took the new students this year like a week to implement a reader for it in their own programs even with a full byte-by-byte specification

additional wisdom for free

if your format looks something like (optionally, with repeating metadata/payload sections)...

  • file headed
  • object metadata
  • binary object payload

... then explicitly specify the format your payload is in, either in the file header or the per-object metadata, separately from the file format version

it probably costs you like a couple bytes but now if your encoding ever changes - eg. because you're ingesting a new pixel format you didn't anticipate or a new more efficient encoding emerges five years from now - you can just integrate it without changing your format, and maintain binary compatibility with everything you already have. and your old code will be binary compatible enough to be able to say "i recognize this file but not this encoding" and fail gracefully about it


You must log in to comment.

in reply to @eniko's post:

in reply to @mrhands's post:

in reply to @SomeEgrets's post:

so like the way i see this, your file/format version tells your code what fields to expect in the file header and object metadata header

the encoding specifies what format your actual payload is in like how mp4 and mkv files are a defined format but the actual data they contain can be h264 or hevc encoded in any number of pixel formats

so like if you were storing imagery for example (although there are already plenty of formats for this...) the file version might tell you to expect certain fields in the metadata in a certain binary format and layout

and then your encoding tells you whether the image data itself is just raw 8-bit samples, interleaved mono12, a common colour format like yuv422, or an encoded format like LZW or huffman coding or JPEG or HEIF or whatever

container format & metadata structure vs actual data encoding

Pinned Tags