Plan, just the right amount, not too much, not too little.

A few months ago I built some code that I knew would fail, in a few months time when the data had matured.

Plan was to come up with a plan on how to deal with it when it needed to be dealt with. I put some clear errors in my data editor telling me I was hitting the problem and continued development.

Then I added procedural data packs, and forgot to put the same warnings there. Today I spent a few hours debugging the problem until the light dawned and I realized my plan to make a plan had failed.

Time to really make a plan!

The problem involves fast searches by category in a dictionary of categorized words. The searches take a set of categories and the form "match any, match all, exclude any, exclude all".

switch (rFilter.m_MatchType)
{
case MatchFilterRuntime::eInactive:
return true;
case MatchFilterRuntime::eMatchAll:
//Include if ALL were found
return ((CategoryFlags & rFilter.m_CategoryFlags) == rFilter.m_CategoryFlags);
case MatchFilterRuntime::eMatchAny:
//Includ if ANY were fould
return ((CategoryFlags & rFilter.m_CategoryFlags) != 0);
case MatchFilterRuntime::eExcludeAll:
//Include if not all were found
return !((CategoryFlags & rFilter.m_CategoryFlags) == rFilter.m_CategoryFlags);
case MatchFilterRuntime::eExcludeAny:
//Include if NONE were found
return ((CategoryFlags & rFilter.m_CategoryFlags) == 0);
}

As you can see above up to now I've been able to reduce my category checks from a set of strings to fast and furious bitwise operations. The error I just is is having more than 32 categories in a single dictionary.

When I first wrote the code I had the sneaky hope that 32 categories in a dictionary would be plenty, that I could just split my data into more dictionaries but as the data takes shape this now seems unlikely. So plan A [my favorite, do nothing] is a fail.

Now I'm contemplating my options:

- Larger single sized bit storage size. 64 bits might buy another three months development right? 128?

- A proper bitset object

I think the second option is the winner and should be fun to implement.

Given I already have category sets the bitset could just be a byte collection of known size for any dictionary. I looked at TBitSet in unreal C++ but it doesn't seem to provide the functionality I want and adds some level of complication.

Of course, now I look at how much code I've got passing these flags around as int's and I don't whether to be proud of the quanity or sad that I didn't just bite this off when I knew it was a problem and save the cost of retro fitting the new structure everywhere! Don't over engineer until you know the requirements right? Actually, I'll stand by that, which this will waste a day of my time, I've saved that time elsewhere by not engineering to specs made to suit the elegance of the solution, not the problem. So I hope anyway.

I'll post the code for the new bitset here when it's working smoothly, probably tomorrow.