Yumina the Ethereal

ss (2013-07-07 at 02.37.39)

So Yumina’s finally gone gold master. I came onto the Yumina project pretty late, as a result of my involvement in Seinarukana and Trample on Schatten I was asked if I could also work on the code for Yumina to resolve outstanding text formatting and interface issues. Over time, it ended up ballooning into a whole lot more as more issues were uncovered. It’s been pretty tricky getting this far, but I think the results are worth it.  Here’s a little of what I’ve done while on the game.

Note that there’s a lot of nitty gritty details about the internals of the game. While they’re not story spoilers, they might spoil some non-obvious aspects of the game.  Since this post is all about technical issues and fixes, it might not come off as very flattering. I don’t really want to give the game a bad reputation, cause it’s a lot of fun. Please keep that in mind while you’re reading. I’ll be off praying I’m not breaking NDA by talking about this stuff!

Making it all work

My primary (original) goal of working on Yumina was to resolve issues with the game UI. Anyone familiar with working on Japanese RPGs is that while English might not necessarily be more verbose, it certainly takes up a lot more space. That means you either have to abuse abbreviations, shorten text, or you have to make more space for the text. Additionally, a lot of the UI related text was part of the game’s source code, which would need to be properly localized.

There's be way too much to show you if I went through all the stuff that was changed, but in short, through the hard work of the Yumina English staff, the game looks amazing.

There’s be way too much to show you if I went through all the stuff that was changed, but in short, through the hard work of the Yumina English staff, the game looks amazing.

Most of the UI could be modified through image work, but if you wanted to make adjustments to the layout, you’d have to dig through the code. Yumina, unlike a lot of projects I’ve worked on in the past, was coded in c++. Instead of using a type of scripting format for building on screen UI, the entirety is coded directly in the games c++ source code. I’ve never worked on a c++ project nearly the size of Yumina, it’s absolutely huge. Individual segments of the game could often be controlled by systems with sometimes upwards of tens of thousands of lines of code, and identifying the code that controlled individual visual elements was an arduous task. Simply identifying what needed to be changed, and what to change it to, was a huge challenge.

There's pages and pages of code like this to define UI coordinates and sizes. It's not always so clear cut how it works.

There’s pages and pages of code like this to define UI coordinates and sizes. It’s not always so clear cut how it works.

It doesn’t help that I’m not very up to date on c++ coding. I’ve done it a bit in the past, but it’s not something I spend a lot using, so there was a bit of a learning curve. Once I got myself familiarized with the overall design things became much easier to work with.

That’s a good thing, because it quickly became apparent that there were other issues that needed addressing. For one, the game didn’t work at all on computers that didn’t support Japanese text. As a requirement for localization, the games should run on any English PC without any need to fiddle around with localization settings. The game also had some difficulties displaying English text, where some letters would be inconsistently aligned. It was a subtle problem, but it was one of those things that just drove you nuts.

This is from seinarukana, but both games have the same issue. Notice how the 'r' in particular appears out of alignment.

This is from seinarukana, but both games have the same issue. Notice how the ‘r’ in particular appears out of alignment.

The text issue was pretty interesting tracking down. One thing I have to say, text rendering is a complicated thing. The text goes through a lengthy pipeline for where each character receives proper formatting, outlining, spacing, and anti-aliasing before it finally gets output to the screen. I spent a good while changing things through trial and error to try to track down the cause of the text problems, resulting in many strange and amusing results along the way. The offending code ended up being the anti-aliasing. It might actually be a bug, but the method the game uses to request anti-aliased characters from the operating system will in some cases return incorrect dimensions for the character, resulting in it being displayed incorrectly in game. By tweaking the anti-aliasing settings, we were able to get the game to display text correctly.

While debugging the text issues and trying to understand the text drawing code through trial and error, a variety of unusual but amusing variations were created.

While debugging the text issues and trying to understand the text drawing code through trial and error, a variety of unusual but amusing variations were created.

Finding out the reason for the game not running on an English OS was not difficult, but it ended up being an interesting problem. In order to look up a file from within the game archives quickly, the archives headers act as a binary search tree for all the files they contained. A binary search tree for the unfamiliar is a method to speed up searching through a list. In this implementation, it would start at the center of the list, compare the file name there to the desired file to see if it was high or low, and then it would keep reducing the list by half until it found the desired file. The benefit is obvious, searching for a file in an archive with 5000 files would only require some 20~ or so comparisons, rather than comparing each one. The trouble here is that alphabetical sorting of Japanese text will actually occur differently depending on the language of your operating system. Since the file search relies on on the comparison between file names to produce the same search results on each system, it would cause it to be unable to find the right file, and ultimately crash.

At first, I had tried to simply get the game to utilize Japanese locale for text sorting. This works, but only if you have Unicode language support, not a valid solution since not all JAST’s customers will have this available.  To resolve the issue, the file search code was re-written to use pre-computed lookup tables that are generated when the game first starts.  It adds a second or two to the game startup time, but it works on all systems and actually provides a small speed boost when loading game assets.

This wasn’t the only compatibility issue we ran into though. Nearing the end of our release window, we caught wind of another major problem: the game would crash under some installs of Windows 8. Why only on windows 8, but not other operating systems? It was very peculiar issue as it wasn’t consistent between different systems, some machines would crash, some wouldn’t, and the crashes occurred at seemingly random locations.

I set up a series of virtual machines to test with, and in one of them I was able to come up with a set of reproducible steps to get the game to crash. The game would end up always crash on the second battle you fought, it didn’t matter which two, crashing immediately after the first round of combat. Using visual c++ remote debugger, I would step through the games execution, but the actual location of the crash in code appeared to occur at different places in the code each time. That’s pretty much the worst case scenario, because rather than the crash being caused by a specific identifiable piece of code, it means that memory was being corrupted somewhere, and there’s no easy way to figure out what’s causing it.

The windows 8 crash bug resulted in more of this than I care to recall.

The windows 8 crash bug resulted in more of this than I care to recall.

Now I said before, but my c++ is somewhat weak, It’s been a very long time since I’ve had to put it to use. But my ability to debug what appeared to be a case of memory corruption in a huge project such as this was considerably worse, in that I had never actually done it before. I had no idea how to proceed. I had asked for some pointers from more experienced acquaintances on the internet, but without looking at the code, they wouldn’t be able to help me.

Lacking any clear directive I simply flailed around disabling systems and code at random until I stumbled on something that would keep it from crashing. And as luck would have it, I did eventually stumble on a clue. It turns out that the game wouldn’t crash if I completely disabled the enemy AI scripts. Well, it would still crash eventually, but it would no longer crash right away in the test case. Having a base to start from, it’s all about working backwards from there to find out why the AI would crash the game.

So a little bit about how Yumina works. A lot of the control code that directs the game flow and story is run from external scripts that are compiled into a bytecode format that’s readable by the game, and that includes the enemy AI. After a whole bunch of back and forth between the game’s code and the AI scripts, I worked out that stripping out variable assignments from the scripts would prevent the immediate crash. See, when the scripts are converted into code readable by the game, each variable gets converted to a numerical id, which is used as an array reference within the game engine. What ends up happening is that on the second battle, the variable references are coming back as garbage, so when it assigns a variable a value in script, it would try to write that to a location that is out of bounds of the original array, causing memory elsewhere to get corrupted and eventually leading the game to crash.

Why was that happening then? Well, it wasn’t obvious right away, but it seems the script file pointers get uninitialized after each battle, but are only properly initialized the first time the battle system is initialized. This means that on subsequent battles, it caused variable indexes to be read incorrectly. It’s somewhat unusual that this occurred only on some Windows 8 installs, but I imagine it has to do with how Windows 8 handles unused memory. The bug is actually present in all versions of the game that were released, but since Windows 8 wasn’t around back then, no one would have noticed. How weird is that?

Making some changes

There were some changes that were made to the game as well, which I think is important to communicate to people. I think it’s all good stuff, but you should probably be aware of it anyways.

Bag management

Here’s one that players of the Japanese version of the game might be familiar with. Yumina has a dungeon crawling aspect to it, where you kill mobs and get loot. You have a limited bag capacity though, but if you want you can take a break and put items into storage to sell later. That is, if you’re willing to painstakingly transfer items one at a time.

You would either be very well acquainted with this screen, or you just wouldn't bother.

You would either be very well acquainted with this screen, or you just wouldn’t bother.

See, the game only had the ability to move items into or out of storage one item at a time. I’m told this issue was resolved for the fandisc version of the game, but that doesn’t help us here. After playing about halfway through the game I just got fed up with how long it took to move items and fired up visual studio, and got coding. Now there’s a batch move option that lets you select and move several stacks of items at once. The interface is similar to the batch discard system, in that you select as many items as you want, and then hit move all. If the move will exceed the cap on the number of each item you can store, it will simply try to move as many items from each stack as possible. You won’t have to suffer the one-at-a-time movement of items that players of the original release had.

Horray! So much easier.

Horray! So much easier.

Dungeon generation/layout

You might want to skip this if you don’t really want to know how some of the internals work. As I mentioned, part of the game involves crawling through dungeons, completing quests, and vanquishing bad dudes. The dungeon floors themselves are fairly small and simplistic, but you go through a lot of them while playing. Some of the key dungeon floors, like the boss floors or bonus floors are hand made, but the majority are randomly generated when you visit them.  After playing for a while, I started to see a bit of a pattern.

Dungeon floor layouts often looked like this.

Dungeon floor layouts often looked like this.

Sometimes like this.

Or maybe something like this.

Maybe they look like this.

Perhaps it looked more like this?

Or if you're really lucky, it might look like this.

If you were really lucky, you might get something unique like this.

Suffice to say, there’s not a lot of variety to the randomly generated maps. Not wanting to give anything away here, but if you’re a completionist aiming to 100% the game, you’ll probably be spending a lot of time in the dungeons.  So why not do some digging, and figure out why they’re so consistent.

The algorithm for generating dungeon maps is actually quite interesting. So you start with a rectangle, containing the entire floor’s maximum size. Then it splits that rectangle in two, and marks the two as ‘connected’. Then each rectangle gets split until each area reaches the minimum size. Then it puts a randomly sized room inside each area, and then connects the connected pairs with a walkway between the two.

Taking a closer look at it though, the maximum map size often only has enough room for four connected rooms. Also, because of the way the splitting occurred, you would always have a connecting path between the top left and top right rooms, as well as with the top left and bottom left rooms. That didn’t leave much room for variation, most of the time you’d end up with the map being roughly in a ‘C’ shape.

After giving it some thought, I made some tweaks to the methodology. I lowered the minimum distance between rooms, as well as making the splits no longer mandatory. Rooms will sometimes also have modifiers, such as pillars, or odd shapes to them. The result is more interesting maps. Just these small tweaks resulted in all kinds of interesting variations. Not being symmetrical helps a lot as well. There’s even a few rare unique floor shapes and layouts that you can find. I’m hoping it will make things more interesting to play. Of course, to maintain balance, on average the amount of tiles and mandatory encounters (ones where there are no way to path around an enemy) should be roughly the same. The goal was to get more interesting maps without affecting gameplay.

Here's an example of a floor generated using the new algorithm.

Here’s an example of a floor generated using the new algorithm.

Balance

I didn’t have any hand in any decisions regarding play balance, but I’ll tell you what I do know. First, the game’s difficulty and leveling curve are unchanged from the Japanese version. The game is known for being challenging, but we felt that people would not want the basic game balance adjusted, and I agree with them.

There was some changes made regarding quests however. In Yumina, they are optional missions that give you a reward on completion. Some of these quests however we felt had very unreasonable requirements. For example, one quest required you to obtain a rare item (5% drop) from a boss that only occurs once on a specific floor (if you’re lucky), meaning you’d have to leave and re-enter the dungeon to take another pass at it. You actually needed to collect multiples of this item as well! You could spend hours and hours grinding trying to complete some of these quests. The game has some facility built in to make it a little easier; the most expensive item in the game is an item that increases item drop rates by 50% for the next encounter (lasts two tiles of movement).

It’s still unreasonable though, so for the English version, quests that require drops from rare monsters will now drop at a much higher chance than normal (50%~100%). That does lessen the value of some of the drop boosting items, but we feel that it’s a positive change to the game experience.

Game bugs

There’s a few random bugs that we stumbled upon that would have been present. They’re not big things, mostly quest requirements, or other subtle things, but we were able to fix a good bunch of them while working on the game. Hopefully, this should result in a better experience overall.

Summary

Anyways, working on Yumina has been a lot of fun. It’s the first project really of it’s kind I’ve had a chance to work on, and as you can imagine, there were all kinds of unique challenges involved. I don’t want to be taking anyones thunder though, the real honor goes to the original project staff that spent a huge amount of time on the translation, editing, working with the thousands of image assets, etc. Still, I hope you liked taking a bit of a peak of what goes behind the scenes, and I really hope you’ll play it once it comes out. It’s a great game!

4 Responses to Yumina the Ethereal

  1. Dark_Shiki says:

    Great post. Mind writing up an explanation of the battle system for those who are curious? This would be helpful for convincing the general RPG crowd. I’ve been asked to do this, but that would require me to load up one of the Japanese trials and dig through it in Japanese…

  2. Raskles says:

    You should post more frequently, you’ve got a lot of interesting information to share 😛

  3. SlashZero says:

    Now that the game is finally out…

    Hats off to you Doddler, for doing what nobody inhouse at JAST probably ever could.

    Especially the batch inventory functions… my god the original sucked in that regard.

  4. LIGHTDX says:

    Ohh, Holy…Man, i 100% completed the game few days ago and i didn’t notice what you did to the inventary system. I moved all the items one by one all the time, i hate me so much Y^Y for that. The change in the dungeon map was really a blessing.

    Thanks for the hard work dude. It was a amazing game and it was even more amazing what you did for everything.

Leave a Reply

Your email address will not be published. Required fields are marked *