In Profundis Progress (6/30)

Ah, progress! Basic support for larger tiles is in now. In fact, I might make the tiles much larger but just display scaled-down versions during the game, which would allow for smooth scaling of the playfield between scales. I used a trick like that fscaling smooth-scaling map screen in Mayflight (a particular favorite effect from that game). I'm not sure I can do this with Pygame however -- one of the reasons I tried moving to pyglet is it can blit images while scaling them in transit. Pygame only scales through the creation of a new surface, which must then be blitted to the destination, which might be too much overhead for me.

One benefit of using larger tiles is the game doesn't have to blit so many images, which is worth another modest framerate increase. The disadvantage, of course, is that less of the world fits on the screen.

In design news, I've started planning out in detail how the random substance properties will work. There are planned to be three kinds of substances: liquids, gases and solids. Liquids are flowing materials like, but not limited to, water. Sand is also a "liquid," for instance. Gases hang in the air and slowly spread out. Both may or may not end up having pressure support, if I can figure out a relatively inexpensive implementation (which is one of the reasons I obsessed over finding ways to speed the game up for a while). The third type, solids, are basically stone walls, which are more reactive in their properties.

So what do I mean by a random substance? The idea is that, at the start of the game, in addition to substances with fairly obvious attributes, there would also be some with unknown properties that must be deduced through observation and experimentation. From the promo video, this is like throwing a torch into a pool of unknown liquid to see if it's flammable. My idea is that the first world the player explores in a campaign would have little or no random substances, but each one after that would have a little more randomness. Another idea, which I mentioned in an interview, is that the planets explored by the player would be divided into solar systems, and each system's would have its own "elements." So, different planets would have different maps and layouts of substances, but the behaviors of those substances would be the same throughout.

If I do this right it could be one of the most interesting aspects of the gameplay. I find that many of my favorite roguelikes are the ones with randomized equipment, which turns figuring out what your stuff does into a logic game. However, it is also true with many of those roguelikes that the identification game isn't as interesting as it could be, or is only a real requirement in the early phases of the game, as once the player learns what everything is there is nothing left to learn.

Anyway, all this is still some time off in the future. I'm going to work on the graphics a bit and fix up the platforming engine before then, I think.

In Profundis Progress (6/29)

Current work is devoted to increasing the tile size and consolidating the tiles into atlases on disk, for ease of editing. Along the way I'm putting in the beginning of different stone types, which will have random properties like gases and liquids.

In Profundis Progress (6/27)

A bit of real progress!  Profiling revealed that, anomalously, most of the processor time was being taked up by one particular lambda.  I've avoided lambdas in the code mostly, partly because I knew the various machine code compilers don't tend to optimize them well.  It didn't help at all that this lambda was being called in the coordinate sort function as a key.  Profiling revealed that this lambda was being called millions of times over the course of a short run of the platform engine.

I replaced it with a class method, and the result is a good 4-5 fps increase.  Further optimization is needed I think, but it's a solid improvement!

In Profundis Progress

So I made a backup copy of the code and spent an hour or two trying out Cython.

Cython (which is different from CPython) is another compiler for Python.  Unlike Psyco or PyPy, it isn't a JIT compiler; it turns Python code into C, and it works on the module level, which requires some elementary reorganization to use.  Not only does it promise some basic speed increase just from the conversion, it is possible to supply some explicit type information to optimize the resulting code still further.  Since that type info is added to the language in the form of new keywords, that has the disadvantage of making the code no longer runnable as pure Python.

The big surprise with Cython is: I got it to work.  There are no Python features in my code that it doesn't support.  This, by itself, is a surprise, and after my latest failures in getting outside code working it gives me a warm feeling to use one of these tools and not be faced with any error messages that I can't fix myself.

Unfortunately, I didn't get any observed speed benefit from it; the framerate actually dipped a little bit.  Still, the relative lack of impossible-to-fix errors is encouraging, and I might play around with this a bit more later.

In the meantime, the thing I can do to improve performance the most is probably to either optimize the cell sort or switch to a different cell calculation selection mechanism.  That's what I'll be working on tomorrow.

In Profundis (6/26, 2)

Profiling has revealed that it's not so much the sort that's taking all the time as the lambda function that determines distance from the player's location.  Hmmm.

In Profundis Progress

Actual work has been light the last few days, as I investigate a couple more possibilities towards speeding it up: eliminating the sort portion of the update loop (mentioned last night), checking into Cython and PyPy in terms of compilers, and trying to get Python's profiling tools to obey me -- I find they are not as simple to use as the documentation on the Python site implies.

In Profundis Design (6/26)

Recently the queue for processing cells changed.  Before it was a simple nested pair of for loops that went through every cell in a rectangular region slightly larger than the screen.  After it uses a substantially more complicated system, where the changed cells from the previous frame are added into a list, the list is sorted by the distance to the player's location, and the first few hundred closest cells are handled, with potential changes from those cells added back into the main list.

This is good for some things, and bad for others.  It makes the preliminary world simulation (two of the generation steps) much faster, and means, if there's nothing much happening near the player, we automatically simulate progressively further areas of the map.

However there is a problem, and the more experienced programmers probably already see what it is.  It's that sort.  Before the change it didn't matter much how large the world was; if it was 2x2 or 100x100, it took the same time to calculate.  The sort operates on every change-flagged space on the map, and it makes a function call for every flagged cell on every frame.  That's roughly 7,000 cell coordinates in a newly-generated terrain world, every frame.

You can actually see for yourself, a bit, how much time the sort takes up.  When you start playing in the prototype, if you turn on the debugging HUD you'll be told the framerate.  On my low-end, single-core laptop, I generally get framerates in the mid-to-high 20s.  The simulation tends to find an equilibrium over time though, and as water pools fewer nearby cells are calculated each frame, meaning the range that gets calculated expands, ultimately decreasing the size of that sort list.  The result is, over time, framerates slowly increase.

This is my primary development concern at the moment.  I'd like to increase framerates by a bit more so that I have more processor time to do effects (maybe related to making fluids less blocky), increase the complexity of the simulation, and perhaps draw a background layer.  Right now I'm getting a handle on Python's profiling tools to make sure my suspicion is correct, but even if it turns out not to be THAT bad I can't help but think I could design this better.  Maybe going back to the rectangle, but increasing its size as visible activity decreases.

In Profundis stuff

$10-and-up backers have had messages sent out informing them of how they can obtain a "playable" copy of the prototype I made the videos from.

More research is being done while I work on the game, in the continued hopes that I can break the code free from reliance on Psyco, a dead-end in terms of Python development.

In Profundis Question

Talking it over with Simon a bit, this business with chasing C++ may be taking too long.  I may go ahead and develop the Python version some more, do some releases of that, and see if it ends up being good enough.  In the meantime I'll read up on C++ and try to get the kinks out of my head concerning it.

The thing about that I'm most worried about is it might turn out to be a dead-end.  A reliance on Psyco almost certainly is; it's been dead in the water since 2009, and hasn't had an official release since 2007.  Going with its successor PyPy is perhaps possible, but I've so far been unable to get Pygame working under PyPy.  I'm pretty sure it works since PyPy has an important profiling tool that uses Pygame for graphics output, but that site is weirdly lacking on details for installing it.  The Windows installer for Pygame refuses to recognize PyPy as Python 2.7 and easy_install seems to work at first but in interactive mode refuses to import it.

Any of you have experience with PyPy?  I apologize this is all taking so long.  I never promised you an instantaneous rose garden, although I dare say you deserve something soon.

In Profundis Progress (6/22)

Did I make a mistake in picking C++?  I know it sounds like I'm dithering between languages, but coming to it from something as luxurious as Python makes C++ seem really... what's the word I'm looking for... Aspergery?

In any case, the development timeframe for this thing is such that I don't think I can become proficient enough in C++ quickly enough to make effective use of it.  But fast-enough Python code relies on either using an out-of-date library (Psyco ceased development with Python 2.6) or a special interpreter which is probably not compatible with the breadth of Python modules I require (pypy).  Further, packaging Python into executables seems to be generally hacky.

I just want to code in peace, and I can't seem to write anything right the first try in C++.  Someone in comments a while back suggested Java.  Maybe I should consider that.

In Profundis Progress (6/21)

Still trying to muddle through.  I have a question for you more experienced C++ programmers.

The Python version is implemented procedurally.  That is my way of saying, there are multiple loops.  In this way, program state (title screen, menus, gameplay, edit mode, etc.) is basically kept in the instruction counter.

It's a comfortable way to program, basically how I would implement things back on the old Commodore 64, but is hard to convert to an event-driven paradigm, which is why the pyglet and, maybe, the C++ versions are bothersome to develop.  (Well that and the fact that C++ is maddeningly finicky.)  I'm trying to design this one using a master event loop that switches off depending on the program state.  Which is not how I ordinarily would do this, but it seems to be what is expected from a more professional program, and when it comes time to make an iOS version would potentially pay off there.

Or will it?  I'm making assumptions that might not be borne out by practical considerations.  It is an alien way for me to code, a method that requires a bit more forethought than I'm used to.  In the past, I've coded in a kind of iterative way: I know generally where I'm going, I think out what will be necessary, then I try to get a bit of the way each time.  This seems like something more architectural.

Or is it?  Does this make sense to any of you?  Is there a better way to do this that I haven't considered?

In Profundis Progress (6/18)

I continue to bash my head bloody against the brick wall of C++.  I have an annoying problem where I try to define a variable in a header, and other files that include that header complain about multiple definition errors despite the presence of a header guard.  This is annoying for me since one design pattern I frequently use is to use one file as a place to store essential global values, like gravity and friction values, for easy reference/changing later, but if I just use that file for those variables the linker balks.

I asked a programmer friend for help and his suggestion managed to resolve the problem (basically to move the values into a different source file), but the reason he gave that it works uses a development metaphor that is nonsensical to me, I don't grasp its context, and so I'm resigned, for the time being at least, to doing it just because it works, not because I understand it, and that makes me nervous.

As for the stuff I'm implementing, I'm working on the data structures used for holding graphics data right now.  Fun!

In Profundis Progress (6/17, #2)

I hate C++ a little less.  It still has much to atone for, however.

In Profundis "progress" (6/17)

Grar, I hate C++!

In Profundis progress (6/15)

After fixing the sight bug, I broke ground on the C++ port.  Getting in the way is the fact that although I'm generally familiar with C-style languages I've never done a major project in it or C++ before, meaning I'm going through that phase where every file has 20 different errors, sometimes due to a problem halfway up the page from where the compiler caught it.  In other words: uphill climb.  Still, going faster than I expected.

I am considering releasing the Python version as a curiosity for the alpha-level backers to mess around with.  It is kind of fun to plop down liquids and sand in edit mode, but it looks real primitive and isn't a real game at this point.  What do you guys think?

In Profundis Progress (6/13)

The source has been reverted back to the Pygame version.  The spectre of performance continues to hang over my head though.  I am beginning to plan the switch to C++.

Back to normal design issues, the switch a couple weeks ago to keeping a cell change list broke vision, since visibility propagation was handled through the CA engine.  Today I worked on fixing that.  Another bug has come to light (ha ha) in the process, which is causing some out-of-vision cells to show incorrectly. Fixing that is the immediate objective.

In Profundis Progress

Ah, the problem that has been holding me up for almost a solid week now has been tracked down.  It was a lot of sweat to discover this fact so I'm going to use very large text.  It is this:

PYGLET IS NOT COMPATIBLE WITH PSYCO

When I tried attaching keypress events to the window they became unreliable.  Removing the call to the psyco compiler causes them to become reliable again.

I feel like I've wasted a week but no matter, I know there isn't some bizarre phantom bug in my code now. 

Oh, if you only knew the effort I spent on this.  The energy I've poured into this.  On looking at every part of my code, of creating test situations, of slowly going between the test situations and my code point for point and finding where the differed, of creating MORE tests.

I feel like curling up in a ball on the floor right now.  But at least the problem has been solved.  In Profundis is special in that it's not just the display that's the biggest processor cost, the program does some heavy calculation behind the scenes.  This means I can't abandon psyco, and that means I must abandon pyglet.


So long, pyglet

In Profundis creaks along

It's been a while now and I still don't have a lot visible to show for it.  Honestly, the switch to pyglet has been going more slowly than I expected.  The thing is, to get the most performance out of it I have to redesign the main loop to be entirely event-driven, and that requires learning pyglet's callback system, the examples of which most frequently use decorator, which are proving unfriendly to circular imports....

Bleaah.  It is coming along, it's annoying frustrating work but it's happening.  And I'm doing it for more reasons than just performance -- being fully event driven will make the design of the thing a lot more professional, and make it easier to adapt to mobile platforms, which mostly use entirely event-driven paradigms anyway.  And pyglet is capable of some special effects, which might help me out in ways related to the oft-commented blockiness of the water effects.

So it's coming along.  I figured you all deserved a status report regarding the game.  This is getting reposted on Kickstarter so it'll go out to everyone.  You should also all know that the game has a placeholder website not at http://www.inprofundis.com/ -- it's just a barebones Wordpress thing at the moment.  I'm going to continue to use the Coin Door Interlock blog for now though for development reports, though.

More news as it happens....

In profundis Progress (6/1)

More boring stuff.  The pyglet changeover continues.

Someone in comments in the last one asked if I was sure that changing to pyglet would be a net performance improvement.  I don't know for sure; he is right, the reputation I have of it is largely anecdotal.

However it does look like, as far as rendering goes, pyglet is rather more flexible.  Or at least the way I was doing it before was less flexible.  Either way, I'm taking the opportunity to change the rendering system a bit.  Which is, again, not much to show.  (Well this is a development diary, heh.  A lot of development, especially in the early phases, looks like this.)