Most of my In Profundis work this week has been in research. Here's what I'm looking into.
There are benefits and drawbacks to using Python as a development language. The biggest drawbacks are performance (although that's not as bad as you'd think) and something called the GIL, or Global Interpreter Lock.
A feature of many dynamic languages (Ruby has one too), the GIL is something that ensures that, even in multithreaded code, only one thread can execute at a time. On a single-core machine this is not so bad since the code is stuck doing this, more or less, anyway, but it means we won't get all the expected performance out of multicore machines that we might expect. (Note: I'm not currently sure if Pygame's rendering respects the GIL.) This is done to simplify the behind-the-scenes details of Python. There are exceptions (especially regarding I/O), but for the most part threading seems like of a convenience feature than something one would turn to to improve speed.
But all is not lost, there are other solutions. One is to use the module multiprocessing, which offers a way around the GIL. Another is to use a version of Python that solves the GIL problem; both Jython and IronPython allow this, although they don't interface to Pygame to my knowledge. And then there's to optimize the cellular loop using something like Numpy and/or Cython.
Other notes:
Discarding Psyco means some hard choices I had to go with in earlier versions, especially concerning Python version, aren't so hard anymore. This opens things up a bit concerning other modules, although I still can't go to Python 3.X because Pygame requires 2.X.
In the last message someone expressed concern that calculating the whole world would make the game unresponsive. This is actually not necessarily the case, as the calculation loop is written in such a way that on a given "frame" it can just calculate part of the world, remembering where it left off to continue later. So, I can calculate one-fifth of the world this frame, get player input, then another fifth the next frame, and so on. It queues cells up in a spiral pattern from around the player's location, and the visible screen is the beginning of each pass of the world, so we don't even have to worry about visible calculation artifacts. Neat, huh? In the future it would be nice to calculate different, non-adjacent areas of the world in different threads to make use of multicore systems -- maybe the multiprocessing module can help with that.
Sunday, March 25, 2012
Tuesday, March 20, 2012
State of the project: 3/20/11
This is rambly and questionably edited, but I wanted to get something out there about the project and what's up with it.
There hasn't been a lot of comment here recently because I've been thinking long and hard about the design, and about what I'm happy with about the direction In Profundis is going, and what I'm not so happy about.
Because of this, a semi-radical redirection of the project is underway. Many computers are not fast enough to run an automation of the size of the game's world as an action game unless I only simulate a zone around the player, and I've been unhappy with my efforts to make a platforming engine. As Miyamoto has been known to say, an idea is something that solves two problems at once, and I think I have the solution: to make In Profundis a quasi-real-time, turn-based game.
So how will this work? Well, first off, character movement won't be on a pixel level. Instead, like in roguelikes, the character will move one cell at a time. There will probably be an animation between cells and some concessions to playability, but you won't be able to stop "between" spaces.
The game will attempt to simulate the entire field each frame. This will greatly reduce framerates, but will mean that the engine can support one of the features I consider to be essential: continuously flowing water between points. If I don't simulate the whole field then liquids will tend to accumulate at the edges of the execution frame, meaning flows like waterfalls and rivers will eventually cease if one end of the flow is outside the frame. Obsessive thinking about the problem has identified no solutions that themselves won't cause bigger problems, so I've come to the conclusion that cheating this isn't the answer.
This means to a degree actually abandoning the platforming physics engine, but I'm okay with it as it has taken a disproportionately large amount of development time to implement, and it's still pretty bad. By simplifying the physics here, I can avoid some of the annoying edge cases that result in frequent character wall embeds, or at least handle them more elegantly.
However if you sit and do nothing, I'm thinking that game time will not freeze, although it may slow down. Not only does this still provide some time pressure, but it also makes it more obvious what direction flows are going without having to add new visual elements to identify them.
What will probably happen is I'll run the world simulation and character turns on different threads. One problem with this, however, is that Python threading is a odd. To my understanding, due to the existence of a thing called the Global Interpreter Lock, threads aren't like real operating system threads, but cannot execute more than one process at a time, even if Python is running on a multiprocessor system. (There is a module to get around this, multiprocessing, and there are versions of Python that either remove the GIL or make it less onerous, but I'm not sure how compatible those are with Pygame.)
But one good thing about this is, since the game won't be so dependent on processor speed, we'll be able to abandon Psyco. Honestly it has been the source of many technical problems. It works great as a drop-in solution that just "works" to make Python faster... so long as all your modules work with it (at least one has mysteriously broken in the past when Psyco's been enabled, which took some time to discover)... and so long as you avoid esoteric language features like generators... and don't use Python versions after 2.6. I've at last reached the point where I'm ready to declare it's just more trouble than it's worth. Fortunately, it's just as easy to remove as it was to add.
As an aside, I am thinking hard about the nature of fluids. The game can currently simulate up to eight kinds of fluids with random properties, but that might be a case of too much randomness. It might be best to stick with water, sand, and a couple of others in each world.
There hasn't been a lot of comment here recently because I've been thinking long and hard about the design, and about what I'm happy with about the direction In Profundis is going, and what I'm not so happy about.
Because of this, a semi-radical redirection of the project is underway. Many computers are not fast enough to run an automation of the size of the game's world as an action game unless I only simulate a zone around the player, and I've been unhappy with my efforts to make a platforming engine. As Miyamoto has been known to say, an idea is something that solves two problems at once, and I think I have the solution: to make In Profundis a quasi-real-time, turn-based game.
So how will this work? Well, first off, character movement won't be on a pixel level. Instead, like in roguelikes, the character will move one cell at a time. There will probably be an animation between cells and some concessions to playability, but you won't be able to stop "between" spaces.
The game will attempt to simulate the entire field each frame. This will greatly reduce framerates, but will mean that the engine can support one of the features I consider to be essential: continuously flowing water between points. If I don't simulate the whole field then liquids will tend to accumulate at the edges of the execution frame, meaning flows like waterfalls and rivers will eventually cease if one end of the flow is outside the frame. Obsessive thinking about the problem has identified no solutions that themselves won't cause bigger problems, so I've come to the conclusion that cheating this isn't the answer.
This means to a degree actually abandoning the platforming physics engine, but I'm okay with it as it has taken a disproportionately large amount of development time to implement, and it's still pretty bad. By simplifying the physics here, I can avoid some of the annoying edge cases that result in frequent character wall embeds, or at least handle them more elegantly.
However if you sit and do nothing, I'm thinking that game time will not freeze, although it may slow down. Not only does this still provide some time pressure, but it also makes it more obvious what direction flows are going without having to add new visual elements to identify them.
What will probably happen is I'll run the world simulation and character turns on different threads. One problem with this, however, is that Python threading is a odd. To my understanding, due to the existence of a thing called the Global Interpreter Lock, threads aren't like real operating system threads, but cannot execute more than one process at a time, even if Python is running on a multiprocessor system. (There is a module to get around this, multiprocessing, and there are versions of Python that either remove the GIL or make it less onerous, but I'm not sure how compatible those are with Pygame.)
But one good thing about this is, since the game won't be so dependent on processor speed, we'll be able to abandon Psyco. Honestly it has been the source of many technical problems. It works great as a drop-in solution that just "works" to make Python faster... so long as all your modules work with it (at least one has mysteriously broken in the past when Psyco's been enabled, which took some time to discover)... and so long as you avoid esoteric language features like generators... and don't use Python versions after 2.6. I've at last reached the point where I'm ready to declare it's just more trouble than it's worth. Fortunately, it's just as easy to remove as it was to add.
As an aside, I am thinking hard about the nature of fluids. The game can currently simulate up to eight kinds of fluids with random properties, but that might be a case of too much randomness. It might be best to stick with water, sand, and a couple of others in each world.
Monday, February 13, 2012
I exist....
I'm still banging around. Unfortunately there are other necessary things going on right now, but I'm still here. I realize that it's been slow going, and it gets embarrassing to say "it's been slow" too often. We're approaching ten months since the Kickstarter project made, and I realize that's pretty sorry. There is no need to remind me of that fact.
I have gotten some inventory stuff and tool stuff put in. The big thing that's been worrying me so far is that I'm thinking random fluids, as they're currently implemented, may not be as interesting as I had projected. It's not that random properties are bad, but when you have several kinds of fluids all mixing in a big basin, I'm not sure they interact in ways that are too clean.
Graphics are another thing that's been worrying me lately. They're okay in a kind of makeshift kind of way, but I haven't really spent much time on making sure the stone looks nice. I'm not talking about photorealistic nice, but in a wasn't-knocked-up-in-thirty-minutes-in-MSPaint nice.
Well, those are the things on my mind concerning the project at the moment. More before long, I expect.
Friday, December 30, 2011
In Profundis design (12/30)
So, here is how tools that have a visible extent in the game world should work. Maybe someone can suggest something if they spot a potential pitfall. This is the system used for ropes and spanners, and will probably be extended for other things.
The tool itself exists in one cell of the world. That cell "is" the tool for purposes of recollecting and destroying. A rope hanging from a piton "is" the cell containing the combined object. Spannners, portable, extendable platforms (not what are sometimes called wrenches -- maybe I should find a better name) have a central point from which they extend, and that point "is" the spanner. Make sense?
Once placed, "shadow" objects get placed through the CA system. I call them shadows because they are "cast" by the primary objects, but to the CA they can still have interactions.
On the primary object's turn, it checks in the spread directions (down for rope, left and right for spannners) and sees if it's empty and suitable for spreading. If so, it places the appropriate object (rope tile, extended platform for spannners) in the place. It also sets a "strength" value on the tool.
When a shadow object's turn comes up, it first checks in the direction to grew from, to see if a matching source object or stronger matching shadow object is there. If it is, and has a strength greater than 1, it tries to spread itself, and if it can produces a clone of itself at one point less strength. If for some reason it doesn't have a matching object, it removes itself from the game world, the "source of the shadow" has gone away, it isn't casting anymore.
Advantages to this system:
- Quick. The processing required for each cell remains light. While tools will probably not become very common objects in the game world, it's important that, if the player were for some reason to buy lots of them and string rope everywhere, the game engine could handle it. This avoids a particular Minecraft problem; there, liquid is handled in a similar source-and-shadow manner, but when a source tile is removed, on slower systems the computer becomes very laggy for a while, while dependent fluids are recalculated. (When I shut off or on the water flow in my multi-story mob grinder, which has many forks, long cascades of liquid, and collecting streams at the bottom leading to the output trough, it's several seconds before I can do anything else.)
- Self-correcting. Every shadow object checks for a source every turn; if the source disappears for some unexpected reason, it does to, and so thus do objects that depend on it.
- Fun to watch. Multi-tile objects don't appear instantly, but over a short period of time as the shadows propagate through he world.
- Has a certain elegance.
- Respects 'c', which is important for avoiding subtle CA calculation problems. 'c', or the "speed of light," is a bit of cellular automation jargon -- it refers to the maximum rate at which changes may propagate through the system. In Profundis' speed of light is one tile/frame. At the moment, no tile can affect the game world more than one immediate space in each direction. If placing a rope instantly placed all the rope tiles into the world, it would break the speed of light. Currently, In Profundis calculates its world in a square spiral pattern emitting from the player's location, skipping every other square, but starting on an even or odd square in successive turns. The result is that a checkerboard-like layout of tiles is calculated every turn. Since no tile can produce and effect that extends out further than one tile orthogonality, we can be sure that changes aren't propagated too fast, producing results like instantly-falling water. This doesn't matter so much for example tools, but might be important for other tools later.
Disadvantages:
- Inherently tile-based. I was worried before about the game's strict grid-based presentation; I've brainstormed ways to effectively implement sloped rock tiles that still handle water effectively for instance. I've spent too long trying to figure that one out however. The project has gone on long enough, and I think now I'm not going to worry about the limitations of the grid-based CA. It certainly hasn't hurt Minecraft any.
- May result in circular, self-dependent shadow objects if those objects aren't designed well. Could a situation arises where multiple-placed primary objects interact in a way that a self-reinforcing system of shadows could set up? Not in the simple situations I have described, I think, but more complex objects might have branching shadows, and if somehow a shadow object can produce another shadow of greater strength than itself this situation could arise, where all sourcing primary objects could be removed but surviving shadows keep each other refreshed.
- Has a certain lack of realism. Imagine a case where a spanner platform extends several cells out into the air, and a boulder falls through the middle of it. When it falls through the platform (spanners aren't strong enough to support such weight) the next platform over will no longer have a source, and will vanish the next frame. But the boulder will then continue to fall, and so the spanner it destroyed will be recast the next frame. The result is a one-tile gap that moves down the spanner like a wave, a behavior that isn't mirrored in real life. Similarly, what if gravity affects spanner sources? If the source is unsupported, it will effectively create a V-shaped platform shadow through the world as it falls, as it created new platforms around it as it descends, while those left behind it by its fall both spread and disappear outward through the route from where the spanner passed.
Still though, I think the mechanism solves more problems than it causes, especially if I keep tool designs under control, and it may be applicable to simulating non-tool objects as well. Remember, it's always best to keep CA effects as simple as possible; the more work you do on each cell, the more likely the whole simulation may, in some situations, start to drag.
The tool itself exists in one cell of the world. That cell "is" the tool for purposes of recollecting and destroying. A rope hanging from a piton "is" the cell containing the combined object. Spannners, portable, extendable platforms (not what are sometimes called wrenches -- maybe I should find a better name) have a central point from which they extend, and that point "is" the spanner. Make sense?
Once placed, "shadow" objects get placed through the CA system. I call them shadows because they are "cast" by the primary objects, but to the CA they can still have interactions.
On the primary object's turn, it checks in the spread directions (down for rope, left and right for spannners) and sees if it's empty and suitable for spreading. If so, it places the appropriate object (rope tile, extended platform for spannners) in the place. It also sets a "strength" value on the tool.
When a shadow object's turn comes up, it first checks in the direction to grew from, to see if a matching source object or stronger matching shadow object is there. If it is, and has a strength greater than 1, it tries to spread itself, and if it can produces a clone of itself at one point less strength. If for some reason it doesn't have a matching object, it removes itself from the game world, the "source of the shadow" has gone away, it isn't casting anymore.
Advantages to this system:
- Quick. The processing required for each cell remains light. While tools will probably not become very common objects in the game world, it's important that, if the player were for some reason to buy lots of them and string rope everywhere, the game engine could handle it. This avoids a particular Minecraft problem; there, liquid is handled in a similar source-and-shadow manner, but when a source tile is removed, on slower systems the computer becomes very laggy for a while, while dependent fluids are recalculated. (When I shut off or on the water flow in my multi-story mob grinder, which has many forks, long cascades of liquid, and collecting streams at the bottom leading to the output trough, it's several seconds before I can do anything else.)
- Self-correcting. Every shadow object checks for a source every turn; if the source disappears for some unexpected reason, it does to, and so thus do objects that depend on it.
- Fun to watch. Multi-tile objects don't appear instantly, but over a short period of time as the shadows propagate through he world.
- Has a certain elegance.
- Respects 'c', which is important for avoiding subtle CA calculation problems. 'c', or the "speed of light," is a bit of cellular automation jargon -- it refers to the maximum rate at which changes may propagate through the system. In Profundis' speed of light is one tile/frame. At the moment, no tile can affect the game world more than one immediate space in each direction. If placing a rope instantly placed all the rope tiles into the world, it would break the speed of light. Currently, In Profundis calculates its world in a square spiral pattern emitting from the player's location, skipping every other square, but starting on an even or odd square in successive turns. The result is that a checkerboard-like layout of tiles is calculated every turn. Since no tile can produce and effect that extends out further than one tile orthogonality, we can be sure that changes aren't propagated too fast, producing results like instantly-falling water. This doesn't matter so much for example tools, but might be important for other tools later.
Disadvantages:
- Inherently tile-based. I was worried before about the game's strict grid-based presentation; I've brainstormed ways to effectively implement sloped rock tiles that still handle water effectively for instance. I've spent too long trying to figure that one out however. The project has gone on long enough, and I think now I'm not going to worry about the limitations of the grid-based CA. It certainly hasn't hurt Minecraft any.
- May result in circular, self-dependent shadow objects if those objects aren't designed well. Could a situation arises where multiple-placed primary objects interact in a way that a self-reinforcing system of shadows could set up? Not in the simple situations I have described, I think, but more complex objects might have branching shadows, and if somehow a shadow object can produce another shadow of greater strength than itself this situation could arise, where all sourcing primary objects could be removed but surviving shadows keep each other refreshed.
- Has a certain lack of realism. Imagine a case where a spanner platform extends several cells out into the air, and a boulder falls through the middle of it. When it falls through the platform (spanners aren't strong enough to support such weight) the next platform over will no longer have a source, and will vanish the next frame. But the boulder will then continue to fall, and so the spanner it destroyed will be recast the next frame. The result is a one-tile gap that moves down the spanner like a wave, a behavior that isn't mirrored in real life. Similarly, what if gravity affects spanner sources? If the source is unsupported, it will effectively create a V-shaped platform shadow through the world as it falls, as it created new platforms around it as it descends, while those left behind it by its fall both spread and disappear outward through the route from where the spanner passed.
Still though, I think the mechanism solves more problems than it causes, especially if I keep tool designs under control, and it may be applicable to simulating non-tool objects as well. Remember, it's always best to keep CA effects as simple as possible; the more work you do on each cell, the more likely the whole simulation may, in some situations, start to drag.
Thursday, December 29, 2011
In Profundis progress
Yep, I'm still alive, although right now I don't quite feel so definitely on the "alive" side, due to a cold that's lingered around a few days.
It's been a hectic month overall. Distractions abound. I presented my final paper for my Master's degree and it went very well. A cousin unexpectedly died in a motorcycle accident. And I've been trying to work in bits on a prototype for a friend's game.
I haven't forgotten about In Profundis though. Hell no.
Saturday, November 19, 2011
In Profundis Progress: Tools, Inventory and personal coding deficiencies
This is about the place where my general lack of experience with modern (that is, post Commodore) development is biting me hard.
Something I've been worried about is the representations of user-placed items in the world. There are three general possibilities here, and I've messed around with each one to some degree.
Possibility one is to implement them as sprites, that is, as moveable objects that can move independently of the cellular grid, like the player can. This is what I tried first, but after a bit of that I realized that it really wasn't a good solution. Part of what makes tools interesting is how they can interact with the game world, and every object that isn't part of the basic world model, even if it's grid-aligned, complicates the code.
However, it gives rise to the problem with possibility two, which is, define special tiles for in-world tool objects. It looks weird when water flows around a piton as if it was a full-block object. And a lot of tools are like this. It's the existence of liquids and gases, which are just cheat analogues for sub-grid-level phenomena, that makes this an issue.
The nature of the game is such that, if push came to shove, I could probably get away with doing it that way anyway, but instead I'm opting for possibility three, which is to give each cell a tool field that can contain its own object. This way liquid can flow past a piton without overwriting it or flowing around it, and objects in lakes can rest there without making an air bubble around them.
Now the better programmers out there are probably wondering why I'm spilling words on such a decision. It's because, really, this is the most complicated project I've ever worked on. Mayflight's world builder was more complex, but its world simulation was just a platforming engine. I'm somewhat out of my depth here. At least writing this in Python means I don't have to muck around with (overt) pointers and worry about the myriad bugs those can introduce.
I visit anna anthropy's blog pretty frequently, and I've been known to chuckle to myself when I read about her aversion to coding. I really can't imagine doing a whole game using, say, Game Maker's abhorrent drag-and-drop scripting functionality -- I always tie each event to a text script. But really, I'm just one step further along the path than she is, and there's a dozen ahead of me.
What is funny is that I was a quick study back in programming classes, but there's something about pointers that screws me over. It's not that I don't understand them -- far from it, I have a strong processor-level understanding of the things. I did a fair bit of assembly programming back on the C64. But something about the way C-style languages approach them trips me up hard. I'm constantly confusing where I need to use * and where I need to use &.
I know that * is a pointer to some value, and & is the value that a pointer points to, but I don't seem to have it internalized to the point where I can write code with those symbols confidently.
I wouldn't be surprised if I had some form of dyslexia tripping me up there. The language is such that you must know what is a value and what is a pointer to a value, but I nearly always get them mixed up. This, coupled with the fact that I was continually stymied by import errors, was probably what sabotaged my attempts at restarting In Profundis in C++. Game Maker is really not that different from C in syntax, but its garbage-collected, reference-counted scripting was the breakthrough feature that got me productive with it.
I'm just blabbing here really; I felt guilty that it had been a little while since I made a post here. Anyway, working in inventory and tool simulation now. I hope to start putting in the backer treasure data before long, so that'll be fun at least.
Monday, November 7, 2011
In Profunds ideas: Flags
So here's something big that I'm playing around with....
In Profundis gives you a limited inventory for carrying around equipment. You tend to use up resources while you're exploring the game world, placing pitons, hanging ropes, and so on. Eventually you have to travel back to base and resupply.
A concern that I have for the game is that this travel back and forth, between base and the frontier of exploration, could get annoying, or worse, boring. But the resource and time costs of getting around the game world are the primary obstacle to exploration. How do I reconcile the two?
My idea is a special inventory item: flags.
After you explore for a while and figure it's time to return to base to resupply/get some rest/buy new stuff, what you can do is plant a flag at a place you'd like to return to later.
After this is done, the challenge is this: return to base without using any other items in your inventory, in order to "establish" the flag. It's like a mini-game in a way. It is okay to use items you've already placed, so you can sort of pave the way for your return trip. If you're forced to use an inventory item the only thing lost is the establishment run. In that case the flag remains where it is, and you can try it again just by returning to it.
The reason this has to be done is because the game is recording how long it takes the player to do it, in order to figure out a realistic travel time, and using resources along the way that might not be available later would spoil the experiment. (It's possible that this might just be "shallow travel" equipment, like jetpacks, rather than things that change the game world to make it easier to traverse, like pitons and spanners.)
After you've established the route, you can later "instantly" warp between base and the flag. Instantly is in quotes because this is only immediate to the player -- the game clock is set ahead by the amount of time it took to make the trip. Since the expedition is charged rocketship rental by the amount of time spent on the planet, this is a substantive cost, but it's your character who has to pay it, not you. The character is still making the trip, in effect, on autopilot. (In fact, I'd like there to be a "montage" of still shots of the trip, that takes a few seconds to play out, whenever warping.)
Continuity of the game world is very important I think. The problem with teleporters is that they basically make a mockery of the spatial extent of the world. I consider this bad because, as the player makes longer and longer expeditions, the increasing time required to get from base to the exploration frontier is, itself, an increase in difficulty. Exploring the other side of the map takes more time, which incurs more spaceship rental costs, which means the player can't afford to waste as much time.
Some of you are no doubt reading this and thinking something along the lines of: WTF? I think I can understand that. But I look at it this way: I've played hundreds, if not thousands, of video and computer games, and I've never known a game to do something like this. Maybe there's a reason for that. But it's a very interesting game mechanic to think about, and I think that will transfer to play. What do you think?
Subscribe to:
Posts (Atom)