I had a bright idea. What if we didn't draw every tile every frame? What if we just scrolled the image from the last frame, and drew in tiles that changed, those scrolling in from the edge, and those around actors?
Today I tried this out. And you know what?
It really didn't make all that much difference, while it cluttered up the code with drawing flags and checks to see if the tile scrolled in. I think the scrolling, even though handled by SDL, end up taking almost as much time as it took to just draw the tiles anew each frame. D'oh!