So somehow, James thought it would be a good idea to go up to the kitchen, get the markers out of the craft drawers, go back downstairs and color the steps along the way. These were the particularly bad spots...there were more, in other colors.
This was after water blotting with a towel.
I still need to try rubbing alcohol, but haven't been too worried about it since we are planning on steam cleaning the basement soon anyway (read: hopefully before the baby is born). Did you know that Crayola has a section of their website dedicated specifically to how to clean up their products from different types of surfaces? In case you need it: http://www.crayola.com/canwehelp/staintips/index.cfm