I opened up the keyboard (instructions from the last time I did this) and found the board controlling the output jacks. It looks like this:

The big solder bits are where the two jacks connect. There are six pins. From looking at the traces (sorry about the poor focus) I could see that the middle two were connected together and the top one is grounded. Testing with a multimeter set to resistance, the middle two pins were normally connected to the ground, but disconnected when a cable was inserted. Looking into the jack, I think maybe the end of the cable pushes apart two bits of metal to break a connection. If we could short circut this detection circut, so that it always read "connected" then the internal speakers would stay on.

I considered short circuting both detection loops and then adding an externally accessible switch to disconnect them, but I decided against that because of the inconvenience of cutting a hole for the switch and mounting it. Instead I decided to modify only one of the jacks so that I would later be able to choose whether to leave the internal speakers on by plugging into one jack or the other. Some soldering later and we had:

It worked exactly as I had hoped. I closed everything back up and made sure it worked before putting in all the screws. Then I painted the modified jack with some white-out. Total time: 3hr.

Now to see how well the internal speakers do at a dance!

Update 2012-05-29: This is working pretty well. You still want a monitor so you can hear yourself, but it's now loud enough that if you can't get a personal monitor with just you as a supplement to the overall monitor mix it's not that bad.