Jonathan Wilkins

I got some great responses, so I thought I'd summarize for anyone who didn't read through all the comments. Any mistakes are likely mine.

The suggestions I had were predominately to use a mild dish soap, like Ivory and to be sure that it's diluted. (juliesimone and industrial_grrl)
From reading several latex care notes, the big thing to watch for is any dish soap that contains any sort of grease cutter. Those are known to destroy latex.

An alternative was liquid Neutrogena (rubberbondage)

eddie_offermann suggested bar soap (like Ivory), but tonyawinter said that she'd ruined clothing with this approach.

eddie_offermann also identified the cause of all of the confusion. Apparently what's generally referred to as latex is actually any of a number of types of synthetic extruded rubber such as neoprene, nitrile or EPDM and each of these has a durability rating (durometer). So each of these has different care needed and no clothing manufacturers are labelling their outfits with the exact type and hardness of latex, so you're generally in the dark. He pointed to http://www.quickcutgasket.com/materials.html for info on some of the various types of rubber and their vulnerabilities.

For now I think I'm going to stick with a thorough rinse in water (with a super diluted Ivory dish soap if needed) and hang drying in a dark, cool place.