I was listening to IskatuMesk's Coffee Hour 8 and realized I never bothered to fix phpBB's inability to nest spoilers inside spoilers either. Granted, this is not a real fix, just a hack, but it's better than noting.
The idea is to create a new [ spoiler2] tag with the exact same settings and code as the original [ spoiler] tag. This lets phpBB nest the two in each other. Quick and easy hack until custom BBcode nesting is fixed in phpBB.