It works as intended. You have 5 monsters occupying all main monster zones. The effect of Subterror Fiendess is simultaneous (and if you do). Hence, even after the Trap Monster is flipped face-down and returned to the S/T, the monster zone is still considered occupied at the time of the activation. Hence, there is no unoccupied main monster zone for the Special Summon of the Subterror from hand/GY.
Please also refer to Rokket Tracer vs Dilingerous Dragon and similar rulings here: