The spot/listen bonuses and penalties for this perk seem to apply in interior areas such as caves, buildings, and so on.
I'd like to suggest that the penalties (but perhaps the bonuses, too, if the DMs think it would be more mechanically balanced) shouldn't be applied in interior areas.
It's sort of weird that someone with this perk would be suffering from sun blindness when they're crawling around in the caves of Undermount at noon.
Does Gloom Walker suffer from the same problem, does anyone know?