See @wwylele's comment on PR 4483 for more details on what causes this behavior.
Hardware testing indicated that FFFFFFFC is the correct mask for all audio formats (mono and stereo PCM8, mono and stereo PCM16, and ADPCM). This fixes broken audio in Luigi's Mansion: Dark Moon and a few other games.