This also removes the dispatch onto main when adding the finished buffers back into the cache. This operation should be fine to run on any thread as long as it's inside the sync block.
