Baseline H.264 gets you I- and P-frames, which is enough to prove a decoder works. B-frames are where the standard stops being polite. They reference pictures both before and after themselves in display order, which means decode order and display order diverge and you have to keep a reordering buffer.
The hard part was the reference picture list construction in §8.2.4. It is a small algorithm with a large number of edge cases, and getting the initial ordering wrong produces video that looks almost right — which is worse than video that looks broken.
Once the lists were correct, motion compensation for bidirectional macroblocks fell into place, and main-profile clips decoded frame-for-frame against the reference. 4K preview now plays in real time straight onto GPU textures.
Reading the standard twice was, once again, faster than guessing once.