Resolved -
About 1 in 300 messages were getting stuck in the streaming pipeline. These messages were not lost but became stuck, due to a bad node in Kubernetes. This node was replace and the data pipeline is returning to normal now.
Mar 17, 22:08 PDT