Solution? New rhythm game using quarter notes, quarter rests, and eighth notes. Once students get good at the game I'll add in other harder rhythms and hopefully get us back onto track. I call this new rhythm game Rhythmic Concentration. It was based off of a music concentration that many of you probably have heard about.
Here is how it works:

Everyone in the class sits in a circle on the floor.

Each student is given a basic rhythm that was written on an index card. (I used only quarter notes, quarter rests, and eighth notes, however you could add in other rhythms your students know)
I then put a piece of tape or something to mark the 'beginning/end' of the circle. The person on the right of the marker is #1. The goal of the game is to be #1 at the end of the game.
The #1 person claps their rhythm and then #2 claps their rhythm immediately after. This continues all the way around the circle. During the first round, I normally just let them practice their rhythm. However, after that it gets harder.
Students continue clapping their rhythms as they continue around the circle. When someone misses their turn or hesitates that student has to go to the end (to the left of the marker). Their rhythm stays where it is and then students fill in the spaces. For example: if #5 of #10 misses their turn #10 shifts to #9; #9 shifts to #8; #8 shifts to #7; #7 shifts to #6; #6 shifts to #5; and #5 shifts to #10!
For those visual learners like myself, a model of the switch!:

After that it gets even more difficult. I start sending people to the end of the circle for clapping the wrong rhythm, not doing it in beat, and I have even added the metronome for some of my older students and they have to stay in beat.

My students really seem to like it. What I like most about it is it is fast paced, so those that get frustrated with waiting for everyone to get a turn don't feel that way during this game. I also like it because they are practicing their rhythms without feeling like they are practicing rhythms!

If you try this out, please let me know how it goes or any changes you made to make the game run smoother! Also feel free to ask me questions about it, I'll do my best to answer them. Have fun!