Sounds like you are fishing along the rip rap, if possible keep casting out until you find the sweet spot where the gravel meets the bigger rock, fish always feed along that edge, just reel fast with a high tip to get your rigs back.
If your fishing in the real big rocky stuff, try a float as already mentioned, other then that, make sure you have enough weight to hold the bait to bottom and do not allow it to swing in the current, a sang every time
I fish the GMR quite a bit for channels and smallmouth. I made up a sketch for you of how I rig up for cat's. I pre-tie most of this and have 'em ready when needed. I always cast downstream keeping a tight line. Not afraid to throw this anywhere, if I get snagged the weights normally snap off saving the rest of the setup. Hope this helps.