The tower should be a pile of boulders in thousands of years, especially since it is strongly jointed, and freeze-thaw weathering should be aggressive.
Rapid erosion during Flood runoff is a more straightforward explanation.
During the Retreating Stage, ostensibly erosion-resistant rocks were eroded and transported great distances.
I have personally studied the main mountain ranges of the United States, and since the Flood was global, I expect similar patterns of long-distance transport of resistant rocks elsewhere.
Oard Floods often leave behind erosional remnants (figure 1).
Many erosional remnants were left behind during the catastrophic Lake Missoula flood, such as Steamboat rock at the upper end of Upper Grand Coulee (figure 2) and Umatilla Rock, which splits the plunge pools seen at Dry Falls, Washington, USA.
There could occasionally be so many fine-grained sediments mixed in with the coarse gravel that the flow would have been a type of mass flow, such as a turbidity current, debris flow, or hyperconcentrated flow.
The long-distance transport of rocks has been well documented in the United States.