Normally crushed materials don't work well because the shape of the particles. Sand is smoothed by years of waves, crushed rock is jagged. This leads to issues with viscosity.

A direct example: railroad rocks. Rail tracks are laid on very well engineered rocks, that are designed to lock together under the weight of a train. They have a roughly 1x1x2 cross sectional ratio, specifically for that purpose. You don't want that happening when you pour concrete.