It's only negligible if the rock is a negligible amount of extra weight. Otherwise, the energy to transport will still come from somewhere & not be negligible unless all that energy is from a non-carbon fuel source (bulk transportation is nowhere near decarbonized).