That's how all space debris tracking works. Luckily orbits are rather predictable paths, but there's still atmospheric drag, lunar and solar gravity, non-uniform Earth gravity, solar photon pressure and measurement imprecision to deal with.