The rate of growth is immaterial if the die-off is because of a permanent change, which there is every reason to believe is the case (based on every model of climate change, and it will only get worse).

Also, the rest of the trophic web that the kelp forms the basis for is hosed. Since they are one of the most productive ecosystems, it's kind of a big deal.

There are different types of kelp, not all of which are as sensitive to water temperature. For example, SoCal's kelp forests were not as devasted as NorCal's kelp forest. The biggest concern for SoCal's kelp forests are controlling the urchin population. Moreover, kelp populations are easily restored through cuttings--an entire forest can be repopulated in less than a decade because kelp grow so quickly.

Kelp die-off is a problem, but it's a temporary and addressable problem.