Can you be more specific on why you think the t-test is invalid when the data is non-normal? Duration data may be non-normal, but given finite visits and a reasonable number of visits, sampling for the mean surely is.
You are right that in the presence of sufficient data, everything converges to the normal and we're fine.
However the difference between the t-test and the normal to which it is an approximation is that the t-test takes into account corrections the rate of convergence to the estimated normal. The details of that convergence is dependent upon the distribution of individual samples, and the t-test is predicated on the assumption that each and every sample is from a normal distribution.
