If you seek out English speaking peer groups, then yes learning German will be harder. Luckily(?) for me I am one of only two English speakers in my company, so none of my friends are English speakers. My wife, on the other hand, has an almost entirely Anglo peer group, making picking up German casually near impossible.