"Linode Manager user passwords are not stored in our database, but their salted and cryptographically hashed representations are."

Sounds like it's not stored in plaintext.

And then "There were occurrences of Lish [the Linode Shell] passwords in clear text in our database."

"There were occurrences of Lish passwords in clear text in our database."

Which is entirely different from "all passwords were stored in plain text". We don't know whether there were 1, 100 or a majority of such passwords. It's a bad oversight in each case, but the latter is quite a bit worse than the former.

Right, we don't know because they failed to disclose this.

Wouldn't you rather have them working on assessing, containing and repairing the damage rather than catering to the impatient internet crowd that is so used to immediate updates on everything that they can't fathom putting together a responsible, useful, correct response might actually take a while? Honestly, these people don't understand what it takes to run a company and handle such an incident. As far as i'm concerned, the indignant vocal minority could shove it while I was working on resolving the problem for the majority of customers that exactly that from us.

Quoting Linode:

"There were occurrences of Lish passwords in clear text in our database. We have corrected this issue and have invalidated all affected Lish passwords effective immediately."

Presumably, they know how many customers are affected, because they invalidated their passwords. They didn't tell us.

I don't understand where you are coming from here.

Why do you think companies can only do one thing at a time ? And where have you worked where software engineers are drafting press releases or conducting security audits ? And why shouldn't keeping customers (who have their own apps) informed be their top priority ?

  Why do you think companies can only do one thing at a time?

Amdahl's law. Recovering from a catastrophic hack isn't parallellizable. Nobody can draft a press release before the security audit has been performed. You can't inform anyone before the security audit has been performed. The software engineers can't go to work before certain parts of the security audits have been performed. Certain parts of the security audit can't be performed before other parts have been performed.

  And where have you worked where software engineers are 
  drafting press releases or conducting security audits?

A bootstrapped startup? Depending on the scale of the company (and I have no clue how many people work at Linode), the work may be sequential simply by lack of manpower.

I believe the correct approach for us is to assume all Lish password where in clear text until a good explanation is given. It's not usual to store some password in clear text and encrypt the others, right?

Could be that the passwords were hashed, but something logged a password, or there was leakage somewhere during the login process.

All it takes is one debug log statement to leak through from development and you can be logging plain-text passwords on every login.

Storing passwords in clear text is definitely not normal.

Also the social engineering implications could be huge. Many people use the same password across various sites so in theory there could be a lot of compromised email accounts. Which could then mean compromised internet banking sites, trading sites etc.

Bear in mind this can be as simple as a handful of people emailing their LISH password to support while asking if it is correct etc.

Or someone mistakenly logging "Bad LISH Password: 'FOO' for user 'BAR'" somewhere that logs to the DB.

Or users accidentally typing their password in the username field.

Nevertheless, this isn't the first time Linode has needed to spend their benefit-of-the-doubt points.

Also (and correct me if my understanding of this is wrong since I've never used it), LISH is just a remote terminal service. You still need to know the VM's root password. And frankly, if you're doing it right, the root password should be a long, unique random string that you store somewhere safe and never use since you should be using keys to login to your box on a day to day basis.

It's quite likely that someone has left root logged in on hvc0 (the LISH console) on their Linode, while logging out of LISH. Probably more than one person.

Also, access to LISH even without a root password provides access to some scrollback output, which could expose sensitive information.

Also, LISH allows multiple connections, all of which see the same console, so the attacker can just connect to it and wait for a root prompt to appear when root logs in next. (Does changing the LISH password prevent this attack if they're already connected? I doubt it.)

Also, most distribution boot processes can be messed with at boot from the console. For example, you can ctrl-c to stop important daemons from loading. In some cases you may be able to get a shell without the root password.

Also, Magic SysRq can be accessed over a serial console by sending a BRK. You do not need to be logged in to do this, and it could be used to kill processes, reboot, etc. I don't know if LISH allows sending BRK.

Yeah but most of these are generic problems with providing a remote serial console (excluding LISH allowing multiple shared connections to one console, which is obviously bad).

They've now reset all the passwords and fixed the bug that meant some of them were being stored without encryption. So what we're saying is there was window of a week or so where only LISH users affected by the clear text bug may have been open to an attack if they happened to use LISH during this time frame and the attacker was targeting them. Not great, I agree, but could have been worse.

Hopefully, they'll change their setup going forward so that each LISH connection to the same VM gets its own console.

Shoot, you are correct. Definitely an attack vector.

The connection is to the same TTY when I log in via the web terminal and SSH at the same time, so if you are logged in via LISH anyone with the LISH password has access to the logged in user's console + scrollback.

I wasn't able to send BRK from Putty though.

I'm pretty sure this is correct. The LISH passwords just give you console access via ssh (as an alternative to the web-based terminal in Linode Admin), which is essentially the equivalent of getting to an ssh prompt. A login/password would still be required to access the machine.

It doesn't appear that you can reboot or enter single user mode from the LISH console login prompt.