I'm confused. How are you supposed to have a conformance test without first having a spec? To what would the implementations be conforming, if not a spec?

Sorry, What I meant to really say is specs are not sufficient. Sure specs are important. But even with specs, without comprehensive conformance tests implementations will diverge.

I work on the WebGL spec and tests based off the OpenGL ES spec and tests. Even though the OpenGL ES spec is long and detailed, every edge case for which there is no test is broken or different in one driver or another.

In an ideal world, the spec would be the test. A spec is just a human readable test. A test is just a machine readable spec.

It's entirely possible to build a conformance test suite around an existing 'black box' of code and accept that whatever it does, even stuff that might be considered a bug, is what any compliant implementation should also do.

(You could also fix some of the more egregious bugs, and then expect implementations to follow your new behaviour)

> Specs are important but I'd argue a conformance test is equally or even more important.

I agree, but the article mentions tests prominently. They didn't miss that out.

Here are the tests we wrote for HTML-to-Markdown conversion. Could be useful when switched the other way around :-)

https://github.com/bergie/to-markdown/blob/master/test/tests...

Also worth looking at the tests for Text::Markdown which is the canonical version of Markdown on CPAN.

https://metacpan.org/source/BOBTFISH/Text-Markdown-1.000031/...

> Without, conformance tests different interpretations of a spec lead to divergent behavior.

I think you mean either:

> Without conformance tests incorrect interpretations of a spec lead to divergent behavior

Or

> Without conformance tests different interpretations of an ambiguous spec lead to divergent behavior.

Conformance tests should test specific behavior defined in the spec. Tests will never (or rarely) prove spec conformance but spec conformance should always prove test conformance.

I have yet to encounter a spec that was not in some way ambiguous.

1 + 1 = 2 Is fairly unambiguous until you introduce operator overloading (and why i despise overloading).

> 1 + 1 = 2 Is fairly unambiguous until you introduce operator overloading (and why i despise overloading).

It is only unambiguous if you specify the numbering system you are using ;)

</pedantic>

Is there actually a notation and + operator for which 1 + 1 = 2 is ambiguous? Only being semi-rhetorical, I'd be interested to know if there is one. (Modular arithmetic is not considered ambiguous, at least not in the math neck of the woods.)

Not really answering your question, but more to the point of the discussion in my opinion, a() + b() could be ambiguous if a() and b() are functions with side effects and the evaluation order of the +-operator is not specified.

1 + 1 = 10b is what he's saying.

Yes, that's what I meant - even "1 + 1 = 2" can be ambiguous if no context is provided.

> 1 + 1 = 2 Is fairly unambiguous until you introduce operator overloading (and why i despise overloading).

What is it specifying—or, perhaps a better question, what would an implementation look like? It might be more precise to say "I have yet to encounter an implemented spec that was not in some way ambiguous."

1 + 1 != 2 if the 1s are magnitudes of vectors - if the directions are opposite, then they can be 0. In fact, 0 <= 1 + 1 <= 2 in that system.

That's not how addition works. You're either adding the vectors or you're adding the magnitudes. You don't get to sleight of hand by writing one and adding the other.

Overloading is a feature of natural language though, and therefore completely unavoidable in writing a spec.

Hee hee, your interpretation of his sentence is introducing ambiguities :)