Deprecating the old pubrules on Aug 1st, 2016

[Denis Ah-Kang]

Hi,

When the 2014 process document was released [1], we agreed on a 2-year
transition period in which WGs could still operate under the 2005
process. Starting from August 1st, all the WGs must use the 2015
process document.
Since we no longer want to invest resources in maintaining the old
pubrules [2], we think the end of the transition period is a good
opportunity to make Specberus [3] the official TR document checker.

The latest version of Specberus is now available at:
https://www.w3.org/pubrules/

That version is still missing a few pieces (docs and rules) and contains
a couple of bugs but we will take care of fixing everything in the next
days.

The major changes you will see are:
- a new UI
- only the process document 2015 supported
- only HTML5 documents supported. Since 2015, only a handful of
documents are still being published in HTML4 or XHTML1.0, and since
supporting these doctypes is too costly in terms of resources, we
prefer to help people migrate their documents to HTML5
- the service getdocrdf (e.g. [4]) being replaced with an API producing
JSON. If you rely on that service, do let me know and I will show you
how to do the migration. Note, tr.rdf remains untouched.

In terms of new features, Specberus provides a REST API [5] you can use
to check your documents (e.g. [6]) or extract their metadata (e.g. [7]).

If you are planning to publish new documents on /TR in the upcoming
weeks, I strongly suggest you to run it through the new pubrules and
report bugs on github [8].

Marcos, Shane and Tab, Echidna is already using Specberus to check
documents before publishing them so I don't expect major problem
for Respec and Bikeshed but let me know if I overlooked something.

As usual, feel free to ping us on irc (#pub) if there's any problem
with the new pubrules.

Denis


[1] https://lists.w3.org/Archives/Member/chairs/2014JulSep/0048.html
[2] https://www.w3.org/2005/07/pubrules
[3] https://github.com/w3c/specberus
[4]
https://www.w3.org/2001/10/getdocrdf?docAddr=https://www.w3.org/TR/2016/WD-reporting-1-20160407/
[5] https://github.com/w3c/specberus/blob/master/README.md#5-rest-api
[6]
https://www.w3.org/pubrules/api/validate?url=http://www.w3.org/TR/2016/WD-charmod-norm-20160407/&profile=auto
[7]
https://www.w3.org/pubrules/api/metadata?url=http://www.w3.org/TR/2016/WD-charmod-norm-20160407/&profile=auto
[8] https://github.com/w3c/specberus/issues

[Shane McCarron]

Just to clarify something...

I assume the policy for document format is that the PRIMARY format must be
HTML5. It remains the case that if a document has alternative versions in
whatever format, those will continue to be permitted. We often include PDF
or EPub versions of Recommendations. Moreover, if we update the RDFa
family of Recommendations again, we would of course include a version of
XHTML+RDFa that is encoded in XHTML+RDFa.

[1] https://www.w3.org/TR/xhtml-rdfa/

Post by Denis Ah-Kang
Hi,
When the 2014 process document was released [1], we agreed on a 2-year
transition period in which WGs could still operate under the 2005
process. Starting from August 1st, all the WGs must use the 2015
process document.
Since we no longer want to invest resources in maintaining the old
pubrules [2], we think the end of the transition period is a good
opportunity to make Specberus [3] the official TR document checker.
https://www.w3.org/pubrules/
That version is still missing a few pieces (docs and rules) and contains
a couple of bugs but we will take care of fixing everything in the next
days.
- a new UI
- only the process document 2015 supported
- only HTML5 documents supported. Since 2015, only a handful of
documents are still being published in HTML4 or XHTML1.0, and since
supporting these doctypes is too costly in terms of resources, we
prefer to help people migrate their documents to HTML5
- the service getdocrdf (e.g. [4]) being replaced with an API producing
JSON. If you rely on that service, do let me know and I will show you
how to do the migration. Note, tr.rdf remains untouched.
In terms of new features, Specberus provides a REST API [5] you can use
to check your documents (e.g. [6]) or extract their metadata (e.g. [7]).
If you are planning to publish new documents on /TR in the upcoming
weeks, I strongly suggest you to run it through the new pubrules and
report bugs on github [8].
Marcos, Shane and Tab, Echidna is already using Specberus to check
documents before publishing them so I don't expect major problem
for Respec and Bikeshed but let me know if I overlooked something.
As usual, feel free to ping us on irc (#pub) if there's any problem
with the new pubrules.
Denis
[1] https://lists.w3.org/Archives/Member/chairs/2014JulSep/0048.html
[2] https://www.w3.org/2005/07/pubrules
[3] https://github.com/w3c/specberus
[4]
https://www.w3.org/2001/10/getdocrdf?docAddr=https://www.w3.org/TR/2016/WD-reporting-1-20160407/
[5] https://github.com/w3c/specberus/blob/master/README.md#5-rest-api
[6]
https://www.w3.org/pubrules/api/validate?url=http://www.w3.org/TR/2016/WD-charmod-norm-20160407/&profile=auto
[7]
https://www.w3.org/pubrules/api/metadata?url=http://www.w3.org/TR/2016/WD-charmod-norm-20160407/&profile=auto
[8] https://github.com/w3c/specberus/issues

[Philippe Le Hegaret]

Post by Shane McCarron
Just to clarify something...
I assume the policy for document format is that the PRIMARY format must
be HTML5. It remains the case that if a document has alternative
versions in whatever format, those will continue to be permitted. We
often include PDF or EPub versions of Recommendations.

Correct. This change doesn't impact alternative versions.

Post by Shane McCarron
Moreover, if we
update the RDFa family of Recommendations again, we would of course
include a version of XHTML+RDFa that is encoded in XHTML+RDFa.
[1] https://www.w3.org/TR/xhtml-rdfa/

The main bottleneck/limitation for us is the validation. If you get your
primary document to validate, then you don't have to worry. If you don't
get it to validate, then it's a different story and we'd need to look at
the specifics. It might be that we could allow the exception and/or
conclude that the validator needs an upgrade. We refrain as much as
possible from granting exceptions because otherwise we might as well
give up on the rule, which would trigger a set of consequences.

Philippe

[Shane McCarron]

Post by Shane McCarron
Moreover, if we

Post by Shane McCarron
update the RDFa family of Recommendations again, we would of course
include a version of XHTML+RDFa that is encoded in XHTML+RDFa.
[1] https://www.w3.org/TR/xhtml-rdfa/

The main bottleneck/limitation for us is the validation. If you get your
primary document to validate, then you don't have to worry. If you don't
get it to validate, then it's a different story and we'd need to look at
the specifics. It might be that we could allow the exception and/or
conclude that the validator needs an upgrade. We refrain as much as
possible from granting exceptions because otherwise we might as well give
up on the rule, which would trigger a set of consequences.

Of course.

I do have a concern about HTML5 extension specifications as well as with
bits of HTML5 that are in WhatWG specs but NOT in W3C specs.

Does the validator have a mode that only permits W3C-approved HTML5?


Shane McCarron
Projects Manager, Spec-Ops

[Philippe Le Hegaret]

Post by Shane McCarron
I do have a concern about HTML5 extension specifications as well as with
bits of HTML5 that are in WhatWG specs but NOT in W3C specs.
Does the validator have a mode that only permits W3C-approved HTML5?

The first concern here is: we want to make sure that the community at
large can read the document.

So, it's not ok to use extensions or markup that are not widely
supported, since it impacts the readers. The validator helps us in that.
We also favor using markup from Recommendation rather than Working
Drafts but again, that's first a readability issue for us. We authorized
the use of HTML5 before it became a recommendation for example (ie it
wasn't "W3C-approved" yet). If your recommendation has some obscure
implementations and readers won't be able to read your specification,
then our advise is for you to use fallbacks as well (see MathML for
example).

If I remember correctly, the validator does have a W3C profile for HTML5.

Philippe

[Michael[tm] Smith]

Post by Shane McCarron
...
Does the validator have a mode that only permits W3C-approved HTML5?

The https://validator.w3.org/nu/ checker implements W3C-conforming checks
in any cases where the W3C HTML fork states requirements that differ from
the upstream HTML spec

—Mike

[Shane McCarron]

Post by Michael[tm] Smith

Archived-At: <
...
Does the validator have a mode that only permits W3C-approved HTML5?

The https://validator.w3.org/nu/ checker implements W3C-conforming checks
in any cases where the W3C HTML fork states requirements that differ from
the upstream HTML spec

Good to know. What about "extensions" that W3C has approved (e.g.,
longdesc) or things in the WhatWG version that are not in the W3C version
(e.g., microdata)?

[Michael[tm] Smith]

Post by Shane McCarron

Post by Michael[tm] Smith

Archived-At: <
...
Does the validator have a mode that only permits W3C-approved HTML5?

The https://validator.w3.org/nu/ checker implements W3C-conforming checks
in any cases where the W3C HTML fork states requirements that differ from
the upstream HTML spec

Good to know. What about "extensions" that W3C has approved (e.g.,
longdesc)

The W3C HTML checker supports longdesc

Post by Shane McCarron
or things in the WhatWG version that are not in the W3C version
(e.g., microdata)?

schema.org supports (and even promotes) the use of microdata, and many
authors and orgs are using it producively. I hope you’re not suggesting it
would be a good idea to start having the W3C HTML checker flag errors in
the millions of schema.org-conforming documents containing microdata that
authors have created based on the schema.org guidelines

—Mike

[Shane McCarron]

Post by Michael[tm] Smith

Post by Shane McCarron
or things in the WhatWG version that are not in the W3C version
(e.g., microdata)?

schema.org supports (and even promotes) the use of microdata, and many
authors and orgs are using it producively. I hope you’re not suggesting it
would be a good idea to start having the W3C HTML checker flag errors in
the millions of schema.org-conforming documents containing microdata that
authors have created based on the schema.org guidelines

Oh - personally I would be happy with that change as I think Microdata is
pretty broken. But no, what I am asking is if the W3C profile that we use
for validating W3C publications restricts the checking to things that are
actually approved by the W3C. W3C recommendations should not be using
microdata - at least not in the primary format that we are checking as part
of pub rules. I imagine there are lots of other things that are getting
thrown into the WhatWG version of HTML that are also not included in HTML
as Recommended by the W3C. We should not be permitting those in our formal
publications either.

[m***@marcosc.com]

Post by Michael[tm] Smith

Post by Shane McCarron
or things in the WhatWG version that are not in the W3C version
(e.g., microdata)?

schema.org supports (and even promotes) the use of microdata, and many
authors and orgs are using it producively. I hope you’re not suggesting it
would be a good idea to start having the W3C HTML checker flag errors in
the millions of schema.org-conforming documents containing microdata that
authors have created based on the schema.org guidelines

Oh - personally I would be happy with that change as I think Microdata is pretty broken. But no, what I am asking is if the W3C profile that we use for validating W3C publications restricts the checking to things that are actually approved by the W3C. W3C recommendations should not be using microdata - at least not in the primary format that we are checking as part of pub rules. I imagine there are lots of other things that are getting thrown into the WhatWG version of HTML that are also not included in HTML as Recommended by the W3C. We should not be permitting those in our formal publications either.

That's silly: we should, as PLH, said, use what browsers can render. We shouldn't use PubRules for petty WHATWG vs W3C HTML battles as that only harms users.

--
Shane McCarron
Projects Manager, Spec-Ops

[Michael[tm] Smith]

Post by Shane McCarron

Post by Michael[tm] Smith
schema.org supports (and even promotes) the use of microdata, and many
authors and orgs are using it producively. I hope you’re not suggesting it
would be a good idea to start having the W3C HTML checker flag errors in
the millions of schema.org-conforming documents containing microdata that
authors have created based on the schema.org guidelines

Oh - personally I would be happy with that change as I think Microdata is
pretty broken.

While I can understand very well might be happy with it personally, I doubt
the tens or hundreds of thousands of authors out there who are using it
productively and are getting the results from it that solve problems for
them would be happy about it in the way you are personally.

Post by Shane McCarron
But no, what I am asking is if the W3C profile that we use
for validating W3C publications restricts the checking to things that are
actually approved by the W3C. W3C recommendations should not be using
microdata - at least not in the primary format that we are checking as part
of pub rules.

Are you aware of any that actually are? Are there any real-world actual
cases of that? And in the long list of problems that we’re here to try to
solve together, is this really one that you want to prioritize?

I work on the W3C HTML checker on my own dime, and on my own time, and your
partisan agitation about this seriously bums me out and de-motivates me
from feeling enthusiastic about working with you on getting other actual
real problems solved together.

But if you want me to flip the bit completely, go ahead and keep it up.

Post by Shane McCarron
I imagine there are lots of other things that are getting
thrown into the WhatWG version of HTML that are also not included in HTML
as Recommended by the W3C.

Yeah? You imagine? Maybe rather than imagining you could find specific cases
of that. Ones that are actually causing any real problems from anybody?

Or better yet, maybe you can use your imagination to think about actual
real problems to spend your time trying to help solve.

—Mike

Post by Shane McCarron
We should not be permitting those in our formal
publications either.

[Shane McCarron]

Post by Michael[tm] Smith

Archived-At: <
But no, what I am asking is if the W3C profile that we use
for validating W3C publications restricts the checking to things that are
actually approved by the W3C. W3C recommendations should not be using
microdata - at least not in the primary format that we are checking as

part

of pub rules.

Are you aware of any that actually are? Are there any real-world actual
cases of that? And in the long list of problems that we’re here to try to
solve together, is this really one that you want to prioritize?

Real-world cases of what? Specs using features that are not part of the
baseline W3C recommendations? I have no idea. I brought this up because
the W3C pubrules have always been super restrictive in what can be included
in a formal publication (at least once it is past the WD stage). Microdata
was an example that is fresh in my mind. I could have mentioned any of the
new elements coming in via extensions or being added / changed in HTML
5.1. It's the same problem.

The stated (and probably reasonable) rationale for these restrictions in
formal publications from the W3C has always been that we must ensure
everyone can access our content. So we cannot rely upon features that are
not broadly supported. That doesn't just mean in the latest user agents.
It also means in assistive technologies, tool chains, translation engines,
search engines, what have you.

There is no way I am saying anything here that should come as a surprise to
anyone who has been involved in making standards.


I work on the W3C HTML checker on my own dime, and on my own time, and your

Post by Michael[tm] Smith
partisan agitation about this seriously bums me out and de-motivates me
from feeling enthusiastic about working with you on getting other actual
real problems solved together.
But if you want me to flip the bit completely, go ahead and keep it up.

Partisan agitation? I am sorry that you feel that way. I was just trying
to understand the scope of the publication rules changes and how they are
going to be enforced.

And fwiw, I also work on this on my own dime.

Post by Michael[tm] Smith

I imagine there are lots of other things that are getting
thrown into the WhatWG version of HTML that are also not included in HTML
as Recommended by the W3C.

Yeah? You imagine? Maybe rather than imagining you could find specific cases
of that. Ones that are actually causing any real problems from anybody?

I don't believe for a minute that any of this nonsense causes any real
problems for anyone (at least outside of ATs). Pubrules has always been a
draconian mess. Basically the bane of my existence for the last 15 years.
Every. Single. Time. I go to publish a spec I run into some crap that makes
me waste hours / days sorting it out.
I am attempting to explore what sort of fresh hell the new changes will
bring. And ensure that the tools and specifications I maintain are in a
good place to help me and others avoid spending too much time there. I am
sorry if exploring that offends you. If you read back you will see that I
was not advocating for anything, nor asking for any changes. I was just
asking how it will work. Because I honestly have no idea.

Post by Michael[tm] Smith
Or better yet, maybe you can use your imagination to think about actual
real problems to spend your time trying to help solve.

Seriously? That was unworthy.

[Philippe Le Hegaret]

Post by Shane McCarron
I was just
trying to understand the scope of the publication rules changes and how
they are going to be enforced.

We're changing from:

[[
All normative representations MUST validate as one of the following:
HTML 4.x, or some version of XHTML or XHTML+RDFa that is a W3C
Recommendation. HTML5 is also permitted with the following limitations:

Inline markup for SVG 1.1 or MathML 2.0 is permitted but only with
a (fallback) alternative.
If the HTML5 validator issues content warnings, the publication
request must include rationale why the warning is not problematic.

Note: Please consider how your content will render in browsers that do
not support HTML5.
]]

to

[[
All normative representations MUST validate as HTML5 with the following
limitations:
Inline markup for MathML is permitted but should use a (fallback)
alternative.
If the HTML5 validator issues content warnings, the publication
request must include rationale why the warning is not problematic.
]]
https://github.com/w3c/specberus/pull/388

This check happens in:
https://github.com/w3c/specberus/blob/master/lib/rules/validation/html.js

I will note that the current XHTML+RDFa recommendation is ok with check:


https://validator.w3.org/nu/?doc=https%3A%2F%2Fwww.w3.org%2FTR%2Fxhtml-rdfa%2F

Philippe

[Shane McCarron]

Post by Philippe Le Hegaret
[[
All normative representations MUST validate as HTML5 with the following
Inline markup for MathML is permitted but should use a (fallback)
alternative.
If the HTML5 validator issues content warnings, the publication request
must include rationale why the warning is not problematic.
]]
Sure. So my follow-up (but non-specific) question was "which profile of

html5"? To which the answer was that there is a W3C profile of html5 in
the validator. Which is great, and I assume what is used. My follow-up
question to that was if that profile includes things that are not part of
the baseline HTML5.

Obviously I can just poke at it to find out. But I was hoping someone more
clueful than I would just have a definitive answer.