Discussion:
[XeTeX] Primitive parity, \expanded and \Ucharcat
Joseph Wright
2018-05-03 21:38:27 UTC
Permalink
Hello all,

In adding features to expl3, the LaTeX team have been making use of a
variety of 'new' (post-e-TeX) 'utility' primitives in various engines.
Almost always these originate in pdfTeX and have migrated to other
engines, but are not in any way tied to PDF output, etc. Depending on
the exact engine in use, some or all of these primitives may be
unavailable, and that then limits macro-level features.

It seems sensible long-term to have cross-engine feature stay 'in sync'
with each other. In particular, (u)pTeX has picked up a number of pdfTeX
features, meaning that XeTeX often is the most 'limited' engine. The
team would like, if possible, to have a common feature set in all
engines in this regard. At the same time, there are a few 'bits and
pieces' that make sense to raise at the same time. I'll lay out the
various areas below.

Doing the work here is non-trivial, but luckily there is an automated
build system available via GitHub which is allowing us (me/David
Carlisle) to do some testing. I'm building up patches in various
branches at https://github.com/josephwright/texlive-source: assuming
these look good, I'll merge them as required and send diff files to
where/whoever is best. The branches on GitHub should hopefully have
clear names for what they address.

The areas we are keen to look at are as follows:

- 'pdfutils': (u)pTeX has picked up a number of pdfTeX primitives, and
a subset have made their way into XeTeX too. However, XeTeX is still
missing several, most notably an expandable RNG. We are part-way
though working out patches to add the rest to XeTeX (RNG is done,
file data and timer to do)

- banners: pdfTeX and LuaTeX have \<engine>banner, other engines lack
that. The banner includes TeX version and details of the TeX system,
so is potentially useful. Adding this to (u)pTeX/XeTeX looks
straight forward: still to-do.

- \expanded: This was slated for pdfTeX 1.50 but that has never
appeared, but the primitive is useful as it allows 'function-like'
expandable macros. We can see this begin very useful for simplifying
some macro code, and in many ways it feels like an e-TeX primitive.
The GitHub expanded branch adds it to pdfTeX/XeTeX/(u)pTeX

- Allowing \Ucharcat (XeTeX) to make \active tokens: this was raised
recently on the XeTeX list, but fits here as we've put a branch
together to show it works

It's likely I'll finish the outstanding patches by the weekend. Note
that at present each feature addition is in a separate Git branch, so to
add all of them I'll have to do a little tidying up: that will happen
once I know which of these suggestions are useful.

Feedback most welcome.

Joseph


--------------------------------------------------
Subscriptions, Archive, and List information, etc.:
http://tug.org/mailman/listinfo/xetex
Norbert Preining
2018-05-04 07:43:44 UTC
Permalink
Hi Joseph,
It seems sensible long-term to have cross-engine feature stay 'in sync' with
Very much agreed upon that.
I'll merge them as required and send diff files to where/whoever is best.
For (u)ptex a pull request at https://github.com/texjporg/tex-jp-build
might be useful. This is the main development area for all Japanese TeX
engine stuff.
them I'll have to do a little tidying up: that will happen once I know which
of these suggestions are useful.
I think all are fine.

Best

Norbert

--
PREINING Norbert http://www.preining.info
Accelia Inc. + JAIST + TeX Live + Debian Developer
GPG: 0x860CDC13 fp: F7D8 A928 26E3 16A1 9FA0 ACF0 6CAC A448 860C DC13


--------------------------------------------------
Subscriptions, Archive, and List information, etc.:
http://tug.org/mailman/listinfo/xetex
Joseph Wright
2018-05-04 08:31:14 UTC
Permalink
Hello Norbert,
Post by Norbert Preining
I'll merge them as required and send diff files to where/whoever is best.
For (u)ptex a pull request at https://github.com/texjporg/tex-jp-build
might be useful. This is the main development area for all Japanese TeX
engine stuff.
I see that the \expanded code has already been picked up in a branch
there: https://github.com/texjporg/tex-jp-build/tree/expanded. So I'm
guessing that is likely to happen.

At the moment what I'm aiming to do is get everything in one place so it
can be reviewed, etc., and commented on. Particularly in the case of
\expanded, the wider plan only works if there is general
(pdfTeX/XeTeX/(u)pTeX) agreement on taking the patch.

Once we have that agreement, putting in pull requests, diffs, etc.
against the 'right' places should be easy enough (at least in the sense
I'm happy to sort it).
Post by Norbert Preining
them I'll have to do a little tidying up: that will happen once I know which
of these suggestions are useful.
I think all are fine.
I don't imagine there is anything particularly controversial, but there
is also the technical business (I'm no WEB expert).

Joseph


--------------------------------------------------
Subscriptions, Archive, and List information, etc.:
http://tug.org/mailman/listinfo/xetex
Joseph Wright
2018-05-04 09:02:52 UTC
Permalink
Post by Joseph Wright
- 'pdfutils': (u)pTeX has picked up a number of pdfTeX primitives, and
  a subset have made their way into XeTeX too. However, XeTeX is still
  missing several, most notably an expandable RNG. We are part-way
  though working out patches to add the rest to XeTeX (RNG is done,
  file data and timer to do)
To be clear, the full set of primitives here is

- \pdfrandomseed
- \pdfsetrandomseed
- \pdfuniformdeviate
- \pdfnormaldeviate

- \pdfresettimer
- \pdfelapsedtime

- \pdffilesize
- \pdffilemoddate
- \pdffiledump
- \pdfcreationdate

of which the first set is done (a working branch for XeTeX).

Joseph


--------------------------------------------------
Subscriptions, Archive, and List information, etc.:
http://tug.org/mailman/
Hironori KITAGAWA
2018-05-05 07:09:48 UTC
Permalink
Hello Joseph and all,
Post by Joseph Wright
- banners: pdfTeX and LuaTeX have \<engine>banner, other engines lack
that. The banner includes TeX version and details of the TeX system,
so is potentially useful. Adding this to (u)pTeX/XeTeX looks
straight forward: still to-do.
I made a branch for \eptexbanner for e-(u)pTeX as
https://github.com/texjporg/tex-jp-build/tree/banner.
I think it would be more convenient to use a common name for these "banner"
primitives such as \enginebanner, rather than different name for each engine
(\pdftexbanner, \xetexbanner etc.).

Regards
Hironori KITAGAWA
<***@yahoo.co.jp>


--------------------------------------------------
Subscriptions, Archive, and List information, etc.:
http://tug.org/mailman/listinfo/xetex
Jonathan Kew
2018-05-13 12:36:35 UTC
Permalink
Post by Joseph Wright
Hello all,
Modulo any issues that show up in testing, all of the above is now
done and on my GitHub fork
(https://github.com/josephwright/texlive-source/tree/Ucharcat: this
branch has 'all the stuff' on it).
I know that https://github.com/texjporg/tex-jp-build already has a
branch for \expanded. What's the best way to request 'officially'
that the changes go into pdfTeX/XeTeX? I can send a .diff to the
pdfTeX dev list, and put in a pull request on SourceForge for XeTeX,
if that's best.
Thanks for working on these things, Joseph.
For xetex, a pull request would be the best approach, I think; or if
it's feasible to do separate PRs for each feature, that would probably
make reviewing and tracking the changes easier.
Is there documentation of the added features available somewhere, so
we can more accurately understand what we're thinking of adding? Thanks!
JK
Excellent: I'll start on putting something together later today.
Do you want all PRs against master or can they be 'chained'? Adding
primitives, it's easier if you are working knowing which others have
been created.
"Chained" should be fine, I expect; I doubt there'd be any reason we'll
want to take a later one but decide against an earlier one.

At which point perhaps a single PR is just as good, as long each feature
is a separate commit so that it comes in manageable chunks. From a quick
glance at your fork, it looks like that's how it would naturally appear.
So, feel free to do whichever seems easiest for you.

JK


--------------------------------------------------
Subscriptions, Archive, and List information, etc.:
http://tug.org/mailman/listinfo/xetex
Joseph Wright
2018-06-18 20:28:17 UTC
Permalink
Post by Jonathan Kew
Post by Joseph Wright
Hello all,
Modulo any issues that show up in testing, all of the above is now
done and on my GitHub fork
(https://github.com/josephwright/texlive-source/tree/Ucharcat: this
branch has 'all the stuff' on it).
I know that https://github.com/texjporg/tex-jp-build already has a
branch for \expanded. What's the best way to request 'officially'
that the changes go into pdfTeX/XeTeX? I can send a .diff to the
pdfTeX dev list, and put in a pull request on SourceForge for XeTeX,
if that's best.
Thanks for working on these things, Joseph.
For xetex, a pull request would be the best approach, I think; or if
it's feasible to do separate PRs for each feature, that would
probably make reviewing and tracking the changes easier.
Is there documentation of the added features available somewhere, so
we can more accurately understand what we're thinking of adding? Thanks!
JK
Excellent: I'll start on putting something together later today.
Do you want all PRs against master or can they be 'chained'? Adding
primitives, it's easier if you are working knowing which others have
been created.
"Chained" should be fine, I expect; I doubt there'd be any reason we'll
want to take a later one but decide against an earlier one.
At which point perhaps a single PR is just as good, as long each feature
is a separate commit so that it comes in manageable chunks. From a quick
glance at your fork, it looks like that's how it would naturally appear.
So, feel free to do whichever seems easiest for you.
JK
Hello Jonathan,

Have you been able to look at my merge requests? We are now moving to
using \expanded for other engines: it's going into pdfTeX and (u)pTeX,
and is already in LuaTeX. Ideally, we'd like to avoid XeTeX being 'left
behind'.

Joseph


--------------------------------------------------
Subscriptions, Archive, and List information, etc.:
http://tug.org/mailman/listinfo/xetex
Jonathan Kew
2018-06-18 20:47:45 UTC
Permalink
Post by Joseph Wright
Post by Jonathan Kew
Post by Joseph Wright
Hello all,
Modulo any issues that show up in testing, all of the above is now
done and on my GitHub fork
(https://github.com/josephwright/texlive-source/tree/Ucharcat: this
branch has 'all the stuff' on it).
I know that https://github.com/texjporg/tex-jp-build already has a
branch for \expanded. What's the best way to request 'officially'
that the changes go into pdfTeX/XeTeX? I can send a .diff to the
pdfTeX dev list, and put in a pull request on SourceForge for
XeTeX, if that's best.
Thanks for working on these things, Joseph.
For xetex, a pull request would be the best approach, I think; or if
it's feasible to do separate PRs for each feature, that would
probably make reviewing and tracking the changes easier.
Is there documentation of the added features available somewhere, so
we can more accurately understand what we're thinking of adding? Thanks!
JK
Excellent: I'll start on putting something together later today.
Do you want all PRs against master or can they be 'chained'? Adding
primitives, it's easier if you are working knowing which others have
been created.
"Chained" should be fine, I expect; I doubt there'd be any reason
we'll want to take a later one but decide against an earlier one.
At which point perhaps a single PR is just as good, as long each
feature is a separate commit so that it comes in manageable chunks.
From a quick glance at your fork, it looks like that's how it would
naturally appear. So, feel free to do whichever seems easiest for you.
JK
Hello Jonathan,
Have you been able to look at my merge requests? We are now moving to
using \expanded for other engines: it's going into pdfTeX and (u)pTeX,
and is already in LuaTeX. Ideally, we'd like to avoid XeTeX being 'left
behind'.
Joseph
Hi Joseph,

Sorry for the slowness at this end, I know it's been a while... Copying
Post by Joseph Wright
Sorry, I just got home from a work trip overseas, and haven't had a
chance to look into this in any detail yet. (I suspect it's probably
fine, but would like to make sure I understand what's being added.)
Can you point me at documentation of the new primitives (particularly
\expanded), to help understand exactly how it's supposed to work?
(Apologies if this has been posted somewhere already, and I've missed
it...)
Thanks,

JK


--------------------------------------------------
Subscriptions, Archive, and List information, etc.:
http://tug.org/mailman/listinfo/xetex

Loading...