This post is nice: the writer first explains a problem, using a simple example. In the next section, they reflect a bit about the problem, and then they casually mention two tools they built. In my opinion, this is amazing: you sponsor you project, while also making the problem it solves clear: use their tool to test how portable your code is
Sure, but the pojt here is that if we say "Write in X" we generally understand it to mean "Treat X like a standard and don't get too colloquial with the stylings."
Pedantry is worthwhile, but it can be a diminishing returns game.
On the example of 'echo \n' - it's not defined in POSIX, therefore a script written in "POSIX shell" must simply never hit that case.
TFA kinda implies you can't target POSIX shell. That's silly, of course you can. The question is, what tools are there to check for compliance. Whether running on 14 shells is a good such tool - idk. Something specifically searching for POSIX violations might be better.
Well, with C language it's pretty much the same. You are supposed to "just" never write (or rather, most of the time, to just not execute) anything that is UB. And lots and lots of people to this day continue to believe that can do this (most of the time, they're wrong).
`local` for example is present in many shells (almost all of them), but they decided to leave it out uniquely because of ksh93 (scope is different). It became undefined behavior.
When the spec was written, ksh was important. Since then, it has only been revised but not updated and I consider it to be obsolete.
So, if you follow POSIX strictly, you then lose local scope on functions, which is more likely to cause bugs and hard to catch with a linter like you suggested. You're left with a broken feature set (on many other angles too) that is not actually practical. Even spellcheck makes concessions.
The simple fact that POSIX doesn't define any long options for basically any CLI tools just makes me outright ignore it when writing scripts.
I use long options in scripts so that you can actually understand which options are used. Short options are for interactive shells since those commands nobody is really gonna look at again.
That is another can of worms. For now, I'm worried only about the shell interpreter and builtins.
`echo` in most shells is not `/bin/echo`, it's actually an internal implementation (external echo means forking at every output, which is super slow).
The portability of other cor utils is also a clown fiesta, and most likely the best solution for now is "use gnu", which sometimes feels like "best viewed in internet explorer"-era shinenigans.
I don’t get this point either. If local is not in the POSIX spec, I guess you can’t use it if you want to be POSIX compatible. Just because many shells do it doesn’t mean it’s POSIX.
You can target POSIX if you want to, but doing that doesn't guarantee shell scripts will work.
The blog post stresses this, the difference between POSIX and portability.
If you want portability, testing is better for now. One of the goals of these projects is to more precisely capture a retrospec (what actually works, not what was specified), it's the same thing they did with HTML5.
because the problem here is educating people with slipshod ideas about 'sh' being 'the POSIX shell' or (worse) 'the Bourne shell'. Both M. Chazelas and M. Gaigalas are making the point that 'sh' is a language that one aims to write in, not one of the many programs that sort of, sometimes, if invoked in the right way, implement that language; and a subordinate point that people are generally very poor about doing that when yet they insist that they are writing 'POSIX shell script'.
Fun facts: Standardization led by existing practice is not a simple process.
POSIX/SUS standardization is not a static thing. The long discussed thorny issue of echo has now subtly changed from all of those explanations given about it over the years, because in 2024 the standard was changed.
M. Chazelas's own quite famous 2013 StackExchange answer on the subject has not yet been updated with the change that now incorporates -e and -E into the rule. Amusingly, it was M. Chazelas that raised defect 1222 that caused this change.
Stéphane (not "M") Chazelas is a genius. I have written him several times over the past 30 years, encouraging him to write a book about shell, POSIX, and other topics that he is an expert in. He writes very well too. I fear we will all be the poorer if/when he retires.
"M." is the French abbreviation for "monsieur", the equivalent of "Mr." in English. So "M. Chazelas" is the correct way to politely refer to Stéphane Chazelas by his last name, since he's French. (Not that writing "Mr. Chazalas" would be wrong either when writing in English, but "M. Chazelas" is entirely correct).
That is an unambiguous program that works in every shell. In a well-written shell program, the only things that should follow a single backslash in a double quoted string are
\ " ` $
(Although I found that this option is only on in ysh, not in shopt --set strict:all ... arguably that should be changed)
That seems to be be an entirely-different question - `echo "c:\\new"` still differs in behavior between bash and dash - dash parses backslashes in both the double-quoted string, and then echo does another backslash parsing pass, still printing a newline; whereas bash prints a backslash + n.
Additionally, the ksh family has `print`, which solves some of the issues.
And the story repeats all over again, each shell solving the problem in a different way :) That's one of the main reasons I went for solving the portability problem from the scripting side, not the interpreter side.
This post was thought provoking, I wonder, is the hidden argument here that the posix spec for a shell is not well specified if there is so much variance between the implementations?
Or is the fundamental issue simply a matter of history? Both?
I honestly don't know! What I know is that there is an intersection of features that work with polyfills and tweaks in all shells. That intersection has a lot of the spec, but also has a lot that is not specified anywhere.
The author has neglected the pdksh family of shells which includes all of Android, so this is a sizable oversight.
This would include the MirBSD mksh, and oksh from OpenBSD.
They are much smaller than ksh93 and bash, and I would suggest that their syntax extensions be given preference, as they originally represented ksh88 (from which the POSIX shell standard was derived).
I particularly like ksh93. It's a virtual machine, which makes it super fast for a lot of scenarios.
The ksh family is my personal favorite, in particular mksh and ksh93. Both with excelent feature sets. I even made a platformer game that works on them https://github.com/alganet/tuish/blob/main/examples/game.sh (and zsh/bash/busybox too) to show how feature-complete they are.
Shell interpreters are such a broad subject, I could go all day talking about cool things that can be done with them.
I want to get people focusing less on the spec. It's for whoever implements interpreters, not people who write scripts. And there's a gap on that, which I'm trying to cover (with full ksh support, much more than you think).
ksh2020 is `v-` based, the repo I mentioned is an `u+` fork with backports from the failed attempt. Most notably, it doesn't change the build system.
It's what Debian and Fedora use, so AT&T doesn't matter anymore. Real-world upstream is the repo I linked, which is maintained. All the 2020 drama is now obsolete.
They are not the same. For example, `[[` apparently works on Alpine's ash but not on Debian's dash. BusyBox ash has some bash compatibility options enabled in Alpine.
It's ironic that a post about shell differences glosses over this. Maybe not surprising that slop is sloppy.
Strict POSIX conformance is arguably worse. I mean, have you seen what it advises for shebangs? First of all:
The shell reads its input from a file (see sh), from the -c option or from the system() and popen() functions defined in the System Interfaces volume of POSIX.1-2017. If the first line of a file of shell commands starts with the characters "#!", the results are unspecified.
Ah, so shebangs are not required to be supported, already a great start.
Applications should note that the standard PATH to the shell cannot be assumed to be either /bin/sh or /usr/bin/sh, and should be determined by interrogation of the PATH returned by getconf PATH, ensuring that the returned pathname is an absolute pathname and not a shell built-in. [...]
Furthermore, on systems that support executable scripts (the "#!" construct), it is recommended that applications using executable scripts install them using getconf PATH to determine the shell pathname and update the "#!" script appropriately as it is being installed (for example, with sed). For example:
#
# Installation time script to install correct POSIX shell pathname
#
# Get list of paths to check
#
Sifs=$IFS
Sifs_set=${IFS+y}
IFS=:
set -- $(getconf PATH)
if [ "$Sifs_set" = y ]
then
IFS=$Sifs
else
unset IFS
fi
#
# Check each path for 'sh'
#
for i
do
if [ -x "${i}"/sh ]
then
Pshell=${i}/sh
fi
done
#
# This is the list of scripts to update. They should be of the
# form '${name}.source' and will be transformed to '${name}'.
# Each script should begin:
#
# #!INSTALLSHELLPATH
#
scripts="a b c"
#
# Transform each script
#
for i in ${scripts}
do
sed -e "s|INSTALLSHELLPATH|${Pshell}|" < ${i}.source > ${i}
done
Marvelous. What a robust foundation of useful and hard-to-misuse utilities.
It definitely builds outside docker. It's a musl-cross-make toolchain, you can procure the dependencies locally if you don't like the Docker recipes.
Feel free to open an issue if you feel like that's a challenge. Likely, you can get it to work but checksum reproducibility will be hard without a controlled environment like docker.
ECMAScript has a pretty massive amount of fully-specified behavior though; the things that differ between those implementations is nearly-entirely limited to fresh additions like `require` or whatever.
The echo thing would be like if ECMAScript allowed stuff like `"123" == 123` to give either false or true; and then indeed many things would probably break if moved across implementations.
C is the closer comparison, and indeed much software that could easily be portable (and might claim it is) often depends on implementation-specific things like 8-bit bytes, 32-bit int, assuming int8_t/etc in stdint.h exist, twos complement (before C23 at least), arithmetic shift right, etc.
reply