Syllable Desktop 0.6.7 Released

 

 

System Structure

Syllable is an operating system with a high degree of compatibility with the POSIX standards and de facto conventions in Unix systems and Linux distributions. This is a double edged sword. It makes it much easier to get existing third-party software to run on Syllable, but it also burdens us with ancient methods from the beginning of computing that have partly lost their meaning. One example is the layout and naming of files in the system and in additional software packages. We have always tried to strike a balance between following conventions and inventing more modern and user friendly methods. With a decade of experience, we have gathered the confidence to make a number of extra steps towards more user friendly structures, with as little disruption to compatibility as possible.

Instead of trying to shield people from the workings of their system, we want them to be able to go into the directories of the system or a separate package just with a filer, and be able to make some sense out of the files there and find what they are looking for. To that end, we are introducing a series of new measures to rename and move subdirectories that often form the starting point for searches for information.

The system tools Packager and Builder were further developed to support these restructurings, among other things to improve the use of symlinking for package registration. Packager is included in the system, but Builder needs to be installed as part of the Developer's Delight package, if you want to compile software yourself. When Builder is installed on Syllable 0.6.6, an older version is used that is still compatible with the older system structures. The Syllable branch in Builder/packages/ is also still largely compatible with Syllable 0.6.6, so that some packages can still be built on older systems. After the release of Syllable 0.6.7, Builder and its recipe tree will be updated to the latest versions, using the new structures.

Graphical programs that, besides the desktop menu, also need to be available from the command line are now kept in the search path locations and symlinked from /Applications/. Previously, this was usually done the other way around, but that meant that the program in /Applications/ couldn't be moved.

Packages are in the process of moving to clearer names for their main package directory. Instead of the traditional, short, lowercase names that are confusing to common people, common typographical standards are now being used. Resource packages have project names, which are proper names, so they are capitalised. Generic terms remain lowercased. When names consist of multiple words, CamelCase is used to avoid spaces. When uppercase abbreviations would lead to words of only capitals being concatenated, hyphens are used for more clarity. Abbreviations are expanded when they don't take too much space; roughly twenty characters maximum. Exceptions can be made to keep names recognisable to people who already know the short names, and to keep them roughly in the same place in alphabetical order. Around half the resource packages in the system are currently renamed.


The internal structure of resource packages is in the process of being heavily reorganised, doing away with the traditional Unix naming that is confusing and has lost its meaning over time. The structure of the symlink pools has changed accordingly. Subdirectories are being moved and renamed thus:

etc             -> settings
bin             -> programs
sbin            -> system-programs
libexec         -> framework/executables
lib             -> framework/libraries
include         -> framework/headers
share/aclocal   -> framework/AutoConfigure
share/pkgconfig -> framework/PackageConfigure
lib/pkgconfig   -> framework/PackageConfigure
share           -> data
share/doc       -> documentation
man             -> manuals
share/man       -> manuals
info            -> manuals/info
share/info      -> manuals/info

Subdirectories that are not immediately meaningful to common users are tucked away in the framework subdirectory, a name which should at least signal to users that it should not be tampered with without appropriate knowledge. The documentation directories are split between an unmanaged and a managed (structured manuals) part, to make a distinction in how they should be used and the degree to which a user could tamper with them.

Since there is no standard for the location of REBOL and Boron libraries, they can go into framework/REBOL.

The Syllable-specific initialisation scripts were consolidated into one subdirectory. This leaves room for adding the common Unix script classes for stop, reload, restart and status commands:

early-init      -> tasks/setup
init            -> tasks/start

An extra subdirectory applications was introduced for resource packages that contain graphical programs that should appear in an applications menu. This was prompted by the port of the VICE emulator. This and other packages need to be in /resources/, but want to make entry points in the Syllable menu like /Applications/ do.
Support in the menu needs to be implemented separately.

A few packages need extra, internal compatibility symlinks because they don't fully support this reorganisation. On the other hand, such links can also be used to solve interdependencies between packages.

The symlink pool /resources/index/ and Packager have compatibility facilities to support most older binary packages.


Moved bin/ to programs/.
Most common people would probably associate “bin” with a dustbin, and wonder why every package has its own trashcan. We'd have to explain that it means “binaries”, then explain what binaries are and what a binary number system is, and then explain that bin/ directories don't just contain binaries, and that binaries can also be found elsewhere... bin/ has only been a fitting name for a short period in the earliest Unix history, before the advent of scripts. Since the invention of libraries, there are also binary files elsewhere.

Moved sbin/ to system-programs/.
The “root” super user in Syllable Server has already been renamed to “system”. This is also planned for Desktop. system-programs/ will then be a matching name for programs that need to be executed by the super user.

Moved share/ to data/.
share/ originally meant files that can be shared over a network between multiple machines, because they're data instead of architecture dependent binaries. This name is not very meaningful on Syllable, because disk space is not an issue anymore for these files and we build self-contained systems that can be operated off-line from a network. share/ directories are symlinked to make them available as an aggregate within the system, but most resource subdirectories are shared that way.

init/ and early-init/ were already consolidated into tasks/. The remaining subdirectories are not meaningful to common users, because they contain support files for programs/ and applications/, so I folded them into a new subdirectory framework/ to signify their function and move them out of sight.

Moved lib/ to framework/libraries/.
Libel? Liberation? Makes it clear what we're talking about. It's still not the same as a public library, but at least it's clear that it belongs to the framework of a resource package, and libraries is the term used everywhere in system and programming documentation, so it would be confusing for techies to change it more than this.

Moved libexec/ to framework/executables/.
libexec/ clearly has a confused personality, so this makes that explicit. It can contain both support programs that don't need to be in the search path and libraries, which are both executables, so it clearly belongs to the framework.

Moved include/ to framework/headers/.
Include which in what? We're talking about compiled language programming headers for the libraries, and that's still a very technical term, so we tuck it away in framework/.

Moved share/aclocal/ to framework/AutoConfigure/.
This contains M4 macros for the GNU AutoTools, AutoConfigure and AutoMake, that many packages use for source code configuration. Very technical and applies to the libraries, so it's part of the framework. Besides, M4 macros are scripts, so it's not even proper data/. Not even techies can know what aclocal/ means until they find the explanation. Having a directory named local inside the directory share/ is also pretty disingenious. This is not a universally used configuration system, so we name it after the clearest component, that also has a relation to the original name.

Consolidated lib/pkgconfig/ and share/pkgconfig/ into framework/PackageConfigure/.
This is for another software configuration system named pkg-config, and contains data files describing properties of the package. Again, this is mostly meant for other packages to use the libraries in this resource package, so it's part of the framework. We use the clear CamelCase version of the name of the configuration system. Packages in the wild are confused about whether this is data or belongs with the libraries. Consolidating those two locations into one simplifies our search path for PackageConfigure files.

Moved lib/REBOL/ to framework/REBOL/.
Programming languages tend to have their own hierarchy under lib/ for their support files.
There is no standard place for REBOL and Boron libraries, so we are free to change this.
Having it immediately under framework/ instead of under framework/libraries/ shortens paths.

Moved etc/ to settings/.
etc/ must be the dumbest computer term ever introduced. It seems the designers were completely out of inspiration. I imagine it was scribbled on a napkin over lunch at the end of the other brilliant ideas for the Unix filesystem layout, and later implemented without further thought. While it's understandable that these people didn't count on common people needing to understand their system in the early 1970's, the naming is even of exceptionally poor quality in a technical context. What's really stupefying, though, is that these design choices have been defended as high culture by Unix types ever since.
Note that eventually, we shouldn't have changing data in program directories, so in most cases we direct this settings directory to the global and unfortunately standard /etc/.

Unlike earlier restructurings, the locations of these directories are critical to the operation of the software packages. Many components have accompanying search paths, that I have adapted in different places in the system. However, packages don't always use these and may try to find components in fixed locations or relative to themselves. Many packages configured with the GNU AutoTools support build options to change these locations. As much as possible, Builder now uses those options to perform these moves, so the packages are aware of them. However, many package configurations are partly broken, especially regarding this because few systems are interested in changing these locations. Such packages may need to be repaired by adding compatibility symlinks from the traditional subdirectories to the new ones.

This can actually be turned into an advantage. Quite a few resource packages act as platforms for other packages. Those other packages contain extensions in the same location as in the platform package. In other systems, these files would end up in the very same directories. In Syllable, the platform packages can't find those modules because packages are built into their own, separate directories. However, they are registered in our symlink pools to create the same view as in other systems. By using the traditional subdirectories in platform packages to symlink into the symlink pools, they can be pointed to the extensions in other packages.

Moved ^/lib/ to ^/libraries/.
This is the subdirectory relative to a program executable, that can contain private dynamic libaries for the program. Changed it to match the other renaming.

Some of these changes still have the old counterparts in the system, so that most old resource packages and applications will still work on systems with the new structure, until they are obsolete.


Reorganised the init scripts.

We've been porting more Unix server packages over time. They may need setup and teardown procedures that are usually categorised in five types: start, stop, reload, restart and status. We have init/ and early-init/ subdirectories within ported packages. init/ corresponds to the start script in Unix systems, which is actually a better name, because common users have no idea what “init” means. There's no real equivalent to early-init/, and we use that mainly to set variables in users' shell environments. Eventually, we would have to support the other script types, as well. Having them each in a different subdirectory within a package would be a mess.

Created a new (optional) subdirectory tasks/ within resource packages. Builder and Packager work together to symlink the content into /resources/index/ and other symlink pools.
Moved the init/ subdirectory to tasks/start/.
Moved early-init/, having no naming identification with init/ anymore, to tasks/setup/.
This name also better reflects the fact that setup scripts are called at every shell start, not just at system start-up or user log-in.
Future scripts can be added under tasks/ as tasks/stop/, tasks/reload/, tasks/restart/ and tasks/status/.


Reorganised, unified and integrated the documentation in ported packages.

Third-party packages have at least six locations for documentation. We collect loose documentation files in a documentation/ subdirectory. Many packages also have freeform documentation in a share/doc/ subdirectory. Traditionally, “man” pages are in man/ but GNU has been moving them to share/man/ lately. They prefer “info” files, anyway, and have first defined those in info/ but are now moving them to share/info/.

This is a mess for users. We want them to be able to go into a package directory just with a filer, and find documentation they are looking for. We were already moving share/man/ and share/info/ back to man/ and info/ because the share/ subdirectory is not very meaningful in Syllable. We build a self-contained system that typically doesn't share system files over networks, and we symlink several subdirectories to make them globally available within the system, not just the share/ hierarchy.

Any files in share/doc/ are now merged into documentation/. The man and info pages should not go in there, because they're structured databases that are meant to be read in their respective viewers. man/ and share/man/ are now moved to manuals/, a much clearer name. Because they have a structured content, it's possible to also move info/ and share/info/ to manuals/info/. The latter was pioneered by GoboLinux. Builder and Packager work together to accomplish this. As much as possible, Builder uses packages' configure options so they are aware of the moves.
Remaining files are moved afterwards.


The GCC libraries and headers installed in the system now have priority over a separately installed full GCC package. This protects system integrity while allowing to install versions of GCC older than the system libraries. This is a cleaner way than manipulating the GCC package, which was done before.

There are several legitimate reasons for installing a GCC package that is not in sync with the system libraries. The GCC package may be newer than the system release. It may be older, or an old GCC may be required to compile some program. If the versions diverge too much, this can disrupt the working of the system.
Originally, I removed the shared libraries from the GCC packages, but that may not be enough. Then I removed the complete lib/ subdirectory, but GCC does not appreciate that in all cases. Currently, the GCC packages have their include/ and lib/ subdirectories linked to the versions in the system. However, that may not work when the GCC version is too different from the system. Also, it makes the complete /system/index/include/ and /system/index/lib/ structures appear in /resources/index/, where they may conflict with other versions of the same packages.

What's really needed is to give the system libraries priority over GCC, while letting GCC keep its own versions for when it needs those internally. I chose to do that through the system paths. Other options are installing GCC in a separate package pool, or making an exception for it when symlinking in Packager.

Note that other packages in /resources/ still have priority over /system/resources/, so apart from GCC, it's still possible to overrule system components.


The symlink pools were moved from /system/indexes/ and /resources/indexes/ to /system/index/ and /resources/index/ to shorten paths throughout the system. The old locations are still available as compatibility symlinks to the new locations, to support packages that were built before Syllable 0.6.7.


/usr/ is now a symbolic link to /resources/ instead of the other way around. In the next step of this migration, they will be separated into two directories.


Implemented, in Packager, extra symlinking in the symlink pools to point to the main version of installed resource packages. This allows to point to packages in versioned pools without mentioning the version. The package symlink points to the main/currently selected version.

The links are made in the symlink pools, not in the installed packages. This allows to have multiple symlink pools making different selections of the same packages, for example for different user accounts or software build environments.


Symlinks that are made from the system into resource packages (usually for compatibility reasons) can be completed by Builder. So far, Builder added the complete path of the specific resource package, including the version for packages in the system area. This is not robust against upgrades of a versioned package or a change of the package name, because these symlinks are not included in single binary packages.

Now, Builder completes incomplete paths in a recipe based on the symlink pool of the package. Thus, the symlink does not contain the package name and version anymore and remains valid when those change.