FYI: I wrote xdg basedir package for Go.
https://github.com/zchee/go-xdgbasedir

I want to take a stab on this.

@xiaq are you OK with the following design?

• ~/.elvish/ exists: Do not change behavior because this is most likely a legacy (pre-xdg) install. Output a message on launch to advertise moving to $E:XDG_CONFIG_HOME/elvish.
• Add a flag -config that defaults to $E:XDG_CONFIG_HOME/elvish or ~/.config/elvish.
  dataDir from

  elvish/runtime/runtime.go

  Line 53 in 702edb3

  var dataDir string

  gets assigned that flag value.

I would use @zchee's implementation with the Native mode to honor Apple's guidelines, see https://github.com/zchee/go-xdgbasedir/blob/7a0b8922e178cedb342090b90ef662084d88eba8/xdgbasedir.go#L16-L27

I'm OK with following XDG. However if we are going to move directory, better make it the last directory move we ever have to do. The current .elvish directory has 3 things:

• .elvish/rc.elv: This is configuration.
• .elvish/db: This is user's data.
• .elvish/lib: This is libraries.

XDG has a clear distinction between configuration and data, so clearly rc.elv and db should be in different places. I don't know which bucket .elvish/lib falls into though, the specification does not say. It's worthwhile researching what other languages that respect XDG do.

I'd also prefer that we have a separate implementation in Elvish instead of depending on go-xdgbasedir. I don't mean to discredit their effort, but the main reason is that this behavior needs to be documented in Elvish's documentation, and it's easier to make sure that the behavior and documentation are in sync when they are part of the same codebase.

Elvish also creates temporary files for the socket used for communication with the storage daemon; this is now always /tmp/elvish-$uid. We can use XDG's runtime directory for this, but this can be done separately.

To summarize there are several more things to do:

1. Researching where to put .elvish/lib.
2. Writing Elvish's own implementation for finding the directories.
3. Documenting the behavior in Elvish's documentation.
4. [Optional] Follow XDG for the runtime directory too.

maybe the lib should be sourced from multiple places, to also account for potential elvish code managed by system package manager and maybe to separate epm fetched libs and user's. something like /usr/share/elvish/ -> $XDG_DATA_HOME/elvish/lib -> $XDG_CONFIG_HOME/elvish/lib. maybe thats overkill.

@SolitudeSF Yes, I think this is a good opportunity to introduce a system-wide lib too.

On a related note, after reading the XDG spec, it seems that the particular definition it uses for "data files" is not so much as "user's activity data", but more like "assets used in programs", as it cites /usr/share as one of the default paths for data.

In this light, it seems that @fehnomenal's original idea of treating everything as config would make a lot of sense. And since we are back to dealing with just one directory, it makes more sense to implement it ourselves now.

A system-wide lib would be really cool. In preparation to it I will implement the complete XDG Basedir spec. The lib could then be placed somewhere in $XDG_DATA_DIRS.

For the current task I will place

• rc and libs into $XDG_CONFIG_HOME/elvish
• db into $XDG_CACHE_HOME/elvish (it only contains "temporary" data right? I.e. is not configuration)
• the socket into $XDG_RUNTIME_DIR/elvish.sock
• daemon logs into $XDG_DATA_HOME/elvish (debatable but is there a better place?)

The specification defines cache as "non-essential data", which is pretty vague. But a useful litmus test would be "if cache files are removed, the behavior of the program does not change, except maybe the next startup is slower". This is not true for db.

The XDG base directory specification is written in too vague a way, and it is impossible to make good choices. Treating the entire .elvish directory as config might as well be the least bad one.

Hmm, after looking at how Fish stores its history in ~/.local/share/fish, I am leaning a bit towards treating db as a data file now.

The way I think about it is whether the information is unique to a system, in which case it belongs in ~/.local/share/appname, or is part of the apps configuration, in which case it belongs in ~/.config/appname. In other words, is it something I probably want to sync to every system on which I run the app (such as when setting up a new system)? If yes, ~/.config, else ~/.local/share.

Call me old-fashioned, but I find XDG confusing and cumbersome, I much prefer plain dot-dirs under $HOME. But I won't object to the change, as I see a lot of other tools following XDG 😐. Also, it's great if a system-wide config dir comes from this 😄

Right. I am less sure whether the complexity is worthwhile too, especially as the spec doesn't really have good definitions of how to use the different kinds of directories.

The spec is also more complex than I had, and likely most people had, expected. It's a good idea to adopt some standard, but that doesn't equate to adopting any standard. In this particular case, I am no longer sure whether the XDG base directory spec is the correct standard to adopt.

People expect that XDG is supported. Storing everything in $HOME is incorrect approach.

If one wants to see how the XDG spec can be adapted to different situations, the arch wiki page is quite good: https://wiki.archlinux.org/index.php/XDG_Base_Directory

@xiaq ultimately, adopting a standard that no one else is using is largely equivalent to not adopting a standard. A standard can only functionally be considered a "standard" if it is actually in widespread use, and in this problem space there are really only two candidates that pass that test: XDG, or chuck everything in $HOME/.<something>.

I also agree with @krader1961 that the primary distinction between the use of XDG_DATA_HOME and XDG_CONFIG_HOME is by whether or not data is machine-specific, in practice. To my knowledge, most applications that follow the XDG spec do store "history" files in XDG_DATA_HOME.

FWIW, the current situation causes problems when ~/.elvish is on a shared filesystem such as NFS. In my case I symlink it to my Dropbox filesystem folder. This causes Dropbox to create copies with names like db (Krader MacBook Pro's conflicted copy 2020-03-11). The important thing here is that the command history database (the db file) is fundamentally local to a machine. Whereas everything else, including the .elvish/lib directory, is almost certainly stuff you want shared with all the machines on which you use elvish.

The XDG standard is not the only way to deal with the above problem. But it's probably the solution with the broadest support.

If nothing else the situation regarding the location of the Elvish database file needs to be changed. I've been using the rsync command to copy my ~/.elvish directory to the VMs that I use for testing changes. When syncing to Windows 10 that results in this error:

rsync: [receiver] rename "/c/Users/krade/.elvish/.db.T34m17" -> ".elvish/db": Device or resource bus
y (16)

Even on UNIX like systems, where that operation does not result in a fatal error, it is arguably wrong to copy the interactive command database from one system to another when syncing the Elvish config files. Syncing interactive command and location history between systems should be done independent of the Elvish config.

@xiaq Regarding the idea of treating everything as config, the rule of thumb I've found best to apply is that it's best if "config" is restricted to some finite, enumerable collection of configurable values, while "data" captures any data that could not be reconstructed simply by enumerating the configurable values, and setting them according to your preferences. This makes it a distinction between "inconvenient to reproduce" and "potentially infeasible to reproduce". Things like history are by nature "data", while by contrast things like "beep/don't beep on tab completion with no candidates" are config.

Shells tend to have a more complicated relationship with this than most GUI applications, as their config is often more nuanced, and itself may be "non-enumerable" due to being dynamically scripted. However, this should (IMO) be read as making it more like data, rather than making everything config.

@eternaleye I disagree - I think "config" should be anything that is needed to bring the program/application to the settings that you need to use it. In this sense, rc.elv (Elvish), init.el (Emacs), and other code-based configuration files must definitely be considered config, despite their non-enumerable nature.

Another way to put it: config is everything I could not bear to lose (e.g. I would check it into a git repo, possibly share it among machines, etc.), and data is anything that could be eventually reconstructed (e.g. history, command DB, etc.).

I agree with @eternaleye's interpretation. I could (with difficulty) reconstruct my shell configuration, but I could never possibly reconstruct the tens of thousands of unique command invocations stored in my shell history. Another example of XDG "data" that might be more illustrative is video game save files, which are infeasible to reproduce. An example of XDG "config" would be the graphics/audio settings inside the game, which are easily reproduced.

Of course, shell config is much more complicated than a bunch of checkboxes and dropdowns, so it hews closer to XDG "data" in terms of how strictly it needs to be preserved (and why I keep mine backed up and in Git), as @eternaleye mentioned. But I just take this as an illustration of what the XDG config/data distinction is intended to mean in general. For a shell, I would expect it to take after Fish and put config/functions in $XDG_CONFIG_HOME/elvish and history or other user data in $XDG_DATA_HOME/elvish.

EDIT: @krader1961 marked this as confusing — is there a specific point I should elaborate on? My only real disagreement is with the idea that history "could be eventually reconstructed". I couldn't reconstruct mine, and I would be irritated to lose it. I think we otherwise already agree that Elvish config should be in XDG_CONFIG_HOME and history should be in XDG_DATA_HOME (or maybe XDG_STATE_HOME).

The XDG base directory spec was actually updated shortly after the last comment was posted :)

It now includes the concept of "state data" that is a good home for ~/.elvish/db:

There is a single base directory relative to which user-specific state data should be written. This directory is defined by the environment variable $XDG_STATE_HOME.

...

$XDG_STATE_HOME defines the base directory relative to which user-specific state files should be stored. If $XDG_STATE_HOME is either not set or empty, a default equal to $HOME/.local/state should be used.

The $XDG_STATE_HOME contains state data that should persist between (application) restarts, but that is not important or portable enough to the user that it should be stored in $XDG_DATA_HOME. It may contain:

• actions history (logs, history, recently used files, …)

• current state of the application that can be reused on a restart (view, layout, open files, undo history, …)

The remaining problem is where lib should go. I actually agree with @SolitudeSF that libraries should be searched in both config directories and data directories, because there are two different types of libraries:

• Libraries you write yourself and modify frequently. Such libraries can be thought as an extension of rc.elv, you want to bring it across your machines. So these libraries should be considered configs.
• Libraries somebody else wrote, and you downloaded. Such libraries can be easily recreated from a epm:install command, so should be considered data.

To summarize:

The actual directory used by Elvish will be base directory + /elvish; for example the default path for db will be ~/.local/state/elvish/db.

For macOS I think I'll follow XDG too, since Apple's guidelines seem to be geared towards GUI applications. Having to type ~/Library/Application Support/Elvish/rc.elv to change your startup config also feels ridiculous.

I generally like the proposal. Would rc.elv then live in ~/.config/elvish/ as well? That would make it easier to keep it as a checked-out git repository, rather than mixed with other files in ~/.config/.

Would rc.elv then live in ~/.config/elvish/ as well?

Yes.

This is now merged into the master branch. I'll close this issue after having the behavior properly documented.