Updated 2013-01-28 02:22:41 by RLE ▲

Discussion arising from A template virtual filesystem

SEH 20061030 -- The re-written and much-improved template vfs which I presented at Tcl'2006 [1], and the collection of virtual filesystems built from it, have been uploaded to the CVS repository of the TclVFS project. They have also been released in a zip file as a subsidiary project on the sourceforge project page of the FILTR [2].

The new template vfs sacrifices simplicity somewhat, in exchange for API's and functions intended to make it as easy as possible to write functional, scalable new virtual filesystems.

There is a new, greatly simplified programming interface which allows one to author a new vfs merely by overloading a few simple procs.

Error handling is significantly improved.

A complete set of caching functions is included with user-settable cache dwell times. These are intended to make virtual filesystems operating over networks much more efficient.

I hope the availability of this template will prompt more people to start writing virtual filesystems of their own. They are an incredible problem-solving tool. On to the core!

jcw - Wow. FWIW, I'm working towards a distributed model for Metakit-like views in the context of the Vlerq / Ratcl / Tequila project. Once that comes within reach, it will all hopefully slot right into several of your VFS designs.

SEH As a self-teaching exercise, I wrote a sample virtual filesystem that utilizes as much of the Tclvfs API as possible, although its function is trivially simple. The virtual filesystems included with the Tclvfs distribution are instructive, but they take a lot of shortcuts.

Because it simply mirrors a real filesystem location, I believe this vfs is maximally featureful.

My aim for this vfs is to use it as a template for rapid development of new and more complex filesystems.

The Mount, Unmount and handler procedures are completely generic and should never need customization for new filesystems. Thus the task of creating a new virtual filesystem is reduced to filling in the procedures for handling the eight subcommands of the Tclvfs API, as well as mounting and unmounting specifics.

SEH -- 10/14/04 -- I tidied things up a bit, and added an execution trace to the 'close' command so that it will throw a proper error if an error occurs in the close callback procedure. Not too relevant for this vfs, but for more complex ones built on the template.

I also added a "-volume" option to the Mount command, so you can now do things like:
 ::vfs::template::Mount -volume $env(HOME) C:/

on Unix, and make pathnames originating on Windows acceptable. Problems syncing cross-platform filesystems solved! Schweet!

12/07/04 -- I had to rewrite the Access procedure to correct a misunderstanding about how the access handler worked; I took the opportunity to optimize it to minimize the number of disk reads required.

12/22/04 -- I made it more of a real template by removing references to "::vfs::template" within procedure bodies and replaced them with [namespace current]. Now the Mount, Unmount and handler procedures can be copied and pasted directly into new vfs code without editing.

20040720 CMcC: I just saw a real use for this - a trace fs. Someone was trying to work out why copy -force was failing, it would have been nice to be able to say to them: mount this on your directory and look at the log. Trivial addition to this template - log each argument to, and result from, underlying file system.

snitvfs has a reimplementation of this, in Snit.

LV Would it be beneficial a) to include this template with tclvfs and b) to upgrade the existing tclvfs examples with the features your template provide?

SEH I would love to have my work incorporated into the tcllib, but I don't think it's my decision.

CMcC With all respect to SEH, whose idea the template vfs is, I would like to suggest that the template vfs either be rewritten in Snit, or a parallel version in Snit written and maintained as part of the same package. I have rewritten Steve's template vfs in Snit, for the reasons I give below (although I'm not suggesting that my version is necessarily the best possible version.)

The reasons to use Snit for template vfs:

  1. Very often in tclvfs code one is forced to carry state around with each mount-point. There's no easy/neat way to do this in pure templates (as code inspection will reveal :)
  2. Consistent -option handling, with defaults.
  3. Clearer interface definition and composition - Snit is designed to plug components together.
  4. OO paradigms are interconvertable more easily than rewriting something into OO. Now's a good time to make the choice to go OO - before it's too late.
  5. Easier to extend a Snit type to extend functionality than to do the same with pure-namespace implementations.
  6. (add here)

Rubbish reasons not to use Snit for vfs:

  1. I don't know/use Snit ... <contra>either it's worth using or not, lack of training isn't an excuse.</contra>.
  2. I don't like Snit ... <contra>suggest another system, suggest some improvements - argue about it.</contra>
  3. I don't like OO / I've got my own OO style ... <contra>two words: code reuse</contra>

Real reasons not to use Snit for tclvfs:

  1. Snit's slower than pure template code aka Snit is no speed demon... This is the hardest one to answer for tclvfs file systems which are speed critical - but what are you doing using tclvfs for a speed-critical facility? Snit's getting a lot of optimisation attention, and this will only improve the performance over time.

Finally, the best reason for using Snit for tclvfs: Much of the need for this Template vfs is that the tclvfs raw interface has some anomalies and some lacks.

The template vfs is really a shim to provide:

  1. a consistent interface to tclvfs where there are some incongruities (smoothing out the rough spots.)
  2. genuine extensions of tclvfs functionality which could be better implemented in tcl than in C.

Some of the provisions in point 1 can eventually be moved into the tclvfs C core. Doing this over an OO interface is (I believe) easier and less disruptive than doing it over a pure namespace implementation.

Therefore: I would like to argue, passionately, for at least *some* OO interface, and preferably Snit (because it seems to be the ostensive standard for pure tcl OO,) whether or not this is maintained in parallel with or alternative to a pure namespace template tclvfs shim. I would also like to see the existing tclvfs' migrated to whatever system is adopted, and new tclvfs' adhering to whatever that system is.

Addendum: Of course, if it's a choice between a raw namespace template vfs and a no template vfs at all, I would emphatically prefer the namespace template vfs.

SEH - 10/18/04 - My motivation for writing the template vfs was both to teach myself the Tclvfs API, and to verify that the Tclvfs package was in good enough shape to write software tools around, since I have been bitterly disappointed in the past by the gulf between claims of maturity in software and emergent reality. I'm very enthusiastic about the vfs concept, and I think it's a solution looking for problems. But the bottom line is the Tclvfs package is still officially in a beta state, and writing the template vfs did indeed reveal significant issues, some of which it appears will require changes in the core to address. Given that, I find it hard to justify the extra time and effort necessary to increase the sophistication of the vfs's as you suggest. No amount of polish in the vfs code will compensate for the inherent flaws in the underlying package. If the maintainers of the package are willing to commit to bringing it out of beta and fixing the identified issues, I will revisit the possibility of taking the vfs's to the next level. Until then, I'm going to put my available time into solving the problems I face in the present that the Tclvfs package and the vfs's are capable of helping with in their current state.

On the other hand, I was also hoping to spark interest in the Tclvfs package, which seemed to be languishing, so I am heartened by your involvement and advocacy. I hope the friction and noise will help to motivate the maintainers.

P.S. By way of further motivation, I think there are enough bits floating around now to envision, if consolidated, a sophisticated content management/version control/code distribution and installation service that would knock Bitkeeper on its heels and blow the doors off anything offered by any competing scripting technology.

LV Have the tclvfs issues mentioned by SEH been communicated to the tclvfs project via the bug report and feature request queues?

SEH -- They're all in there.

Let's take 'em one at a time:

[1004273] Can't read from channel in close callback [3]

I have written this tip [4] which I believe will fix the problem. CMcC

Category DiscussionCategory VFS