Skip to content

P
pyfs
Commit 079fa398 by willmcgugan
Options

Lots of docs, some cosmetic changes

parent a9aecf14
Showing with 133 additions and 86 deletions
docs/contrib/index.rst
fs.contrib
=========
==========
The ``fs.contrib`` module contains a number of filesystem implementations provided by third parties.
......
docs/index.rst
......@@ -17,11 +17,13 @@ Guide
introduction.rst
getting_started.rst
concepts.rst
opening.rst
interface.rst
filesystems.rst
contrib.rst
expose.rst
utilities.rst
commands.rst
Code Documentation
------------------
......
docs/interface.rst
......@@ -3,11 +3,11 @@ Filesystem Interface
It requires a relatively small number of methods to implement a working FS object.
If you are looking to implement a working FS object, derive a class from fs.base.FS and implement the essential methods (below). Be sure to convert all exceptions to instances of :class:`fs.errors.FSError`.
If you are looking to implement a working FS object, derive a class from :py:class:`fs.base.FS` and implement the essential methods (below). Be sure to convert all exceptions to instances of :class:`fs.errors.FSError`.
It may also be worthwhile implementing some of the non-essential methods, as the default implementations may not be optimal. For example, the method :meth:`fs.base.FS.move` is implemeneted as a file copy followed by a delete, but many filesystems can move a file without copying data.
It may also be worthwhile implementing some of the non-essential methods, as the default implementations may not be optimal. For example, the method :meth:`fs.base.FS.move` is implemented as a file copy followed by a delete, but many filesystems can move a file without copying data.
If the filesystem you are implementing maps paths to the native filesystem, be sure to implement `getsyspath`. Doing so will improve performance, especialy when copying / moving files between FS objects.
If the filesystem you are implementing maps paths to the native filesystem, be sure to implement :py:meth:`~fs.base.FS.getsyspath`. Doing so will improve performance, especially when copying / moving files between FS objects.
Essential Methods
-----------------
......@@ -28,7 +28,7 @@ The following methods are required for a minimal Filesystem interface:
Non - Essential Methods
-----------------------
The following methods have default implementations in fs.base.FS and aren't required for a functional FS interface. They may be overriden if an alternative implementation can be supplied:
The following methods have default implementations in :py:class:`fs.base.FS` and aren't required for a functional FS interface. They may be overriden if an alternative implementation can be supplied:
* :meth:`~fs.base.FS.copy` Copy a file to a new location
* :meth:`~fs.base.FS.copydir` Recursively copy a directory to a new location
......@@ -53,7 +53,7 @@ The following methods have default implementations in fs.base.FS and aren't requ
Utility Methods
---------------
The following members have implementations in fs.base.FS and will probably never require a non-default implementation, although there is nothing to prevent a derived class from implementing these:
The following members have implementations in :py:class:`fs.base.FS` and will probably never require a non-default implementation, although there is nothing to prevent a derived class from implementing these:
* :meth:`~fs.base.FS.createfile` Create a file with data
* :meth:`~fs.base.FS.getcontents` Returns the contents of a file as a string
......
docs/opener.rst
.. automodule:: fs.opener
:members:
\ No newline at end of file
docs/opening.rst 0 → 100644
Opening Filesystems
===================
Generally, when you want to work with the files and directories of any of the supported filesytems,
you create an instance of the appropriate class. For example, the following opens the directory /foo/bar::
from fs.osfs import OSFS
my_fs = OSFS('/foo/bar')
This is fine if you know beforehand where the directory you want to work with is located, and on what medium.
However, there are occasions where the location of the files may change at runtime or should be specified in a config file or from the command line.
In these situations you can use an _opener_ which is a generic way of specifying a filesystem. For example, the following is equivalent to the code above::
from fs.opener import fsopen
my_fs = fsopen('/foo/bar')
The `fsopen` method takes a string that identifies the filesystem, but if called with a regular path, it will return an OSFS instance.
To open a different kind of filesystem, you specify it with a URI like syntax. The following code opens an ftp filesystem rather than a directory on your harddrive::
from fs.opener import fsopen
my_fs = fsopen('ftp://example.org/foo/bar')
For further information regarding filesystem openers see :doc:`opener`.
\ No newline at end of file
fs/base.py
......@@ -18,13 +18,11 @@ __all__ = ['DummyLock',
'flags_to_mode',
'NoDefaultMeta']
import os, os.path
import sys
import os.path
import shutil
import fnmatch
import datetime
import time
import re
try:
import threading
except ImportError:
......@@ -41,6 +39,7 @@ class DummyLock(object):
directly use the Lock class from the dummy_threading module, since
it attempts to sanity-check the sequence of acquire/release calls
in a way that breaks when real threading is available.
"""
def acquire(self,blocking=1):
......@@ -136,9 +135,8 @@ def synchronize(func):
class FS(object):
"""The base class for Filesystem abstraction objects.
An instance of a class derived from FS is an abstraction on some kind of filesytem, such as the OS filesystem or a zip file.
An instance of a class derived from FS is an abstraction on some kind
of filesytem, such as the OS filesystem or a zip file.
"""
_meta = {}
......@@ -150,6 +148,7 @@ class FS(object):
:type thread_synchronize: bool
"""
super(FS, self).__init__()
self.closed = False
self.thread_synchronize = thread_synchronize
......@@ -166,8 +165,10 @@ class FS(object):
"""Recommends the use of caching. Implementations are free to use or
ignore this value.
:param enabled: If True the implementation is permitted to cache directory
structure / file info.
:param enabled: If True the implementation is permitted to aggressively cache directory
structure / file information. Caching such information speeds up most operations,
particularly for network based filesystems. The downside of caching is that
changes made to directories or files outside of this interface may not be picked up.
"""
pass
......@@ -175,8 +176,11 @@ class FS(object):
def close(self):
"""Close the filesystem. This will perform any shutdown related
operations required. This method will be called automatically when
an the filesystem object is cleaned up, but it is a good idea to call
it explicitly."""
the filesystem object is garbage collected, but it is good practice
to call it explicitly so that any attached resourced are freed when they
are no longer required.
"""
self.closed = True
def __getstate__(self):
......@@ -255,9 +259,9 @@ class FS(object):
return True
def getsyspath(self, path, allow_none=False):
"""Returns the system path (a path recognised by the OS) if present.
"""Returns the system path (a path recognised by the OS) if one is present.
If the path does not map to a system path (and allow_none is False)
If the path does not map to a system path (and `allow_none` is False)
then a NoSysPathError exception is thrown. Otherwise, the system
path will be returned as a unicode string.
......@@ -316,25 +320,31 @@ class FS(object):
"""Open a the given path as a file-like object.
:param path: a path to file that should be opened
:param mode: ,ode of file to open, identical to the mode string used
:param mode: mode of file to open, identical to the mode string used
in 'file' and 'open' builtins
:param kwargs: additional (optional) keyword parameters that may
be required to open the file
:rtype: a file-like object
"""
raise UnsupportedError("open file")
def safeopen(self, *args, **kwargs):
"""Like 'open', but returns a NullFile if the file could not be opened.
def safeopen(self, path, mode="r", **kwargs):
"""Like :py:meth:`~fs.base.FS.open`, but returns a :py:class:`~fs.base.NullFile` if the file could not be opened.
A NullFile is a dummy file which has all the methods of a file-like object,
but contains no data.
:rtype: file-like object
:param path: a path to file that should be opened
:param mode: mode of file to open, identical to the mode string used
in 'file' and 'open' builtins
:param kwargs: additional (optional) keyword parameters that may
be required to open the file
:rtype: a file-like object
"""
try:
f = self.open(*args, **kwargs)
f = self.open(path, mode, **kwargs)
except ResourceNotFoundError:
return NullFile()
return f
......@@ -367,7 +377,7 @@ class FS(object):
raise UnsupportedError("check for file")
def __iter__(self):
""" Iterates over paths returned by listdir method with default params. """
""" Iterates over paths returned by :py:meth:`~fs.base.listdir` method with default params. """
for f in self.listdir():
yield f
......@@ -395,8 +405,8 @@ class FS(object):
:type files_only: bool
:rtype: iterable of paths
:raises ResourceNotFoundError: if the path is not found
:raises ResourceInvalidError: if the path exists, but is not a directory
:raises :py:class:`fs.errors.ResourceNotFoundError`: if the path is not found
:raises :py:class:`fs.errror.ResourceInvalidError`: if the path exists, but is not a directory
"""
raise UnsupportedError("list directory")
......@@ -413,8 +423,9 @@ class FS(object):
the name of each item in the directory, it returns a tuple of the
name and the info dict as returned by getinfo.
Depending on the filesystem, this may be more efficient than calling
getinfo() on each individual item returned by listdir().
This method may be more efficient than calling
:py:meth:`~fs.base.FS.getinfo` on each individual item returned by :py:meth:`~fs.base.FS.listdir`, particularily
for network based filesystems.
:param path: root of the path to list
:param wildcard: filter paths that match this wildcard
......@@ -423,8 +434,8 @@ class FS(object):
:param files_only: only retrieve files
:type files_only: bool
:raises ResourceNotFoundError: If the path is not found
:raises ResourceInvalidError: If the path exists, but is not a directory
:raises :py:class:`fs.errors.ResourceNotFoundError`: If the path is not found
:raises :py:class:`ResourceInvalidError`: If the path exists, but is not a directory
"""
path = normpath(path)
......@@ -445,7 +456,8 @@ class FS(object):
dirs_only=dirs_only,
files_only=files_only)]
def _listdir_helper(self, path, entries,
def _listdir_helper(self, path,
entries,
wildcard=None,
full=False,
absolute=False,
......@@ -457,6 +469,7 @@ class FS(object):
that directory, this method applies the semantics of the listdir()
keyword arguments. An appropriately modified and filtered list of
directory entries is returned.
"""
if dirs_only and files_only:
raise ValueError("dirs_only and files_only can not both be True")
......@@ -485,11 +498,12 @@ class FS(object):
absolute=False,
dirs_only=False,
files_only=False):
"""Generator yielding the files and directories under a given path.
"""Iterator yielding the files and directories under a given path.
This method behaves identically to listdir() but returns a generator
This method behaves identically to :py:meth:`fs.base.FS.listdir` but returns an iterator
instead of a list. Depending on the filesystem this may be more
efficient than calling listdir() and iterating over the resulting list.
efficient than calling :py:meth:`fs.base.FS.listdir` and iterating over the resulting list.
"""
return iter(self.listdir(path,
wildcard=wildcard,
......@@ -504,14 +518,20 @@ class FS(object):
absolute=False,
dirs_only=False,
files_only=False):
"""Generator yielding paths and path info under a given path.
"""Iterator yielding paths and path info under a given path.
This method behaves identically to listdirinfo() but returns a generator
This method behaves identically to :py:meth:`~fs.base.listdirinfo` but returns an iterator
instead of a list. Depending on the filesystem this may be more
efficient than calling listdirinfo() and iterating over the resulting
efficient than calling :py:meth:`~fs.base.listdirinfo` and iterating over the resulting
list.
"""
return iter(self.listdirinfo(path,wildcard,full,absolute,dirs_only,files_only))
return iter(self.listdirinfo(path,
wildcard,
full,
absolute,
dirs_only,
files_only))
def makedir(self, path, recursive=False, allow_recreate=False):
"""Make a directory on the filesystem.
......@@ -522,9 +542,9 @@ class FS(object):
:param allow_recreate: if True, re-creating a directory wont be an error
:type allow_create: bool
:raises DestinationExistsError: if the path is already a directory, and allow_recreate is False
:raises ParentDirectoryMissingError: if a containing directory is missing and recursive is False
:raises ResourceInvalidError: if a path is an existing file
:raises :py:class:`fs.errors.DestinationExistsError`: if the path is already a directory, and allow_recreate is False
:raises :py:class:`fs.errors.ParentDirectoryMissingError`: if a containing directory is missing and recursive is False
:raises :py:class:`fs.errors.ResourceInvalidError`: if a path is an existing file
"""
raise UnsupportedError("make directory")
......@@ -534,8 +554,8 @@ class FS(object):
:param path: Path of the resource to remove
:raises ResourceNotFoundError: if the path does not exist
:raises ResourceInvalidError: if the path is a directory
:raises :py:class:`fs.errors.ResourceNotFoundError`: if the path does not exist
:raises :py:class:`fs.errors.ResourceInvalidError`: if the path is a directory
"""
raise UnsupportedError("remove resource")
......@@ -544,14 +564,14 @@ class FS(object):
"""Remove a directory from the filesystem
:param path: path of the directory to remove
:param recursive: pf True, then empty parent directories will be removed
:param recursive: if True, empty parent directories will be removed
:type recursive: bool
:param force: if True, any directory contents will be removed
:type force: bool
:raises ResourceNotFoundError: If the path does not exist
:raises ResourceInvalidError: If the path is not a directory
:raises DirectoryNotEmptyError: If the directory is not empty and force is False
:raises :py:class:`fs.errors.ResourceNotFoundError`: if the path does not exist
:raises :py:class:`fs.errors.ResourceInvalidError`: if the path is not a directory
:raises :py:class:`fs.errors.DirectoryNotEmptyError:` if the directory is not empty and force is False
"""
raise UnsupportedError("remove directory")
......@@ -561,6 +581,7 @@ class FS(object):
:param src: path to rename
:param dst: new name
"""
raise UnsupportedError("rename resource")
......@@ -595,15 +616,13 @@ class FS(object):
in info dictionaries for most implementations:
* "size" - Number of bytes used to store the file or directory
* "created_time" - A datetime object containing the time the resource
was created
* "accessed_time" - A datetime object containing the time the resource
was last accessed
* "modified_time" - A datetime object containing the time the resource
was modified
* "created_time" - A datetime object containing the time the resource was created
* "accessed_time" - A datetime object containing the time the resource was last accessed
* "modified_time" - A datetime object containing the time the resource was modified
:param path: a path to retrieve information for
:rtype: dict
"""
raise UnsupportedError("get resource info")
......@@ -679,7 +698,7 @@ class FS(object):
error_callback=None):
"""Create a new file from a string or file-like object asynchronously
This method returns a threading.Event object. Call the `wait` on the event
This method returns a `threading.Event` object. Call the `wait` method on the event object
to block until all data has been written, or simply ignore it.
:param path: a path of the file to create
......@@ -783,17 +802,18 @@ class FS(object):
:param path: root path to start walking
:param wildcard: if given, only return files that match this wildcard
:type wildcard: A string containing a wildcard (e.g. *.txt) or a callable that takes the file path and returns a boolean
:type wildcard: a string containing a wildcard (e.g. `*.txt`) or a callable that takes the file path and returns a boolean
:param dir_wildcard: if given, only walk directories that match the wildcard
:type dir_wildcard: A string containing a wildcard (e.g. *.txt) or a callable that takes the directory name and returns a boolean
:param search: -- a string dentifying the method used to walk the directories. There are two such methods:
* 'breadth' Yields paths in the top directories first
* 'depth' Yields the deepest paths first
:type dir_wildcard: a string containing a wildcard (e.g. `*.txt`) or a callable that takes the directory name and returns a boolean
:param search: a string dentifying the method used to walk the directories. There are two such methods:
* "breadth" yields paths in the top directories first
* "depth" yields the deepest paths first
:param ignore_errors: ignore any errors reading the directory
"""
if wildcard is None:
wildcard = lambda f:True
elif not callable(wildcard):
......@@ -864,11 +884,12 @@ class FS(object):
:param path: root path to start walking
:param wildcard: if given, only return files that match this wildcard
:type wildcard: A string containing a wildcard (e.g. *.txt) or a callable that takes the file path and returns a boolean
:type wildcard: A string containing a wildcard (e.g. `*.txt`) or a callable that takes the file path and returns a boolean
:param dir_wildcard: if given, only walk directories that match the wildcard
:type dir_wildcard: A string containing a wildcard (e.g. *.txt) or a callable that takes the directory name and returns a boolean
:type dir_wildcard: A string containing a wildcard (e.g. `*.txt`) or a callable that takes the directory name and returns a boolean
:param search: same as walk method
:param ignore_errors: ignore any errors reading the directory
"""
for path, files in self.walk(path, wildcard=wildcard, dir_wildcard=dir_wildcard, search=search, ignore_errors=ignore_errors):
for f in files:
......@@ -883,11 +904,12 @@ class FS(object):
:param path: root path to start walking
:param wildcard: if given, only return dictories that match this wildcard
:type wildcard: A string containing a wildcard (e.g. *.txt) or a callable that takes the directory name and returns a boolean
:type wildcard: A string containing a wildcard (e.g. `*.txt`) or a callable that takes the directory name and returns a boolean
:param search: same as the walk method
:param ignore_errors: ignore any errors reading the directory
"""
for p, files in self.walk(path, dir_wildcard=wildcard, search=search, ignore_errors=ignore_errors):
for p, _files in self.walk(path, dir_wildcard=wildcard, search=search, ignore_errors=ignore_errors):
yield p
......@@ -897,6 +919,7 @@ class FS(object):
:param path: a path to the resource
:rtype: integer
:returns: the size of the file
"""
info = self.getinfo(path)
size = info.get('size', None)
......@@ -913,7 +936,8 @@ class FS(object):
be overwritten; If False then DestinationExistsError
will be raised.
:param chunk_size: size of chunks to use if a simple copy is required
(defaults to 16K).
(defaults to 64K).
"""
if not self.isfile(src):
......@@ -969,6 +993,7 @@ class FS(object):
:param chunk_size: Size of chunks to use when copying, if a simple copy
is required
:type chunk_size: integer
"""
src_syspath = self.getsyspath(src, allow_none=True)
......@@ -1002,6 +1027,7 @@ class FS(object):
exceptions when moving files
:param chunk_size: size of chunks to use when copying, if a simple copy
is required
"""
if not self.isdir(src):
if self.isfile(src):
......@@ -1062,6 +1088,7 @@ class FS(object):
:type ignore_errors: bool
:param chunk_size: size of chunks to use when copying, if a simple copy
is required (defaults to 16K)
"""
if not self.isdir(src):
raise ResourceInvalidError(src, msg="Source is not a directory: %(path)s")
......@@ -1101,6 +1128,7 @@ class FS(object):
:param path: a directory path
:rtype: bool
"""
path = normpath(path)
iter_dir = iter(self.listdir(path))
......
fs/opener.py
......@@ -5,14 +5,14 @@ fs.opener
Open filesystems via a URI.
There are occasions when you want to specify a filesytem from the command line or in a config file.
This module enables that functionality, and can return a FS object given a URI like syntax (http://commons.apache.org/vfs/filesystems.html).
This module enables that functionality, and can return an FS object given a URI like syntax (http://commons.apache.org/vfs/filesystems.html).
The `OpenerRegistry` class maps the protocol (file, ftp etc.) on to an Opener object, which returns an appropriate filesystem object and path.
You can create a custom opener registry that opens just the filesystems you require, or use the opener registry defined here (also called `opener`) that can open any supported filesystem.
The `parse` method of an `OpenerRegsitry` object returns a tuple of an FS object a path. Here's an example of how to use the default opener registry::
>>> from fs.opener import opener
>>> opener.parse('ftp://ftp.mozilla.org')
>>> opener.parse('ftp://ftp.mozilla.org/pub')
(<fs.ftpfs.FTPFS object at 0x96e66ec>, u'pub')
You can use use the `opendir` method, which just returns an FS object. In the example above, `opendir` will return a FS object for the directory `pub`::
......@@ -26,12 +26,11 @@ If you are just interested in a single file, use the `open` method of a registry
<fs.ftpfs._FTPFile object at 0x973764c>
The `opendir` and `open` methods can also be imported from the top-level of this module for sake of convenience.
To avoid shadowing the builtin `open` methd, they are named `fsopendir` and `fsopen`. Here's how you might import them::
To avoid shadowing the builtin `open` method, they are named `fsopendir` and `fsopen`. Here's how you might import them::
from fs.opener import fsopendir, fsopen
"""
__all__ = ['OpenerError',
......@@ -55,10 +54,7 @@ __all__ = ['OpenerError',
'DavOpener',
'HTTPOpener']
import sys
from fs.osfs import OSFS
from fs.path import pathsplit, basename, join, iswildcard, normpath
import os
from fs.path import pathsplit, join, iswildcard, normpath
import os.path
import re
from urlparse import urlparse
......@@ -104,7 +100,7 @@ def _parse_name(fs_name):
def _split_url_path(url):
if '://' not in url:
url = 'http://' + url
scheme, netloc, path, params, query, fragment = urlparse(url)
scheme, netloc, path, _params, _query, _fragment = urlparse(url)
url = '%s://%s' % (scheme, netloc)
return url, path
......@@ -172,7 +168,7 @@ class OpenerRegistry(object):
:param fs_url: an FS url
:param default_fs_name: the default FS to use if none is indicated (defaults is OSFS)
:param writeable: if True, a writeable FS will be returned
:oaram create_dir: if True, then the directory in the FS will be created
:param create_dir: if True, then the directory in the FS will be created
"""
......@@ -343,8 +339,6 @@ class ZipOpener(Opener):
@classmethod
def get_fs(cls, registry, fs_name, fs_name_params, fs_path, writeable, create_dir):
append_zip = fs_name_params == 'add'
zip_fs, zip_path = registry.parse(fs_path)
if zip_path is None:
raise OpenerError('File required for zip opener')
......@@ -357,7 +351,7 @@ class ZipOpener(Opener):
open_mode = 'w+'
zip_file = zip_fs.open(zip_path, mode=open_mode)
username, password, fs_path = _parse_credentials(fs_path)
_username, _password, fs_path = _parse_credentials(fs_path)
from fs.zipfs import ZipFS
if zip_file is None:
......@@ -384,11 +378,11 @@ rpc://www.example.org (opens an RPC server on www.example.org, default port 80)"
@classmethod
def get_fs(cls, registry, fs_name, fs_name_params, fs_path, writeable, create_dir):
from fs.rpcfs import RPCFS
username, password, fs_path = _parse_credentials(fs_path)
_username, _password, fs_path = _parse_credentials(fs_path)
if '://' not in fs_path:
fs_path = 'http://' + fs_path
scheme, netloc, path, params, query, fragment = urlparse(fs_path)
scheme, netloc, path, _params, _query, _fragment = urlparse(fs_path)
rpcfs = RPCFS('%s://%s' % (scheme, netloc))
......@@ -411,10 +405,10 @@ examples:
from fs.ftpfs import FTPFS
username, password, fs_path = _parse_credentials(fs_path)
scheme, netloc, path, params, query, fragment = urlparse(fs_path)
scheme, _netloc, _path, _params, _query, _fragment = urlparse(fs_path)
if not scheme:
fs_path = 'ftp://' + fs_path
scheme, netloc, path, params, query, fragment = urlparse(fs_path)
scheme, netloc, path, _params, _query, _fragment = urlparse(fs_path)
dirpath, resourcepath = pathsplit(path)
url = netloc
......@@ -519,7 +513,7 @@ example:
def get_fs(cls, registry, fs_name, fs_name_params, fs_path, writeable, create_dir):
from fs.wrapfs.debugfs import DebugFS
if fs_path:
fs, path = registry.parse(fs_path, writeable=writeable, create_dir=create_dir)
fs, _path = registry.parse(fs_path, writeable=writeable, create_dir=create_dir)
return DebugFS(fs, verbose=False), None
if fs_name_params == 'ram':
from fs.memoryfs import MemoryFS
......@@ -590,8 +584,8 @@ class TahoeOpener(Opener):
fs = TahoeFS(dircap, webapi=url)
if '/' in path:
dirname, resourcename = pathsplit(path)
if createdir:
dirname, _resourcename = pathsplit(path)
if create_dir:
fs = fs.makeopendir(dirname)
else:
fs = fs.opendir(dirname)
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment