webdepcompress / files

     1.1 --- /dev/null	Thu Jan 01 00:00:00 1970 +0000
     1.2 +++ b/.hgignore	Sun Aug 30 12:00:04 2009 +0200
     1.3 @@ -0,0 +1,2 @@
     1.4 +\.py[co]$
     1.5 +\.DS_Store$

     2.1 --- /dev/null	Thu Jan 01 00:00:00 1970 +0000
     2.2 +++ b/setup.py	Sun Aug 30 12:00:04 2009 +0200
     2.3 @@ -0,0 +1,16 @@
     2.4 +from setuptools import setup
     2.5 +
     2.6 +
     2.7 +setup(
     2.8 +    name='WebDepCompress',
     2.9 +    version='0.1',
    2.10 +    url='http://dev.pocoo.org/hg/webdepcompress/',
    2.11 +    license='BSD',
    2.12 +    author='Armin Ronacher',
    2.13 +    author_email='armin.ronacher@active-4.com',
    2.14 +    description='JavaScript and CSS Compression Package',
    2.15 +    long_description=__doc__,
    2.16 +    packages=['webdepcompress', 'webdepcompress.compressors'],
    2.17 +    namespace_packages=['webdepcompress.compressors'],
    2.18 +    platforms='any'
    2.19 +)

     3.1 --- /dev/null	Thu Jan 01 00:00:00 1970 +0000
     3.2 +++ b/webdepcompress/__init__.py	Sun Aug 30 12:00:04 2009 +0200
     3.3 @@ -0,0 +1,111 @@
     3.4 +# -*- coding: utf-8 -*-
     3.5 +"""
     3.6 +    webdepcompress
     3.7 +    ~~~~~~~~~~~~~~
     3.8 +
     3.9 +    This package implements a simple framework-independent system for web
    3.10 +    dependency compression.  With the help of various compresseors it
    3.11 +    compresses JavaScript and CSS if necessary and allows a fallback if
    3.12 +    the files are used uncompressed (developer mode).
    3.13 +
    3.14 +    This package provides a naive, whitespace removing compressor for
    3.15 +    CSS and JavaScript but a powerful, yuicompressor based compressing
    3.16 +    algorithm can be installed from the pypi that is used automatically
    3.17 +    if available.  This makes it possible to ship this package with an
    3.18 +    application for production usage and precompressed files without
    3.19 +    having to install yuicompressor and Java.
    3.20 +
    3.21 +    Declaring Packs
    3.22 +    ---------------
    3.23 +
    3.24 +    Add a file to your package (for example packs.py) with the following
    3.25 +    contents::
    3.26 +
    3.27 +        from webdepcompress import PackManager
    3.28 +
    3.29 +        mgr = PackManager(os.path.join(os.path.dirname(__file__), 'static'),
    3.30 +                          lambda fn, t: '/static/' + fn)
    3.31 +
    3.32 +        mgr.add_pack('default', ['style.css', 'print.css',
    3.33 +                                 'jquery.js', 'application.js'])
    3.34 +
    3.35 +    First you have to create a pack manager.  That manager keeps a registry
    3.36 +    of all your packs, because you can have multiple of those.  The first
    3.37 +    argument to it is the path to where the files are stored.  Most of the
    3.38 +    time it makes sure to point to somewhere inside your package.  That path
    3.39 +    is used as base path for the source files and it will also be used as
    3.40 +    a path for the compressed files when created.  If you want a different
    3.41 +    path for the generated files, you can do so.  Have a look at the
    3.42 +    :cls:`PackManager` documentation for more information.
    3.43 +
    3.44 +    The second argument to the manager is a function that returns the URL
    3.45 +    to the file.  In this case it assumes that the files are available
    3.46 +    as ``/static/filename.css`` and so forth.
    3.47 +
    3.48 +    When you add a pack you call :meth:`~PackManager.add_pack` with the
    3.49 +    name of the pack as first argument and the files it should pack together
    3.50 +    as the second.
    3.51 +
    3.52 +    Using Packs
    3.53 +    -----------
    3.54 +
    3.55 +    Packs provide a simple interface:
    3.56 +
    3.57 +    >>> mgr['default']
    3.58 +    <Pack 'default'>
    3.59 +    >>> print mgr['default']
    3.60 +    <link rel="stylesheet" type="text/css" href="/static/style.css">
    3.61 +    <link rel="stylesheet" type="text/css" href="/static/print.css">
    3.62 +    <script type="text/javascript" src="/static/jquery.js"></script>
    3.63 +    <script type="text/javascript" src="/static/application.js"></script>
    3.64 +
    3.65 +    As you can see, the `__str__` and `__unicode__` special methods of a
    3.66 +    pack return the HTML needed to include the files specified.  Please note
    3.67 +    that it will print out the stylesheets before the scripts by default
    3.68 +    and that scripts and files are kept in the order specified but grouped
    3.69 +    by type.  It also outputs HTML4/HTML5 by default and not XHTML.  This
    3.70 +    can be changed.
    3.71 +
    3.72 +    Packs can be compressed by calling the :meth:`~PackManager.compress`
    3.73 +    method on the manager or of a pack:
    3.74 +
    3.75 +    >>> mgr.compress()
    3.76 +
    3.77 +    That can take a while, after that you can see that the manager spits
    3.78 +    out the compressed includes:
    3.79 +
    3.80 +    >>> print mgr['default']
    3.81 +    <link rel="stylesheet" type="text/css" href="/static/default.compressed.css">
    3.82 +    <script type="text/javascript" src="/static/default.compressed.js"></script>
    3.83 +
    3.84 +    You can pass packs to your template engine easily.  The best idea is to
    3.85 +    forward the pack as a string or list of strings so that the template
    3.86 +    cannot call `compress` on it.
    3.87 +
    3.88 +    Distutils Integration
    3.89 +    ---------------------
    3.90 +
    3.91 +    You can also add a cmdclass to your `setup.py` script that compresses
    3.92 +    the pack automatically::
    3.93 +
    3.94 +        from webdepcompress.support import compress_deps
    3.95 +
    3.96 +        setup(
    3.97 +            ...,
    3.98 +            cmdclasses={'compress_deps': compress_deps},
    3.99 +            webdep_manager='yourapplication.packs.mgr'
   3.100 +        )
   3.101 +
   3.102 +    Then you can compile the deps from the command line::
   3.103 +
   3.104 +        $ python setup.py compress_deps
   3.105 +
   3.106 +    And clean the compressed files again::
   3.107 +
   3.108 +        $ python setup.py compress_deps --clean
   3.109 +
   3.110 +
   3.111 +    :copyright: (c) 2009 by Armin Ronacher, see AUTHORS for more details.
   3.112 +    :license: BSD, see LICENSE for more details.
   3.113 +"""
   3.114 +from manager import PackManager

     4.1 --- /dev/null	Thu Jan 01 00:00:00 1970 +0000
     4.2 +++ b/webdepcompress/compressors/__init__.py	Sun Aug 30 12:00:04 2009 +0200
     4.3 @@ -0,0 +1,1 @@
     4.4 +__import__('pkg_resources').declare_namespace(__name__)

     5.1 --- /dev/null	Thu Jan 01 00:00:00 1970 +0000
     5.2 +++ b/webdepcompress/compressors/base.py	Sun Aug 30 12:00:04 2009 +0200
     5.3 @@ -0,0 +1,23 @@
     5.4 +# -*- coding: utf-8 -*-
     5.5 +"""
     5.6 +    webdepcompress.compressors.base
     5.7 +    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
     5.8 +
     5.9 +    Implements the compressor base.
    5.10 +
    5.11 +    :copyright: (c) 2009 by Armin Ronacher, see AUTHORS for more details.
    5.12 +    :license: BSD, see LICENSE for more details.
    5.13 +"""
    5.14 +
    5.15 +
    5.16 +class CompressorBase(object);
    5.17 +
    5.18 +    def __init__(self, mgr, reporter):
    5.19 +        self.mgr = mgr
    5.20 +        self.reporter = reporter
    5.21 +
    5.22 +    def compress_css(self, dst_filename, files):
    5.23 +        pass
    5.24 +
    5.25 +    def compress_js(self, js, files):
    5.26 +        pass

     6.1 --- /dev/null	Thu Jan 01 00:00:00 1970 +0000
     6.2 +++ b/webdepcompress/compressors/naive.py	Sun Aug 30 12:00:04 2009 +0200
     6.3 @@ -0,0 +1,149 @@
     6.4 +# -*- coding: utf-8 -*-
     6.5 +"""
     6.6 +    webdepcompress.compressors.naive
     6.7 +    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
     6.8 +
     6.9 +    Implements a naive compressor.
    6.10 +
    6.11 +    :copyright: (c) 2009 by Armin Ronacher, see AUTHORS for more details.
    6.12 +    :license: BSD, see LICENSE for more details.
    6.13 +"""
    6.14 +import re
    6.15 +from operator import itemgetter
    6.16 +from webdepcompress.compressors.base import CompressorBase
    6.17 +
    6.18 +
    6.19 +operators = [
    6.20 +    '+', '-', '*', '%', '!=', '==', '<', '>', '<=', '>=', '=',
    6.21 +    '+=', '-=', '*=', '%=', '<<', '>>', '>>>', '<<=', '>>=',
    6.22 +    '>>>=', '&', '&=', '|', '|=', '&&', '||', '^', '^=', '(', ')',
    6.23 +    '[', ']', '{', '}', '!', '--', '++', '~', ',', ';', '.', ':'
    6.24 +]
    6.25 +operators.sort(key=lambda x: -len(x))
    6.26 +
    6.27 +rules = [
    6.28 +    ('whitespace', re.compile(r'\s+(?u)')),
    6.29 +    ('dummycomment', re.compile(r'<!--.*')),
    6.30 +    ('linecomment', re.compile(r'//.*')),
    6.31 +    ('multilinecomment', re.compile(r'/\*.*?\*/(?us)')),
    6.32 +    ('name', re.compile(r'(\$+\w*|[^\W\d]\w*)(?u)')),
    6.33 +    ('number', re.compile(r'''(?x)(
    6.34 +        (?:0|[1-9]\d*)
    6.35 +        (\.\d+)?
    6.36 +        ([eE][-+]?\d+)? |
    6.37 +        (0x[a-fA-F0-9]+)
    6.38 +    )''')),
    6.39 +    ('operator', re.compile(r'(%s)' % '|'.join(map(re.escape, operators)))),
    6.40 +    ('string', re.compile(r'''(?xs)(
    6.41 +        '(?:[^'\\]*(?:\\.[^'\\]*)*)'  |
    6.42 +        "(?:[^"\\]*(?:\\.[^"\\]*)*)"
    6.43 +    )'''))
    6.44 +]
    6.45 +
    6.46 +division_re = re.compile(r'/=?')
    6.47 +regex_re = re.compile(r'/(?:[^/\\]*(?:\\.[^/\\]*)*)/[a-zA-Z]*(?s)')
    6.48 +line_re = re.compile(r'(\r\n|\n|\r)')
    6.49 +ignored_tokens = frozenset(('dummycomment', 'linecomment', 'multilinecomment'))
    6.50 +
    6.51 +
    6.52 +class Token(tuple):
    6.53 +    """Represents a token as returned by `tokenize`."""
    6.54 +    __slots__ = ()
    6.55 +
    6.56 +    def __new__(cls, type, value, lineno):
    6.57 +        return tuple.__new__(cls, (type, value, lineno))
    6.58 +
    6.59 +    type = property(itemgetter(0))
    6.60 +    value = property(itemgetter(1))
    6.61 +    lineno = property(itemgetter(2))
    6.62 +
    6.63 +
    6.64 +def indicates_division(token):
    6.65 +    """A helper function that helps the tokenizer to decide if the current
    6.66 +    token may be followed by a division operator.
    6.67 +    """
    6.68 +    if token.type == 'operator':
    6.69 +        return token.value in (')', ']', '}', '++', '--')
    6.70 +    return token.type in ('name', 'number', 'string', 'regexp')
    6.71 +
    6.72 +
    6.73 +def contains_newline(string):
    6.74 +    """Checks if a newline sign is in the string."""
    6.75 +    return '\n' in string or '\r' in string
    6.76 +
    6.77 +
    6.78 +def tokenize(source):
    6.79 +    """Tokenize a JavaScript source.
    6.80 +
    6.81 +    :return: generator of `Token`\s
    6.82 +    """
    6.83 +    may_divide = False
    6.84 +    pos = 0
    6.85 +    lineno = 1
    6.86 +    end = len(source)
    6.87 +
    6.88 +    while pos < end:
    6.89 +        # handle regular rules first
    6.90 +        for token_type, rule in rules:
    6.91 +            match = rule.match(source, pos)
    6.92 +            if match is not None:
    6.93 +                break
    6.94 +        # if we don't have a match we don't give up yet, but check for
    6.95 +        # division operators or regular expression literals, based on
    6.96 +        # the status of `may_divide` which is determined by the last
    6.97 +        # processed non-whitespace token using `indicates_division`.
    6.98 +        else:
    6.99 +            if may_divide:
   6.100 +                match = division_re.match(source, pos)
   6.101 +                token_type = 'operator'
   6.102 +            else:
   6.103 +                match = regex_re.match(source, pos)
   6.104 +                token_type = 'regexp'
   6.105 +            if match is None:
   6.106 +                # woops. invalid syntax. jump one char ahead and try again.
   6.107 +                pos += 1
   6.108 +                continue
   6.109 +
   6.110 +        token_value = match.group()
   6.111 +        if token_type is not None:
   6.112 +            token = Token(token_type, token_value, lineno)
   6.113 +            may_divide = indicates_division(token)
   6.114 +            yield token
   6.115 +        lineno += len(line_re.findall(token_value))
   6.116 +        pos = match.end()
   6.117 +
   6.118 +
   6.119 +class Compressor(CompressorBase):
   6.120 +    """Basic compressor that just strips whitespace and comments."""
   6.121 +
   6.122 +    def compress_js(self, dst_filename, files):
   6.123 +        dst = open(dst_filename, 'w')
   6.124 +        try:
   6.125 +            for filename in files:
   6.126 +                src = open(filename, 'r')
   6.127 +                try:
   6.128 +                    tokeniter = tokenize(src.read().decode(self.mgr.charset))
   6.129 +                finally:
   6.130 +                    src.close()
   6.131 +                for token in tokeniter:
   6.132 +                    if token.type == 'whitespace':
   6.133 +                        if contains_newline(token.value):
   6.134 +                            dst.write('\n')
   6.135 +                    elif token.type not in ignored_tokens:
   6.136 +                        dst.write(token.value.encode(self.mgr.charset))
   6.137 +                dst.write('\n')
   6.138 +        finally:
   6.139 +            dst.close()
   6.140 +
   6.141 +    def compress_css(self, dst_filename, files):
   6.142 +        # XXX: strip whitespace and comments
   6.143 +        dst = open(dst_filename, 'w')
   6.144 +        try:
   6.145 +            for filename in files:
   6.146 +                src = open(filename, 'r')
   6.147 +                try:
   6.148 +                    dst.write(src.read())
   6.149 +                finally:
   6.150 +                    src.close()
   6.151 +        finally:
   6.152 +            dst.close()

     7.1 --- /dev/null	Thu Jan 01 00:00:00 1970 +0000
     7.2 +++ b/webdepcompress/manager.py	Sun Aug 30 12:00:04 2009 +0200
     7.3 @@ -0,0 +1,139 @@
     7.4 +# -*- coding: utf-8 -*-
     7.5 +"""
     7.6 +    webdepcompress.manager
     7.7 +    ~~~~~~~~~~~~~~~~~~~~~~
     7.8 +
     7.9 +    Implements the pack manager.
    7.10 +
    7.11 +    :copyright: (c) 2009 by Armin Ronacher, see AUTHORS for more details.
    7.12 +    :license: BSD, see LICENSE for more details.
    7.13 +"""
    7.14 +import os
    7.15 +from weakref import ref as weakref
    7.16 +
    7.17 +
    7.18 +CSS_TEMPLATE = '<link rel="stylesheet" type="text/css" href="%s">'
    7.19 +JS_TEMPLATE = '<script type="text/javascript" src="%s"></script>'
    7.20 +DEFAULT_COMPRESSORS = ('yui', 'naive')
    7.21 +
    7.22 +
    7.23 +def get_compressor(choices=None):
    7.24 +    """Return a suitable compressor from the choices or the builtins."""
    7.25 +    def _import(name):
    7.26 +        try:
    7.27 +            mod = __import__(name, None, None, ('Compressor',))
    7.28 +            return getattr(mod, 'Compressor')
    7.29 +        except (ImportError, AttributeError):
    7.30 +            return None
    7.31 +
    7.32 +    if choices is None:
    7.33 +        choices = DEFAULT_COMPRESSORS
    7.34 +    for impname in choices:
    7.35 +        compressor = _import('webdepcompress.compressors.' + impname)
    7.36 +        if compressor is not None:
    7.37 +            return compressor
    7.38 +        compressor = _import(impname)
    7.39 +        if compressor is not None:
    7.40 +            return compressor
    7.41 +
    7.42 +    raise RuntimeError('none of the specified compressors were found')
    7.43 +
    7.44 +
    7.45 +class Pack(object):
    7.46 +    """Represents a pack."""
    7.47 +
    7.48 +    def __init__(self, mgr, name, files):
    7.49 +        self._mgr = weakref(mgr)
    7.50 +        self.name = name
    7.51 +        self._css = []
    7.52 +        self._js = []
    7.53 +        for filename in files:
    7.54 +            assert '.' in filename, 'unknown file without extension'
    7.55 +            ext = filename.rsplit('.', 1)[-1]
    7.56 +            if ext == 'js':
    7.57 +                self._js.append(filename)
    7.58 +            elif ext == 'css':
    7.59 +                self._css.append(filename)
    7.60 +            else:
    7.61 +                assert False, 'unknown extension ".%s"' % ext
    7.62 +
    7.63 +    def get_mgr(self):
    7.64 +        rv = self._mgr()
    7.65 +        if rv is None:
    7.66 +            raise RuntimeError('manager got garbage collected')
    7.67 +        return rv
    7.68 +
    7.69 +    def _get_generated_filename(self, ext):
    7.70 +        return self.get_mgr().build_filename % {'name': self.name, 'ext': ext}
    7.71 +
    7.72 +    def _make_gen_iterator(ext):
    7.73 +        def iter_ext(self):
    7.74 +            mgr = self.get_mgr()
    7.75 +            def _format(link):
    7.76 +                yield mgr.css_template % mgr.link_func(link, ext)
    7.77 +            fn = self._get_generated_filename(ext)
    7.78 +            if os.path.isfile(os.path.join(mgr.directory, fn)):
    7.79 +                yield _format(fn)
    7.80 +                return
    7.81 +            for filename in getattr(self, '_' + ext):
    7.82 +                yield _format(filename)
    7.83 +        return iter_ext
    7.84 +
    7.85 +    iter_css = _make_gen_iterator('css')
    7.86 +    iter_js = _make_gen_iterator('js')
    7.87 +    del _make_gen_iterator
    7.88 +
    7.89 +    def compress(self, compressors=None, reporter=None):
    7.90 +        compressor = get_compressor(compressors)(self.get_mgr(), reporter)
    7.91 +        compressor.compress_css(self._get_generated_filename('css', self._css))
    7.92 +        compressor.compress_js(self._get_generated_filename('js', self._js))
    7.93 +
    7.94 +    def __iter__(self):
    7.95 +        mgr = self.get_mgr()
    7.96 +        iters = self.iter_css, self.iter_js
    7.97 +        if not mgr.css_first:
    7.98 +            iters = reversed(iters)
    7.99 +        for func in iters:
   7.100 +            for item in func():
   7.101 +                yield item
   7.102 +
   7.103 +    def __unicode__(self):
   7.104 +        return u'\n'.join(self)
   7.105 +
   7.106 +    def __str__(self):
   7.107 +        return '\n'.join(x.encode('utf-8') for x in self)
   7.108 +
   7.109 +
   7.110 +class PackManager(object):
   7.111 +
   7.112 +    __module__ = 'webdepcompress'
   7.113 +
   7.114 +    def __init__(self, directory, link_func, css_first=True,
   7.115 +                 css_template=CSS_TEMPLATE, js_template=JS_TEMPLATE,
   7.116 +                 build_filename='%(name)s.compressed.%(ext)s',
   7.117 +                 charset='utf-8'):
   7.118 +        self.directory = directory
   7.119 +        self.link_func = link_func
   7.120 +        self.dst_path = dst_path
   7.121 +        self.css_first = css_first
   7.122 +        self.css_template = CSS_TEMPLATE
   7.123 +        self.js_template = JS_TEMPLATE
   7.124 +        self.build_filename = build_filename
   7.125 +        self.compressors = compressors
   7.126 +        self.charset = charset
   7.127 +        self._packs = {}
   7.128 +
   7.129 +    def compress(self, compressors=None, reporter=None):
   7.130 +        for pack in self._packs.itervalues():
   7.131 +            pack.compress(compressors, reporter)
   7.132 +
   7.133 +    def add_pack(self, name, files):
   7.134 +        self._packs[name] = Pack(self, name, files)
   7.135 +
   7.136 +    def remove_pack(self, name):
   7.137 +        rv = self._packs.pop(name, None)
   7.138 +        if rv is None:
   7.139 +            raise ValueError('no pack named %r found' % name)
   7.140 +
   7.141 +    def __getitem__(self, name):
   7.142 +        return self._packs[name]