Deploying Django

I can either do it the right way or the cheating way in screen, that might crash Django when certain errors are encountered. The screnning way might need this to keep the process running when I logout


TODO, suggested by mark:
with apache in front, mod_wsgi handles the processes for you
.. and standard server config takes care of the apache stuff

----

SSL .. Django admin through port forwarding?

Setting up MSW on Rackspace

./manage.py shell

django.core.exceptions.ImproperlyConfigured: Error loading MySQLdb module: No module named MySQLdb

In root install: pip install mysql-python.

If you I got another error ...

In file included from _mysql.c:29:0:
pymemcompat.h:10:20: fatal error: Python.h: No such file or directory
compilation terminated.
error: command 'gcc' failed with exit status 1

... then you need to install sudo apt-get install python-dev

(I searched for solutions for a long time, trying to find my "/usr/local/mysql" on ubuntu, but I don't have it (even though I have mysql).)


Now ./manage.py shell has

ImportError: No module named OpenSSL

From root, pip install pyopenssl


Now ./manage.py shell has

ImportError: No module named bycrypt

From root, pip install py-bcrypt

AND THEN I GOT MANAGE.PY SHELL!!!
import django! works on it too!!!!!!!!

$ ./manage.py shell
Python 2.7.1+ (r271:86832, Apr 11 2011, 18:13:53) 
[GCC 4.5.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>> import django

Whoa I can even run it on my rackspace IP! :D Even though it crashes, at least it can get the request!

$ ./manage.py runserver 50.57.147.82:8000
Validating models...

0 errors found
Django version 1.3.1, using settings 'MozSecWorld.settings_local'
Development server is running at http://50.57.147.82:8000/
Quit the server with CONTROL-C.
[21/Oct/2011 03:23:36] "GET / HTTP/1.1" 301 0

Traceback (most recent call last):
...
TemplateSyntaxError: Caught ImportError while rendering: No module named jinja2
[21/Oct/2011 03:23:37] "GET /msw/ HTTP/1.1" 500 2909

So I did, from root, pip install jinja2

AND MOZSECWORLD IS UP at http://50.57.147.82:8000/!!!!!!!! :D:D:D

Setting up more Django on a clean Ubuntu server

Get virtualenvs and virtualenvwrapper following this and this and this (with an informative video)

On root account:
1. sudo apt-get install python-setuptools

2. sudo easy_install virtualenv

3.

virtualenv ve_msw
New python executable in ve_msw/bin/python
Installing setuptools............done.
Installing pip...............done.

4. easy_install pip (I could also have done "pip install virtualenv" or "easy_install virtualenvwrapper"

5. pip install virtualenvwrapper

On user account:
6. mkdir ~/.virtualenvs

7. Add to end of ~/.bashrc:

export WORKON_HOME=$HOME/.virtualenvs
source /usr/local/bin/virtualenvwrapper.sh

8.

$ mkvirtualenv ve_msw
New python executable in ve_msw/bin/python
Installing setuptools............done.
Installing pip...............done.
virtualenvwrapper.user_scripts creating /home/haoqili/.virtualenvs/ve_msw/bin/predeactivate
virtualenvwrapper.user_scripts creating /home/haoqili/.virtualenvs/ve_msw/bin/postdeactivate
virtualenvwrapper.user_scripts creating /home/haoqili/.virtualenvs/ve_msw/bin/preactivate
virtualenvwrapper.user_scripts creating /home/haoqili/.virtualenvs/ve_msw/bin/postactivate
virtualenvwrapper.user_scripts creating /home/haoqili/.virtualenvs/ve_msw/bin/get_env_details

9. Added to ~/.virtualenvs/postmkvirtualenv

proj_name=$(echo $VIRTUAL_ENV|awk -F'/' '{print $NF}')
mkdir $HOME/Projects/$proj_name
add2virtualenv $HOME/Projects/$proj_name
cd $HOME/Projects/$proj_name
easy_install readline ipython
pip install -e git://github.com/jbalogh/check.git#egg=check Jinja2==2.3.1 \
    MySQL-python==1.2.3c1 lxml==2.2.6 PIL

10. Now I can call workon ve_msw

MSW already have playdoh set up at the very beginning
11. To set up where virtualenvwrapper's "workon" takes us to add this to
~/.virtualenvs/ve_msw/bin/postactivate

cd ~/path/to/mswdir

Django vendor-local adding package

playdoh (master)$ pip install --no-install --build=vendor-local/packages --src=vendor-local/src -I django-csp
Downloading/unpacking django-csp
  Downloading django_csp-1.0.2.tar.gz
  Running setup.py egg_info for package django-csp

which installed django-csp into vendor-local/packages

and then I added the path to vendor-local/vendor.pth:

packages/django-csp

FYI, the contents:

playdoh/vendor-local/packages/django-csp (master)$ ls -R
PKG-INFO            django_csp.egg-info setup.cfg
csp                 pip-egg-info        setup.py

./csp:
__init__.py   decorators.py middleware.py urls.py       views.py

./django_csp.egg-info:
PKG-INFO             dependency_links.txt
SOURCES.txt          top_level.txt

./pip-egg-info:
django_csp.egg-info

./pip-egg-info/django_csp.egg-info:
PKG-INFO             dependency_links.txt
SOURCES.txt          top_level.txt

The Resource

x-frame-options old notes

Taken from old mozsecworld.org page on x-frame-options

What to do

Add in this line to your HTTP Response Headers: X-Frame-Options: deny

How to check

Firefox's Firebug or Chrome's "Inspect Elements": Go to "Net" option, refresh page, and click on the link that shows up, select "Headers" and you should see "X-Frame-Options: DENY" under "Response Headers"

Terminal: >telnet [insert IP-address of your site] 8000 Press enter and put in GET /en-US/msw/ HTTP/1.1 Press enter twice, and scroll to the very top of the output, should see "x-frame-options: DENY".

What I did

Playdoh automatically sets the "X-Frame-Options" to "deny". But if you want to set it automatically in Django, use response['x-frame-options'] = 'DENY'

In views.py:

# X-Frame-Options
def xfo_deny(request):
    html = " ... my html stuff ... "
    response = HttpResponse(html)
    response['x-frame-options'] = 'DENY'
    return response

---

Notes on how Django does x-frame-options

In vendor/src/commonware/commonware/response/middleware.py:

from django.conf import settings

class FrameOptionsHeader(object):
    """
    Set an X-Frame-Options header. Default to DENY. Set
    response['x-frame-options'] = 'SAMEORIGIN'
    to override.
    """

    def process_response(self, request, response):
        if hasattr(response, 'no_frame_options'):
            return response

        if not 'x-frame-options' in response:
            response['x-frame-options'] = 'DENY'
        

In vendor/src/commonware/commonware/response/decorators.py:

from functools import wraps

from django.utils.decorators import available_attrs


def xframe_sameorigin(view_fn):
    @wraps(view_fn, assigned=available_attrs(view_fn))
    def _wrapped_view(request, *args, **kwargs):
        response = view_fn(request, *args, **kwargs)
        response['x-frame-options'] = 'SAMEORIGIN'
        return response
    return _wrapped_view


def xframe_allow(view_fn):
    @wraps(view_fn, assigned=available_attrs(view_fn))
    def _wrapped_view(request, *args, **kwargs):
        response = view_fn(request, *args, **kwargs)
        response.no_frame_options = True
        return response
    return _wrapped_view


def xframe_deny(view_fn):
    @wraps(view_fn, assigned=available_attrs(view_fn))
    def _wrapped_view(request, *args, **kwargs):
        response = view_fn(request, *args, **kwargs)
        response['x-frame-options'] = 'DENY'
        return response
    return _wrapped_view
        

How adopted-kitsune Django upload works.

0. forms.py

class ImageAttachmentUploadForm(forms.Form):

    """Image upload form."""

    print "############################in form"

    image = forms.ImageField(error_messages={'required': MSG_IMAGE_REQUIRED,

                                             'max_length': MSG_IMAGE_LONG},

                             max_length=settings.MAX_FILENAME_LENGTH)

1. views.py

for fileupload page if request.method == 'POST':
    form = forms.ImageAttachmentUploadForm(request.POST, request.FILES)
    if form.is_valid()

2. playdoh/vendor/src/django/django/forms/fields.py

class ImageField(FileField):
    def to_python(self, data):

        """
        Checks that the file-upload field data contains a valid image (GIF, JPG,
        PNG, possibly others -- whatever the Python Imaging Library supports).
        """

Ubuntu pip install record

MozSecWorld$ sudo pip install -r requirements/compiled.txt 
Downloading/unpacking py-bcrypt==0.2 (from -r requirements/compiled.txt (line 7))
  Running setup.py egg_info for package py-bcrypt
Downloading/unpacking hashlib==20081119 (from -r requirements/compiled.txt (line 6))
  Running setup.py egg_info for package hashlib
    Using OpenSSL version 0x009080ff from
     Headers: /usr/include
     Library: /usr/lib/libssl.so
    no previously-included directories found matching 'build'
    no previously-included directories found matching 'dist'
    no previously-included directories found matching 'RCS'
    no previously-included directories found matching 'CVS'
    no previously-included directories found matching '.svn'
Downloading/unpacking Jinja2==2.5.5 (from -r requirements/compiled.txt (line 2))
  Running setup.py egg_info for package Jinja2
    warning: no previously-included files matching '*' found under directory 'docs/_build'
    warning: no previously-included files matching '*.pyc' found under directory 'jinja2'
    warning: no previously-included files matching '*.pyc' found under directory 'docs'
    warning: no previously-included files matching '*.pyo' found under directory 'jinja2'
    warning: no previously-included files matching '*.pyo' found under directory 'docs'
Downloading/unpacking MySQL-python==1.2.3c1 (from -r requirements/compiled.txt (line 1))
  Running setup.py egg_info for package MySQL-python
    sh: mysql_config: not found
    Traceback (most recent call last):
      File "", line 14, in 
      File "/home/haoqili/Desktop/MozSecWorld/build/MySQL-python/setup.py", line 15, in 
        metadata, options = get_config()
      File "setup_posix.py", line 43, in get_config
        libs = mysql_config("libs_r")
      File "setup_posix.py", line 24, in mysql_config
        raise EnvironmentError("%s not found" % (mysql_config.path,))
    EnvironmentError: mysql_config not found
    Complete output from command python setup.py egg_info:
    sh: mysql_config: not found

Traceback (most recent call last):

  File "", line 14, in 

  File "/home/haoqili/Desktop/MozSecWorld/build/MySQL-python/setup.py", line 15, in 

    metadata, options = get_config()

  File "setup_posix.py", line 43, in get_config

    libs = mysql_config("libs_r")

  File "setup_posix.py", line 24, in mysql_config

    raise EnvironmentError("%s not found" % (mysql_config.path,))

EnvironmentError: mysql_config not found

----------------------------------------
Command python setup.py egg_info failed with error code 1
Storing complete log in /home/haoqili/.pip/pip.log

and then python manage.py runserver still gives "ImportError: No module named bcrypt" error.

so sudo pip install py-bcrypt worked

Preparing to Set up MozSecWorld on Linux

Since the permanent host for MozSecWorld will be on linux.

1. Install libapache2-mod-wsgi. This is a Python apache!
sudo apt-get install libapache2-mod-wsgi

2. Install Apache2. /etc/init.d/apache2 status should say that "Apache2 is running"

3. Configure Apache2.

4. clone repository git clone https://github.com/haoqili/MozSecWorld

5. Get the vendor. cd MozSecWorld/vendor and then git clone --recursive git://github.com/mozilla/playdoh-lib.git .
follow the Setup Section until
python manage.py runserver does not complain about any missing modules

Getting Image Upload to work on Django

Learned form r1cky that jpeg and PIL are packages that need to be recompiled for other people, especially those on different operating systems

haoqili: how can I tell if a package needs to be recompiled on the VM or not?
r1cky: hmm, that is a good question. if it is only .py source files then it doesnt need compiling
r1cky: but some libraries have C in them
r1cky: the ones that we use that need compiling are
r1cky: MySQL-python, Jinja2, PIL, lxml
r1cky: and the jpeg one
r1cky: those are installed separately (not copied into the repository)

1. Install PIL into my vendor-local

playdoh (master)$  pip install -I --install-option="--home=`pwd`/vendor-local" pil
Downloading/unpacking pil
  Downloading PIL-1.1.7.tar.gz (506Kb): 506Kb downloaded
  Running setup.py egg_info for package pil
    WARNING: '' not a valid package name; please use only.-separated package names in setup.py
    WARNING: '' not a valid package name; please use only.-separated package names in setup.py
    
Installing collected packages: pil
  Running setup.py install for pil
    WARNING: '' not a valid package name; please use only.-separated package names in setup.py
    WARNING: '' not a valid package name; please use only.-separated package names in setup.py
    --- using frameworks at /System/Library/Frameworks
    building '_imaging' extension
    /usr/bin/cc -fno-strict-aliasing -O3 -w -pipe -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -DHAVE_LIBZ -I/System/Library/Frameworks/Tcl.framework/Headers -I/System/Library/Frameworks/Tk.framework/Headers -IlibImaging -I/Users/haoqili/.virtualenvs/playdoh/include -I/usr/local/include -I/usr/include -I/usr/local/Cellar/python/2.7.1/include/python2.7 -c _imaging.c -o build/temp.macosx-10.4-x86_64-2.7/_imaging.o
    ...
    /usr/bin/cc -L/usr/local/Cellar/readline/6.2.1/lib -bundle -undefined dynamic_lookup -L/usr/local/Cellar/readline/6.2.1/lib build/temp.macosx-10.4-x86_64-2.7/_imagingmath.o -L/usr/local/lib -L/Users/haoqili/.virtualenvs/playdoh/lib -L/usr/lib -o build/lib.macosx-10.4-x86_64-2.7/_imagingmath.so
    --------------------------------------------------------------------
    PIL 1.1.7 SETUP SUMMARY
    --------------------------------------------------------------------
    version       1.1.7
    platform      darwin 2.7.1 (r271:86832, Jun  6 2011, 13:57:48)
                  [GCC 4.2.1 (Apple Inc. build 5666) (dot 3)]
    --------------------------------------------------------------------
    --- TKINTER support available
    *** JPEG support not available
    --- ZLIB (PNG/ZIP) support available
    *** FREETYPE2 support not available
    *** LITTLECMS support not available
    --------------------------------------------------------------------
    To add a missing option, make sure you have the required
    library, and set the corresponding ROOT variable in the
    setup.py script.
    
    To check the build, run the selftest.py script.
    changing mode of build/scripts-2.7/pilconvert.py from 644 to 755
    changing mode of build/scripts-2.7/pildriver.py from 644 to 755
    changing mode of build/scripts-2.7/pilfile.py from 644 to 755
    changing mode of build/scripts-2.7/pilfont.py from 644 to 755
    changing mode of build/scripts-2.7/pilprint.py from 644 to 755
    
    changing mode of /Users/haoqili/dev/playdoh/playdoh/playdoh/vendor-local/bin/pilconvert.py to 755
    changing mode of /Users/haoqili/dev/playdoh/playdoh/playdoh/vendor-local/bin/pildriver.py to 755
    changing mode of /Users/haoqili/dev/playdoh/playdoh/playdoh/vendor-local/bin/pilfile.py to 755
    changing mode of /Users/haoqili/dev/playdoh/playdoh/playdoh/vendor-local/bin/pilfont.py to 755
    changing mode of /Users/haoqili/dev/playdoh/playdoh/playdoh/vendor-local/bin/pilprint.py to 755
Successfully installed pil
Cleaning up...

2. Trying to get JPEG to be supported:

playdoh/vendor-local (master)$ brew install jpeg
Warning: Xcode is not installed! Builds may fail!
==> Downloading http://www.ijg.org/files/jpegsrc.v8c.tar.gz
######################################################################## 100.0%
==> ./configure --prefix=/usr/local/Cellar/jpeg/8c --disable-dependency-tracking
==> make install
/usr/local/Cellar/jpeg/8c: 17 files, 1.6M, built in 13 seconds

3. Rebuild PIL

)$ pip install PIL==1.1.7 --upgrade
Downloading/unpacking PIL==1.1.7
  Downloading PIL-1.1.7.tar.gz (506Kb): 506Kb downloaded
  Running setup.py egg_info for package PIL
    WARNING: '' not a valid package name; please use only.-separated package names in setup.py
    WARNING: '' not a valid package name; please use only.-separated package names in setup.py
    
Installing collected packages: PIL
  Found existing installation: PIL 1.1.7
    Uninstalling PIL:
      Successfully uninstalled PIL
  Running setup.py install for PIL
    WARNING: '' not a valid package name; please use only.-separated package names in setup.py
    WARNING: '' not a valid package name; please use only.-separated package names in setup.py
    --- using frameworks at /System/Library/Frameworks
    building '_imaging' extension
    /usr/bin/cc -fno-strict-aliasing -O3 -w -pipe -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -DHAVE_LIBJPEG -DHAVE_LIBZ -I/System/Library/Frameworks/Tcl.framework/Headers -I/System/Library/Frameworks/Tk.framework/Headers -IlibImaging -I/Users/haoqili/.virtualenvs/playdoh/include -I/usr/local/include -I/usr/include -I/usr/local/Cellar/python/2.7.1/include/python2.7 -c _imaging.c -o build/temp.macosx-10.4-x86_64-2.7/_imaging.o
    ...
    /usr/bin/cc -L/usr/local/Cellar/readline/6.2.1/lib -bundle -undefined dynamic_lookup -L/usr/local/Cellar/readline/6.2.1/lib build/temp.macosx-10.4-x86_64-2.7/_imagingmath.o -L/usr/local/lib -L/Users/haoqili/.virtualenvs/playdoh/lib -L/usr/lib -o build/lib.macosx-10.4-x86_64-2.7/_imagingmath.so
    --------------------------------------------------------------------
    PIL 1.1.7 SETUP SUMMARY
    --------------------------------------------------------------------
    version       1.1.7
    platform      darwin 2.7.1 (r271:86832, Jun  6 2011, 13:57:48)
                  [GCC 4.2.1 (Apple Inc. build 5666) (dot 3)]
    --------------------------------------------------------------------
    --- TKINTER support available
    --- JPEG support available
    --- ZLIB (PNG/ZIP) support available
    *** FREETYPE2 support not available
    *** LITTLECMS support not available
    --------------------------------------------------------------------
    To add a missing option, make sure you have the required
    library, and set the corresponding ROOT variable in the
    setup.py script.
    
    To check the build, run the selftest.py script.
    changing mode of build/scripts-2.7/pilconvert.py from 644 to 755
    changing mode of build/scripts-2.7/pildriver.py from 644 to 755
    changing mode of build/scripts-2.7/pilfile.py from 644 to 755
    changing mode of build/scripts-2.7/pilfont.py from 644 to 755
    changing mode of build/scripts-2.7/pilprint.py from 644 to 755
    
    changing mode of /Users/haoqili/.virtualenvs/playdoh/bin/pilconvert.py to 755
    changing mode of /Users/haoqili/.virtualenvs/playdoh/bin/pildriver.py to 755
    changing mode of /Users/haoqili/.virtualenvs/playdoh/bin/pilfile.py to 755
    changing mode of /Users/haoqili/.virtualenvs/playdoh/bin/pilfont.py to 755
    changing mode of /Users/haoqili/.virtualenvs/playdoh/bin/pilprint.py to 755
Successfully installed PIL
Cleaning up...

My Project's Packages

Todo:

1. update my own vendor with git clone --recursive git://github.com/mozilla/playdoh-lib.git ./vendor from playdoh doc

2. clean up my imports

3. clean up my vendor-local from virutal enviorments below from /Users/haoqili/.virtualenvs/playdoh/lib/python2.7/site-packages

haoqili@host-3-248:15:44:10:~/.virtualenvs/playdoh/lib/python2.7/site-packages$ ls -l
total 1016
drwxr-xr-x  11 haoqili  staff     374 Jun 15 10:39 Crypto
drwxr-xr-x   4 haoqili  staff     136 Jun 30 09:41 Django-1.3-py2.7.egg
drwxr-xr-x  10 haoqili  staff     340 Jun  6 14:58 Jinja2-2.5.5-py2.7.egg-info
drwxr-xr-x   7 haoqili  staff     238 Jun  6 14:58 MySQL_python-1.2.3c1-py2.7.egg-info
drwxr-xr-x  15 haoqili  staff     510 Jun  6 14:58 MySQLdb
drwxr-xr-x  12 haoqili  staff     408 Jun 15 11:44 OpenSSL
drwxr-xr-x   4 haoqili  staff     136 Jun 15 10:09 Twisted-11.0.0_r32122-py2.7-macosx-10.4-x86_64.egg
-rwxr-xr-x   1 haoqili  staff   18328 Jun  6 14:58 _hashlib.so
-rwxr-xr-x   1 haoqili  staff   70072 Jun  6 14:58 _mysql.so
-rw-r--r--   1 haoqili  staff    2306 Jun  6 14:57 _mysql_exceptions.py
-rw-r--r--   1 haoqili  staff    4559 Jun  6 14:58 _mysql_exceptions.pyc
drwxr-xr-x   5 haoqili  staff     170 Jun 14 16:40 backports
drwxr-xr-x   7 haoqili  staff     238 Jun 14 16:40 backports.ssl_match_hostname-3.2a3-py2.7.egg-info
drwxr-xr-x   5 haoqili  staff     170 Jun  6 14:54 bcrypt
-rw-r--r--   1 haoqili  staff      48 Jun 13 12:00 bleach.egg-link
-rw-r--r--   1 haoqili  staff    6032 Jul  6 13:44 django_csp-1.0.1-py2.7.egg
-rw-r--r--   1 haoqili  staff    6854 Jun 30 09:41 django_ratelimit-0.1-py2.7.egg
drwxr-xr-x   7 haoqili  staff     238 Jun 29 19:46 django_registration-0.7-py2.7.egg-info
-rw-r--r--   1 haoqili  staff     509 Jul  6 13:44 easy-install.pth
drwxr-xr-x   7 haoqili  staff     238 Jun  6 14:58 hashlib-20081119-py2.7.egg-info
-rw-r--r--   1 haoqili  staff    4993 Jun  6 14:58 hashlib.py
-rw-r--r--   1 haoqili  staff    4452 Jun  6 14:58 hashlib.pyc
drwxr-xr-x   9 haoqili  staff     306 Jun  6 14:58 hmac-20101005-py2.7.egg-info
-rw-r--r--   1 haoqili  staff    4531 Jun  6 14:57 hmac.py
-rw-r--r--   1 haoqili  staff    4892 Jun  6 14:58 hmac.pyc
drwxr-xr-x  24 haoqili  staff     816 Jun 13 12:00 html5lib
drwxr-xr-x   7 haoqili  staff     238 Jun 13 12:00 html5lib-0.90-py2.7.egg-info
drwxr-xr-x  49 haoqili  staff    1666 Jun  6 14:58 jinja2
drwxr-xr-x   4 haoqili  staff     136 Jun  6 14:23 pip-1.0.1-py2.7.egg
drwxr-xr-x   8 haoqili  staff     272 Jun 15 11:44 pyOpenSSL-0.12-py2.7.egg-info
drwxr-xr-x   7 haoqili  staff     238 Jun  6 14:54 py_bcrypt-0.2-py2.7.egg-info
-rw-r--r--   1 haoqili  staff     629 Jun 15 10:46 pycrypto-2.0.1-py2.7.egg-info
-rw-r--r--   1 haoqili  staff    8399 Jun 29 20:31 recaptcha_client-1.0.6-py2.7.egg
drwxr-xr-x  18 haoqili  staff     612 Jun 29 19:46 registration
-rw-r--r--   1 haoqili  staff  332005 Jun  6 14:12 setuptools-0.6c11-py2.7.egg
-rw-r--r--   1 haoqili  staff      30 Jun 15 10:09 setuptools.pth
drwxr-xr-x   4 haoqili  staff     136 Jun 15 10:09 zope.interface-3.6.3-py2.7-macosx-10.4-x86_64.egg

Overview of Project

Web app security training sildes

Setting up OWASP play ground

Set up Playdoh and creating virtual enviroments

Mozilla

Playdoh < Django < Virtual Enviorment < MacBook

Playdoh is usually set us in ~/dev
Setup directions.

Set up another virtual environment like playdoh in dev, first use https://gist.github.com/1011209, then do mkvirtualenv [project name]

Set up adium chat for encrypted messages.