Introduction to Django - presentation

Published at March 18, 2013 | Tagged with:

This presentation shows the basics of Django - what is inside the framework and explains the Model-View-Template system. One of the most important parts is a diagram how the request is processed and the response is generated. Shows the project and the application structure and the basic elements - Models, URLs dispatcher, Views and Templates.

Introduction to django from Ilian Iliev

Simple Site Checker and the User-Agent header

Published at Oct. 22, 2012 | Tagged with: , , , ,

Preface: Nine months ago(I can't believe it was that long) I created a script called Simple Site Checker to ease the check of sitemaps for broken links. The script code if publicly available at Github. Yesterday(now when I finally found time to finish this post it must be "A few weeks ago") I decided to run it again on this website and nothing happened - no errors, no warning, nothing. Setting the output level to DEBUG showed the following message "Loading sitemap ..." and exited.
Here the fault was mine, I have missed a corner case in the error catching mechanism i.e. when the sitemap URL returns something different from "200 OK" or "500 internal server error". Just a few second and the mistake was fix.

Problem and Solution: I ran the script again and what a surprise the sitemap URL was returning "403 Forbidden". At the same time the sitemap was perfectly accessible via my browser. After some thinking I remembered about that some security plugins block the access to the website if there is not User-Agent header supplied. The reason for this is to block the access of simple script. In my case even an empty User-Agent did the trick to delude the plugin.

urllib2.urlopen(urllib2.Request(url,
                                headers={'User-Agent': USER_AGENT}))

Final words: As a result of the issue mention above one bug in simple site checker was found fixed. At the same time another issue about missing status and progress was raised, more details can be found at Github but in a few words an info message was added to each processed URL to indicate the progress.

If you have any ideas for improvement or anything else feel free to comment, create issues and/or fork the script.

Automation, Fabric and Django - presentation

Published at Oct. 3, 2012 | Tagged with: , , , , ,

As a follow up post of Automated deployment with Ubuntu, Fabric and Django here are the slides from my presentation on topic "Automation, Fabric and Django". Unfortunately there is no audio podcast but if there is interest I can add some comments about each slide as part of this post.

Automation - fabric, django and more from Ilian Iliev

If there is anything that need explanation feel free to ask.

Django project file structure

Published at Sept. 24, 2012 | Tagged with: , ,

As I promised in Automated deployment with Ubuntu, Fabric and Django I will use this post to explain the file structure that I use for my Django projects and what I benefit from it. So here is my project directory tree.

The structure

~/workspace/<project_name>/
|-- bin
|-- include
|-- lib
|-- local
|-- src
   |-- .git
   |-- .gitignore
   |-- required_packages.txt
   |-- media
   |-- static
   |-- <project_name>
   |   |-- <project_name>
   |   |   |-- __init__.py
   |   |   |-- settings
   |   |   |   |-- __init__.py
   |   |   |   |-- <environment_type>.py
   |   |   |   |-- local.py
   |   |   |-- templates
   |   |   |-- urls.py
   |   |   |-- views.py
   |   |-- manage.py
   |   |-- wsgi.py
   |-- <project_name>.development.nginx.local.conf
   |-- <project_name>.< environment_type>.nginx.uwsgi.conf
   |-- <project_name>.< environment_type>.uwsgi.conf

Explanation

At the top I have a directory named as the project and virtual environment inside of it. The benefit from it is complete isolation of the project from the surrounding projects and python packages installed at OS level and ability to install packages without administrator permissions. It also provides an easy way to transfer the project from one system to another using a requirements file.
The src folder is where I keep everything that is going to enter the version control project source, requirements files, web server configuration etc.
My default .gitignore is made to skip the pyc-files, the PyDev files and everything in the static and media directories.
The media directory is where the MEDIA_ROOT settings point to, respectively static is for the STATIC_ROOT.
All required packages with their version are placed in required_packages.txt so we can install/update them with a single command in the virtual environment.
In a directory with the name of the project is where the python code resides. Inside it the project structure partly follows the new project layout introduced in Django 1.4.
The big difference here is the settings part. It is moved as a separate module where all common/general settings are place in __init__.py

. We have one file for each possible environment(development, staging, production) with the environment specific settings like DEBUG and TEMPLATE_DEBUG. In local.py are placed the settings specific for the machine where the project is running. This file should not go into the repository as this is where the access database password and API keys should reside.
I having one local Nginx configuration file because I use the webserver to serve the static files when working locally this is the <project_name>.development.nginx.local.conf.
For each environment there is also a couple of configuration files - one for the web server(<project_name>.< environment_type>.nginx.uwsgi.conf) and one for the uwsgi(<project_name>.< environment_type>.uwsgi.conf). I make symbolic links pointing to these files so any changes made are automatically pushed/pulled via version control, I only have to reload the configuration. There is no option to change something in the configuration at one station and to forget to transfer it to the rest.

Benefits

  • Complete isolation from other projects using virtual environment
  • Easy to transfer and update packages on other machines thanks to the pip requirements file
  • Media/static files outside the source directory for higher security
  • Web server/uWSGI configuration as part of the repository for easier and error proof synchronization

Probably it have some downside(I can not think any of these now but you never know) so if you think that this can be improved feel free to share your thoughts. Also if there is anything not clear enough, just ask me, I will be happy to clear it.

Automated deployment with Ubuntu, Fabric and Django

Published at Sept. 18, 2012 | Tagged with: , , , , , ,

A few months ago I started to play with Fabric and the result was a simple script that automates the creation of a new Django project. In the last months I continued my experiments and extended the script to a full stack for creation and deployment of Django projects.

As the details behind the script like the project structure that I use and the server setup are a bit long I will keep this post only on the script usage and I will write a follow up one describing the project structure and server.

So in a brief the setup that I use consist of Ubuntu as OS, Nginx as web server and uWSGI as application server. The last one is controlled by Upstart. The script is available for download at GitHub.
In a wait for more detailed documentation here is a short description of the main tasks and what they do:

startproject:<project_name>

  • Creates a new virtual environment
  • Installs the predefined packages(uWSGI, Django and South))
  • Creates a new Django project from the predefined template
  • Creates different configuration files for development and production environment
  • Initializes new git repository
  • Prompts the user to choose a database type. Then installs database required packages, creates a new database and user with the name of the project and generates the local settings file
  • Runs syncdb(if database is selected) and collectstatic

setup_server

  • installs the required packages such as Nginx, PIP, GCC etc
  • prompts for a database type and install it
  • reboots the server as some of the packages may require it

Once you have a ready server and a project just use this the following task to deploy it to the server. Please have in mind that it should be used only for initial deployment.

deploy_project:<project_name>,<env_type>,<project_repository>

  • Creates a new virtual environment
  • Clones the project from the repository
  • Installs the required packages
  • Creates symbolic links to the nginx/uwsgi configuration files
  • Asks for database engine, creates new DB/User with the project name and updates the settings file
  • Calls the update_project task

update_project:<project_name>,<env_type>

  • Updates the source code
  • Installs the required packages(if there are new)
  • Runs syncdb, migrate and collect static
  • Restart the Upstart job that runs the uWSGI and the Nginx server

The script is still in development so use it on your own risk. Also it reflects my own idea of server/application(I am planning to describe it deeper in a follow up post) setup but I would really like if you try it and give a feedback. Feel free to fork, comment and advice.

The road to hell is paved with regular expressions ...

Published at July 31, 2012 | Tagged with: , ,

... or what is the cost of using regular expressions for simple tasks

Regular expressions are one of the most powerful tools in computing I have ever seen. My previous post about Django compressor and image preloading is a good example how useful they might be. The only limit of their use is your imagination. But "with great power, comes great responsibility" or in this case a great cost. Even the simplest expressions can be quite heavy compared with other methods.

The reason to write about this is a question recently asked in a python group. It was about how to get the elements of a list that match specific string. My proposal was to use comprehension list and simple string comparison while other member proposed using a regular expression. I was pretty sure that the regular expression is slower but not sure exactly how much slower so I made a simple test to find out.

import re
import timeit

my_list = ['abc-123', 'def-456', 'ghi-789', 'abc456', 'abc', 'abd']

def re_check():
    return [i for i in my_list if re.match('^abc$', i)]

t = timeit.Timer(re_check)
print 're_check result >>', re_check()
print "%.2f usec/pass" % (1000000 * t.timeit(number=100000)/100000)

def simple_check():
    return [i for i in my_list if i=='abc']

t = timeit.Timer(simple_check)
print 'simple_check result >>', simple_check()
print "%.2f usec/pass" % (1000000 * t.timeit(number=100000)/100000)

The results was 23.99 vs 1.41 usec/pass respectively for regular expression vs direct comparison i.e. the regexp was 17 times slower. The difference from the example above may be OK in some cases but it rises with the size of the list. This is a simple example how something really quick on local version may take significant time on production and even to broke your application.

So, should you learn and use regular expressions?
Yes! Absolutely!

They are powerful and useful. They will open your mind and allow you to do things you haven't done before. But remember that they are a double-edge razor and should be used cautiously. If you can avoid it with other comparison(one or more) just run a quick test to see whether it will be faster. Of course if you can not avoid it you can also think about caching the results.

Django compressor and image preloading

Published at July 30, 2012 | Tagged with: , , , , , ,

Preface: Have you noticed how on some websites when you click on a link that opens a lightbox or any overlay for first time it takes some time to display the border/background/button images. Not quite fancy, right?
This is because the load of this images starts at the moment the overlay is rendered on the screen. If this is your first load and these images are not in your browser cache it will take some time for the browser to retrieve them from the server.

Solution: The solution for this is to preload the images i.e. to force the browser to request them from the server before they are actually used. With a simple javascript function and a list of the images URLs this is a piece of cake:

$.preLoadImages = function() {
    var args_len = arguments.length;
    for (var i=0; i < args_len; i++) {
        var cacheImage = document.createElement('img');
        cacheImage.src = arguments[i];
    }
}

$.preLoadImages('/img/img1.png', '/img/img2.png')
Please have in mind that the code above uses the jQuery library. Specialty: Pretty easy, but you have to hardcode the URLs of all images. Also if you are using Django compressor then probably you are aware that it adds extra hash to the URLs of the images in the compressed CSS files. The hash depends from the COMPRESS_CSS_HASHING_METHOD settings and can not be avoided. It is pretty useful cause it forces the client browser to reload the images every time when something has been changed. unfortunately our hardcoded list of URLs does not have this hash. So wouldn't it be much simpler if instead of hardcoding URLs we just read them from the CSS files? Solution 2:
$.preLoadImages = function() {
    $.get($('link[rel="stylesheet"]')[0].href, function(data){
        r = /url\(['|"]?(\S+\.(gif|jpg|jpeg|png)[^'(]*)['|"]?\)/ig;
        while (match = r.exec(data)){
                var cacheImage = document.createElement('img');
                cacheImage.src = match[1];
            }
        });
}

$.preLoadImages()

Now with the help of regular expressions we can read the image URLs directly from the CSS file together with the hash part. Please note the zero index in the css file selector, if your main CSS is not the first declared style-sheet then you will have to change the index according to its position.

I hope you will find this solutions simple and useful. As always feel free to comment, share and propose code improvements.

Python is not a Panacea ...

Published at June 11, 2012 | Tagged with: , , , ,

... neither is any other language or framework

This post was inspired by the serial discussion on the topic "Python vs other language"(in the specific case the other one was PHP, and the question was asked in a Python group so you may guess whether there are any answers in favor of PHP). It is very simple, I believe that every Python developer will tell you that Python is the greatest language ever build, how easy is to learn it, how readable and flexible it is, how much fun it is to work with it and so on. They will tell you that you can do everything with it: web and desktop development, testing, automation, scientific simulations etc. But what most of them will forgot to tell you is that it is not a Panacea.

In the matter of fact you can also build "ugly" and unstable applications in Python too. Most problems come not from the language or framework used, but from bad coding practices and bad understanding of the environment. Python will force you to write readable code but it wont solve all your problems. It is hard to make a complete list of what exactly you must know before starting to build application, big part of the knowledge comes with the experience but here is a small list of some essential things.

  • Write clean with meaningful variable/class names.
  • Exceptions are raised, learn how to handle them.
  • Learn OOP(Object Oriented Programming)
  • Use functions to granulate functionality and make code reusable
  • DRY(Don't Repeat Youtself)
  • If you are going do develop web application learn about the Client-Server relation
  • Use "layers" to seprate the different parts of your application - database methods, business logic, output etc. MVC is a nice example of such separation
  • Never store passwords in plain text. Even hashed password are not completely safe, check what Rainbow Tables are.
  • Comment/Document your code
  • Write unit test and learn TDD.
  • Learn how to use version control.
  • There is a client waiting on the other side - don't make him wait too long.
  • Learn functional programming.

I hope the above does not sounds as an anti Python talk. This is not its idea. Firstly because there are things that are more important than the language itself(the list above) and secondly because... Python is awesome )))
There are languages that will help you learn the things above faster, Python is one of them - built in documentation features, easy to learn and try and extremely useful. My advice is not to start with PHP as your first programming language it will make you think that mixing variables with different types is OK. It may be fast for some things but most of the times it is not safe so you should better start with more type strict language where you can learn casting, escaping user output etc.

Probably I have missed a few(or more) pointa but I hope I've covered the basics. If you think that anything important is missing, just add it in the comments and I will update the post.

⋘ Previous page Next page ⋙