ilian.io

What is Enterprise Software?

Thu, 19 Feb 2026 00:00:00 +0000

I was a guest on Гласът на IT индустрията (The Voice of the IT Industry) to talk about what enterprise software is and what makes it different — how releases are planned and distributed, and how you debug production problems when you have no direct access to the environment.

Video (in Bulgarian):

From Await Chains to Pipelines Using Queues

Tue, 27 Jan 2026 00:00:00 +0000

I gave this talk at Python Meetup Sofia, showing how to refactor complex chains of async/await calls into cleaner, more scalable queue-based pipelines using Python's asyncio.

Slides:

The code is available on GitHub.

TLS Demystified: Certificates, Keys, and Trust in Practice

Sun, 19 Oct 2025 00:00:00 +0000

My talk from OpenFest 2025 demystifying TLS — how certificates, keys, and trust chains work in practice, making PKI concepts accessible and actionable for everyday engineers.

Video (in Bulgarian):

Check the slides here

Interfaces, Testing and Mocking in Go

Tue, 03 Dec 2024 00:00:00 +0000

Interfaces are abstraction types that define a set of methods and their signatures without providing any implementation — think of them as a contract your module should follow. In this screencast, I show how to use them in Go to create decoupled modules that are easy to test, refactor, and extend.

The code is available on GitHub.

How to use Generics in Go

Mon, 11 Nov 2024 00:00:00 +0000

Several years after their release, Generics in Go are still a bit misunderstood and avoided by many developers. In this screencast, I show how to use them and how they can help you write better code.

The code is available on GitHub.

If you are interested in how generics work under the hood, check out this talk from FOSDEM 2025.

From Django to Gin

Tue, 27 Feb 2024 00:00:00 +0000

I gave this talk at Django Bulgaria Meetup, comparing the experience of building a web service in Django (Python) versus Gin (Go) — looking at routing, ORM vs. direct database access, and ecosystem differences.

Slides:

The code is available on GitHub.

Talks

Mon, 01 Jan 2024 00:00:00 +0000

2026

From Await Chains to Pipelines Using Queues — Python Meetup Sofia

2025

TLS Demystified: Certificates, Keys, and Trust in Practice — OpenFest 2025 (video in Bulgarian)

2024

From Django to Gin — Django Bulgaria Meetup

2023

Debugging in Python — Openfest 2023
From Architecture to Release — Dev Challenge Accepted 2023 (video)
Data Validation in Python — DEV.BG Python user group

2022

Beyond the for-loop — Dev Challenge Accepted 2022 (video)

2021

Lessons Learned The Hard Way — HackConf 2021 (video)
Python and gRPC — DEV.BG Python user group

2019

Mocking in Python Workshop — HackConf 2019

2018

Django Rest Framework Workshop — HackConf 2018

2017

Django and Working with Large Database Tables — Django Stockholm Meetup
Platform Horror Stories — Stockholm Python Meetup

2016

Things We Tend to Overlook Going from Architecture to Release

Thu, 16 Nov 2023 00:00:00 +0000

InfoQ covered my Dev Challenge Accepted talk, where Ben Linders wrote up the key points about what software teams tend to overlook on the way from architecture to shipping.

The piece touches on how oversights happen not from ignorance, but because teams don't think about certain aspects daily — things like metrics, monitoring, and API documentation tend to slip through the cracks. It also covers the challenges of managing coupling with existing codebases, handling compatibility across multiple client versions, and why relying on tribal knowledge instead of documentation is a hidden risk. The broader recommendation is about creating an environment where everyone — product owners, architects, QA — feels safe to raise concerns early, before small gaps compound into bigger problems.

Read the full article on InfoQ →

Debugging in Python

Sat, 04 Nov 2023 00:00:00 +0000

I presented this at Openfest 2023 — a practical look at Python debugging, from print statements and logging to pdb, breakpoints, and IDE debuggers, with code examples in the companion repository.

Slides:

The code examples are available on GitHub.

From Architecture to Release

Sat, 30 Sep 2023 00:00:00 +0000

A talk I gave at Dev Challenge Accepted 2023 about the things we often forget or ignore between designing a system and shipping it — the practices, trade-offs, and steps that fall between architecture and release.

Video:

Check the slides here

Data Validation in Python

Thu, 08 Jun 2023 00:00:00 +0000

A webinar I gave at the DEV.BG Python user group comparing Python's data validation approaches — Pydantic, Django Forms and jsonschema — with practical examples and a repository to follow along.

Slides:

The code examples are available on GitHub.

Beyond the for-loop

Fri, 09 Sep 2022 00:00:00 +0000

I gave this talk at Dev Challenge Accepted 2022, exploring Python's functional-style tools — map, filter, reduce, comprehensions, and generators — as expressive alternatives to writing everything as an explicit loop.

Video:

Check the slides here

Lessons Learned The Hard Way

Fri, 22 Oct 2021 00:00:00 +0000

A talk I gave at HackConf 2021 sharing concrete lessons from real production experience — the subtle bugs, unexpected side effects, and costly mistakes you really don't want to discover for yourself.

Video:

Check the slides here

Python and gRPC

Thu, 14 Oct 2021 00:00:00 +0000

The slides and demo code from my presentation in front of the DEV.BG Python user group, introducing gRPC with Python — covering Protocol Buffers, service definitions, and building a working client-server example from scratch.

Slides:

The demo code is available on GitHub.

Mocking in Python Workshop

Fri, 11 Oct 2019 00:00:00 +0000

The slides from the Mocking in Python workshop I held at HackConf 2019, covering unittest.mock, patching strategies, and writing isolated unit tests with confidence.

Slides:

The workshop code is available on GitHub.

Django Rest Framework Workshop

Fri, 14 Sep 2018 00:00:00 +0000

A hands-on workshop I gave at HackConf 2018, covering REST fundamentals and building APIs with Django REST Framework, with exercises in the companion repository.

Slides:

The repo with the code for the workshop is available on GitHub.

Django and working with large database tables

Sun, 02 Apr 2017 00:00:00 +0000

The slides from my presentation for the Django Stockholm Meetup group. Contains small comparison between MySQL and PostgreSQL in terms of performance when modigying the tables structure.

Django Admin and Celery

Tue, 28 Feb 2017 00:00:00 +0000

A weird race condition

Updated solution, please check below!

Some background: We have a model that is edited only via the Django admin. The save method of the model fires a celery task to update several other records. The reason for using celery here is that the amount of related objects can be pretty big and we decided that it is best to spawn the update as a background job. We have unit and integration tests, the code was also manually tested, everything looked nice and we deployed it.

On the next day we found out that the code is acting weird. No errors, everything looked like it has worked but the updated records actually contained the old data before the change. The celery task accepts the object ID as an argument and the object is fetched from the database before doing anything else so the problem was not that we were passing some old state of the object. Then what was going on?

Trying to reproduce it: Things are getting even weirder. The issues is happening every now and then.

Hm...? Race condition?! Let's take a look at the code:

class MyModel(models.Model):

 def save(self, **kwargs):
 is_existing_object = False if self.pk else True

 super(MyModel, self).save(**kwargs)

 if is_existing_object:
 update_related_objects.delay(self.pk)

So the celery task is called after the "super().save()" and the changes should be already stored to the database. Unless...

The bigger picture: (this is simplified version of what is happening for the full one check Django's source)

 def changeform_view(...):
 with transaction.atomic():
 ...
 self.save_model(...) # here happens the call to MyModel.save()
 self.save_related(...)
 ...

Ok, so the save is actually wrapped in transaction. This explains what is going on. Before the transaction is committed the updated changes are not available for the other connections. This way when the celery task is called we end up in a race condition whether the task will start before or after the transaction is completed. If celery manages to pick the task before the transaction is committed it reads the old state of the object and here is the error.

Solution (updated): The best solution is to use transaction.on_commit. This way the call to the celery task will be executed only after the transaction is completed succesfully. Also, if you call the method outside of transaction the function will be executed immediately so it will also work if you are saving the model outside the admin. The only downside is that this functionality has been added to Django in version 1.9. So it wasn't an option for us. Still, special thanks to Jordan Jambazov for pointing this approach to me, I'll definitely use it in the future.

Unfortunately we are using Django 1.8 so we picked a quick and ugly fix. We added a 60 seconds countdown to the task call giving the transaction enough time to complete. As the call to the task depends on some logic and which properties of the models instance are changes moving it out of the save method was a problem. Another option could be to pass all the necessary data to the task itself but we decided that it will make it too complicated.

However I am always open to other ideas so if you have hit this issue before I would like to know how you solved it.

Platform Horror Stories

Sun, 01 Jan 2017 00:00:00 +0000

A talk I gave at the Stockholm Python Meetup in January 2017, sharing real-world horror stories from platform engineering — the kind of problems you don't see coming until they're already causing trouble.

Slides:

SQLAlchemy and "Lost connection to MySQL server during query"

Sun, 24 Jul 2016 00:00:00 +0000

... a weird behaviour when pool_recycle is ignored

Preface: This is quite a standard problem for apps/websites with low traffic or those using heavy caching and hitting the database quite seldom. Most of the articles you will find on the topic will tell you one thing - change the wait_timeout setting in the database. Unfortunately in some of the cases this disconnect occurs much earlier than the expected wait_timeout (default ot 8 hours). If you are in one of those cases keep reading.

This issue haunted our team for weeks. When we first faced it the project that was raising it was still in dev phase so it wasn't critical but with getting closer to the release data we started to search for solution. We have read several articles and decided that pool_recycle is our solution.

Adding pool_recycle: According to SQL Alchemy's documentation pool_recycle "causes the pool to recycle connections after the given number of seconds has passed". Nice, so if you recycle the connection in intervals smaller that the await_timeout the error above should not appear. Let's try it out:

import time

from sqlalchemy.engine import create_engine


url = 'mysql+pymysql://user:pass@127.0.0.1:3306/db'
engine = create_engine(url, pool_recycle=1).connect()

query = 'SELECT NOW();'

while True:
 print('Q1', engine.execute(query).fetchall())
 engine.execute('SET wait_timeout=2')
 time.sleep(3)
 print('Q2', engine.execute(query).fetchall())

So what does the code do - we create a connection to a local MySQL server and state that it should be recycled every second(line 7). Then we execute a simple query (line 12) just to verify that the connection is working.
We set the wait_timeout to 2 second and wait for 3. At this stage the connection to the server will timeout, but SQL Alchemy should recycle it, so the last query should be executed successfully and the loop should continue.
Unfortunately the results looks like:

sqlalchemy.exc.OperationalError: (pymysql.err.OperationalError)
(2013, 'Lost connection to MySQL server during query') [SQL: 'SELECT NOW();']

Wait, what happened, why is not the connection recycled?

Solution: Well, as with all such problems the solution was much simpler compared to the time it took us to find it (we fought with this for days). The only change that solved it was on line 7:

engine = create_engine(url, pool_recycle=1)

# the result
Q1 [(datetime.datetime(2016, 7, 24, 20, 51, 41),)]
Q2 [(datetime.datetime(2016, 7, 24, 20, 51, 44),)]
Q1 [(datetime.datetime(2016, 7, 24, 20, 51, 44),)]
Q2 [(datetime.datetime(2016, 7, 24, 20, 51, 47),)]

Have you spot the difference? We are not calling the connect() method any more.

Final words: To keep it honest, I don't know why this solved the issue. Hopefully someone more familiar with SQL Alchemy will come with a reasonable explanation for it. The bad part is that the examples in the official docs are using "connect". So it is either a bug or a bad documentation. I will send this article to SQL Alchemy's Twitter account so hopefully we will see some input from them. Till then, if any of you have an idea explanation about the behaviour I'll be happy to hear it.

Service Discovery with Smartstack

Sun, 03 Jul 2016 00:00:00 +0000

A week of innovation

First of all I want to thank Lifesum for having another "Innovation Week", it is a great opportunity and I hope that more companies will start following it. In a few words the idea is to allow everyone from the company to freely pick a project or idea that they want to develop and and work on it for one week. The benefits range from just making people happy because of the break of the routines and the opportunity to work on something a bit different, to seeing some pretty amazing prototypes that can be easily implemented in the company product.

What is Service Discovery?

In summary service discovery is the possibility of the separate services in scalable infrastructure to communicate with each other and to the outside world. In other words - how to route the requests to the corresponding service while providing balanced load on the instances in the pool and monitoring their health. Sounds simple, right?

Well, unfortunately service discovery in the real world is not that simple. In my presentation from the Stockholm Python MeetUp I talked a bit more about the complexity of service discovery, the suboptimal solutions and Smartstack - a solutions invented at AirBnB for simplifying the whole process. You can see more about it in my presentation:

However, the whole idea sounded so awesome that me and my colegue Esma decided to team up on that and try to explore a bit more the opportunities that Smarstack provides for us. In the matter of fact we decided to explore two different approaches: Smartstack and Consul. However we had some issue with the Consul setup and we found that it is not acting exactly the way we so at the end we focused all our attention to Smarstack.

How does it work?

Smartstack consists of two main components - Nerve and Synapse. Never handles the service registration while Synapse reads the information about the available services and configures a local HAProxy that plays as a load balancer for the service pool. For our tests we used Zookeeper as register for the services.

What have we built?

We created a small project consisting of a pool of Zookeeper instances, two node for service A and one node for service B. We tested multiple scenarios of crashes of one or more nodes, both zookepers instances and service instances, how the systems operates during the crashes and how it recovers after the nodes are brought back on.

Results

As a result we created a public repository service-discovery giving details about the whole setup process, our tests and their results. Actually now is the time to praise Esma and her awesome work on conducting the tests and writing the great documentation from the repo above. So, if you have ever wondered about how service discovery works or you just want to test Smartstack, just clone the repo and follow the instruction. We would be happy to hear about your experience.

Dynamic Plans: Connecting Backend and Mobile

Thu, 16 Jun 2016 00:00:00 +0000

A talk I gave at the Stockholm Python Meetup hosted at Lifesum, covering how to use dynamic plans in Django to drive mobile app behaviour from the backend.

Slides:

Django interview questions ...

Sun, 24 May 2015 22:41:24 +0000

... and some answers

Well I haven't conducted any interviews recently but this one has been laying in my drafts for a quite while so it is time to take it out of the dust and finish it. As I have said in Python Interview Question and Answers these are basic questions to establish the basic level of the candidates.

Django's request response cycle
You should be aware of the way Django handles the incoming requests - the execution of the middlewares, the work of the URL dispatcher and what should the views return. It is not necessary to know everything in the tiniest detail but you should be generally aware of the whole picture. For reference you can check "the live of the request" slide from my Introduction to Django presentation.
Middlewares - what they are and how they work
Middlewares are one of the most important parts in Django. Not only because they are quite powerfull and useful but also because the lack of knowledge about their work can lead to hours of debugging. From my experience the process_request and process_response hooks are the most frequently used and those are the one I always ask for. You should also know their execution order and when their execution starts/stop. For reference check the official docs.
Do you write tests? How?
Having well tested code benefits you, the project and the rest of the team that is going to maintain/modify your code. Django's built-in test client, Django Webtest, Factory boy, Faker or whatever you use, I would like to hear about it and how you do it. There is not only one way/tool for it so I would like to know what are you using and how. You should show that you understand how important is to test your code and how to do it right.
select_related and prefetch_related
Django's ORM (as every other) can be both a blessing and a curse. Getting a foreign key property can easily lead you to executing million queries. Using select_related and prefetch_related can come as a saviour so it is important to know how to use them.
Forms validation
I expect from the candidates that they know how to build forms, to add custom validation for one field and how to validate fields that depend on each other.
Building a REST API with Django
The most popular choices out there are Django REST Framework and Django Tastypie. Have you used any of them, why an how. What issues have you faced? Here is the place to show that you understand the concept of REST APIs, the correct request/response types and the serialisation of the data.
Templates
Building website using the built-in template system is often part of your daily tasks. This means that you should understand the template inheritance, how to reuse the block and so on. Having experience with sekizai is a plus.

There are a lot of other things to ask about internationalisation(i18n), localisation(l10n), south/migrations etc. Take a look at the docs and you can see them explained pretty well.

PyCon Sweden 2015

Tue, 19 May 2015 04:47:59 +0000

In a few words PyCon Sweden 2015 was awesome. Honestly, this was my first Python conference ever but I really hope it won't be the last.

Outside the awesome talks and great organisation it was really nice to spend some time with similar minded people and talk about technology, the universe and everything else. I have met some old friends and made some new ones but lets get back to the talk. Unfortunately I was not able to see all of them but here is a brief about those I saw and found really interesting:

It all started with Ian Ozsvald and his awesome talk about "Data Science Deployed" (slides / video). The most important point here were:

log everything
think about data quality, don't use everything just what you need
think about turning data into business values
start using your data

Then Rebecca Meritz talked about "From Explicitness to Convention: A Journey from Django to Rails" (slides / video). Whether the title sounds a bit contradictive this was not the usual Django vs Rails talk. At least to me it was more like a comparison between the two frameworks, showing their differences, weak and strong sides. Whether I am a Django user, I am more and more keen to the position that none of the frameworks is better than the other one, they are just two different approaches for building great web apps.

Flavia Missi and "Test-Driven-Development with Python and Django" (slides / video). TDD will help you have cleaner, better structured and easy to maintian code. If you are not doing it the best moment to start is now. Whether it is hard at the beginning you will pretty soon realise how beneficial it is. Especially if someone pushed a bad commit and the tests saved your ass before the code goes to production.

Later Dain Nilsson talked about "U2F: Phishing-proof two-factor authentication for everyone" (video). Whether I don't use two-factor authentication at the moment I am familiar with the concept and I really like it. The U2F protocol looks like a big step towards making it more applicable over different tools and applications and the key holding devices are more and more accessible nowadays. Maybe it is time for me to get one )))

The second day started with Kate Heddleston who talked about ethics in computer programming (video). About how social networks can be used as a tool for ruining peoples lifes and that we as a developers should take a responsibility and work towards making the internet a safer place for everyone. A place where you can have your privacy and have protection if harassed. It is a big problem which won't be solved in a night, but talking about it is the first step towards solving it.

"Why Django Sucks" by Emil Stenström. Don't rush into raging cause this was one of best talks at the conference. Emil showed us the parts in Django where he sees flaws and that need improvement. The main point was the lack of common template language between Django and Javascript and what possible solutions are available or can be made.

Daniele Sluijters reminded us how easy is to work with "Puppet and Python" (video). No more fighting with Ruby, you can easily use your favorite language to build your own tools and helpers

Dennis Ljungmark and "Embedded Python in Practice". The last time I programmed embedded devices was 15 years ago as a part of short course in the high school. Dennis' work is much more complex than what I did then but his talk reminded me of things that are applicable to general programming. Whether using non-embedded systems we often have much more memory and processing power available that does not mean that we should waste it. So think when you code - RAM is not endless and processors are not that fast as we often wish. Also don't forget that Exceptions wil sooner or later occur so make your code ready to handle them.

"How to Build a Python Web Application with Flask and Neo4j" (video) by Nicole White. Well I have heard about Neo4J, but I have never used it or seen it in action so this one was really exciting. Neo4J offers you a whole new perspective about building databases and relations between objects but it is much far from panacea. Actually I can see it it is more like a special tool then as a general replacement of a relation database but it still worths to be tried. Oh, and the Neo4J browser - totally awesome.

In the lightning talks Tome Cvitan talked about Faker. If you are still not familiar with it now is a good time to try it. Especially if you are writing tests.

At the final Kenneth Reitz told us about "Python for Humans"(video). About the not that obvious things in Python and what solutions are out there. And also about The Hitchhiker’s Guide to Python! a great place for beginners and not only and shared the idea to make Python easier and more welcoming by introducing better tools for higher level of operations.

Finally I want to thank to everyone - the organisers, the speakers, the audience and practically everyone who was a part of the conference. Without you it would be the same (or be at all). Thanks, keep up the good work and hopefully we will sea each other again.

P.S. Have I mentioned that the whole conference was recorded on video so hopefully we will see be able to see all the talks pretty soon. I will try to keep this post updated with the links to the videos and/or slides when they become available. Of course if you know about any published slides from the conference that are not linked here please let me know.

The full set of videos are available at PyCon Sweden's Youtube channel.

Django, pytz, NonExistentTimeError and AmbiguousTimeError

Sun, 29 Mar 2015 23:02:44 +0000

Brief: In one of the project I work on we had to convert some old naive datetime objects to timezone aware ones. Converting naive datetime to timezone aware one is usually a straightforward job. In django you even have a nice utility function for this. For example:


import pytz
from django.utils import timezone


timezone.make_aware(datetime.datetime(2012, 3, 25, 3, 52),
 timezone=pytz.timezone('Europe/Stockholm'))
# returns datetime.datetime(2012, 3, 25, 3, 52, tzinfo=<DstTzInfo 'Europe/Stockholm' CEST+2:00:00 DST>)

Problem: You can use this for quite a long time until one day you end up with something like this:


timezone.make_aware(datetime.datetime(2012, 3, 25, 2, 52),
 timezone=pytz.timezone('Europe/Stockholm'))

# which leads to
Traceback (most recent call last):
 File "<console>", line 1, in <module>
 File "/home/ilian/venvs/test/lib/python3.4/site-packages/django/utils/timezone.py", line 358, in make_aware
 return timezone.localize(value, is_dst=None)
 File "/home/ilian/venvs/test/lib/python3.4/site-packages/pytz/tzinfo.py", line 327, in localize
 raise NonExistentTimeError(dt)
pytz.exceptions.NonExistentTimeError: 2012-03-25 02:52:00

Or this:


 
timezone.make_aware(datetime.datetime(2012, 10, 28, 2, 52),
 timezone=pytz.timezone('Europe/Stockholm'))

#throws
Traceback (most recent call last):
 File "<console>", line 1, in <module>
 File "/home/ilian/venvs/test/lib/python3.4/site-packages/django/utils/timezone.py", line 358, in make_aware
 return timezone.localize(value, is_dst=None)
 File "/home/ilian/venvs/test/lib/python3.4/site-packages/pytz/tzinfo.py", line 349, in localize
 raise AmbiguousTimeError(dt)
pytz.exceptions.AmbiguousTimeError: 2012-10-28 02:52:00

Explanation: The reason for the first error is that in the real world this datetime does not exists. Due to the DST change on this date the clock jumps from 01:59 standard time to 03:00 DST. Fortunately (or not) pytz is aware of the fact that this time is invalid and will throw the exception above. The second exception is almost the same but it happens when switching from summer to standard time. From 01:59 DST the clock shifts to 01:00 standard time, so we end with a duplicate time.

Why has this happened(in our case)? Well we couldn't be sure how exactly this one got into our legacy data but the assumption is that at the moment when the record was saved the server has been in different timezone where this has been a valid time.

Solution 1: This fix is quite simple, just add an hour if the exception occurs.


try:
 date = make_aware(
 datetime.fromtimestamp(date_time, timezone=pytz.timezone('Europe/Stockholm'))
 )
except (pytz.NonExistentTimeError, pytz.AmbiguousTimeError):
 date = make_aware(
 datetime.fromtimestamp(date_time) + timedelta(hours=1),
 timezone=pytz.timezone('Europe/Stockholm')
 )

Solution 2: Instead of calling make_aware call timezone.localize directly.


try:
 date = make_aware(
 datetime.fromtimestamp(date_time, timezone=pytz.timezone('Europe/Stockholm'))
 )
except (pytz.NonExistentTimeError, pytz.AmbiguousTimeError):
 timezone = pytz.timezone('Europe/Stockholm')
 date = timezone.localize(datetime.fromtimestamp(date_time), is_dst=False)

The second solution probably needs some explanation. First lets check what make_aware does. The code bellow is take from Django's sourcecode as it is in version 1.7.7


def make_aware(value, timezone):
 """
 Makes a naive datetime.datetime in a given time zone aware.
 """
 if hasattr(timezone, 'localize'):
 # This method is available for pytz time zones.
 return timezone.localize(value, is_dst=None)
 else:
 # Check that we won't overwrite the timezone of an aware datetime.
 if is_aware(value):
 raise ValueError(
 "make_aware expects a naive datetime, got %s" % value)
 # This may be wrong around DST changes!
 return value.replace(tzinfo=timezone)

To simplify it, what Django does is to use the localize method of the timezone object(if it exists) to convert the datetime. When using pytz this localize method takes two arguments: the datetime value and is_dst. The last argument takes three possible values: None, False and True. When using None and the datetime matches the moment of the DST change pytz does not know how to handle the datetime and you get one of the exceptions shown above. False means that it should convert it to standard time and True that it should convert it to summer time.

Why isn't this fixed in Django? The simple answer is "because this is how it should work". For a bit longer check the respectful ticket.

Reminder: Do not forget that the "fix" above does not actually care whether the original datetime is during DST or not. In our case this was not criticla for our app, but in some other cases it might be, so use it carefully.

Thanks: Special thanks to Joshua who correctly pointed out in the comments that I have missed the AmbiguousTimeError in the original post which made me to look a bit more in the problem, research other solutions and update the article to its current content.

Working with intervals in Python

Sun, 22 Feb 2015 18:55:50 +0000

Brief: Working with intervals in Python is really easy, fast and simple. If you want to learn more just keep reading.

Task description: Lets say that the case if the following, you have multiple users and each one of them has achieved different number of points on your website. So you want, to know how many users haven't got any point, how many made between 1 and 50 points, how many between 51 and 100 etc. In addition at 1000 the intervals start increasing by 100 instead of 50.

Preparing the intervals: Working with lists in Python is so awesome, so creating the intervals is quite a simple task.


intervals = [0] + \ # The zero intervals
 [x * 50 for x in range(1, 20)] + \ # The 50 intervals
 [x * 100 for x in range(10, 100)] + \ # The 100 intervals
 [x * 1000 for x in range(10, 102)] # the 1000 intervals

So after running the code above we will have a list with the maximum number of points for each interval. Now it is time to prepare the different buckets that will store the users count. To ease this we are going to use a defaultdict.


from collections import defaultdict

buckets = defaultdict(lambda: 0)

This way, we can increase the count for each bucket without checking if it exists. Now lets got to counting


for user in users:
 try:
 bucket = intervals[bisect.bisect_left(intervals, user.points)]
 except IndexError:
 # we are over the last bucket, so we put in in it
 bucket = intervals[-1]

buckets[bucket] += 1

How it works: Well it is quite simple. The bisect.bisect_left uses a binary search to estimate the position where an item should be inserted to keep the list, in our case intervals sorted. Using the position we take the value from the invervals that represent the bucker where the specified number should go. And we are ready. The result will looks like:


 {
 1: 10,
 10: 5,
 30: 8,
 1100: 2
 }

Final words: As you see when the default dict is used it does not have values for the empty buckets. This can be good or bad depending from the requirements how to present the data but it can be esily fixed by using the items from the intervals as keys for the buckets.

P.S. Comments and ideas for improvement are always welcome.

Resurection

Sun, 22 Feb 2015 18:38:26 +0000

Well, as some of you may have seen this blog was on hold for quite a long time. There were multiple reasons mainly my Ph.D. and changing my job but it is back online.

So, what is new?

As a start this blog is no longer running on wordpress. The reason is that I had some issues with wordpress - the blog was hacked twice due to security holes in wordpress/plugins, it was terribly slow and the code looked like shit. Lots of inline styles and javascript etc. So I made a simple Django based blog that generates static content. Alse we have new design and new domain, the last one much easier to remember )))

Also the comments are now handled by Disquss and the search functinality is provided by Google. The code of the blog, needs some minor cleaning and then it will be released publicly in the next few weeks. Meanwhile you can check my latest post Working with intervals in Python.

P.S. I have finally finished my Ph.D. so no more university/reasearch job and hopefully more time for blogging.

Python Interview Question and Answers

Sun, 28 Apr 2013 22:49:32 +0000

Update 2: You can also check the follow up post Django interview questions.

For the last few weeks I have been interviewing several people for Python/Django developers so I thought that it might be helpful to show the questions I am asking together with the answers. The reason is ... OK, let me tell you a story first.
I remember when one of my university professors introduced to us his professor - the one who thought him. It was a really short visit but I still remember one if the things he said. "Ignorance is not bad, the bad thing is when you do no want to learn."
So back to the reason - if you have at least taken care to prepare for the interview, look for a standard questions and their answers and learn them this is a good start. Answering these question may not get you the job you are applying for but learning them will give you some valuable knowledge about Python.
This post will include the questions that are Python specific and I'll post the Django question separately.

How are arguments passed - by reference of by value?
The short answer is "neither", actually it is called "call by object” or “call by sharing"(you can check here for more info). The longer one starts with the fact that this terminology is probably not the best one to describe how Python works. In Python everything is an object and all variables hold references to objects. The values of these references are to the functions. As result you can not change the value of the reference but you can modify the object if it is mutable. Remember numbers, strings and tuples are immutable, list and dicts are mutable.
Do you know what list and dict comprehensions are? Can you give an example?
List/Dict comprehensions are syntax constructions to ease the creation of a list/dict based on existing iterable. According to the 3rd edition of "Learning Python" list comprehensions are generally faster than normal loops but this is something that may change between releases. Examples:
```
# simple iteration
a = []
for x in range(10):
 a.append(x*2)
# a == [0, 2, 4, 6, 8, 10, 12, 14, 16, 18]

# list comprehension
a = [x*2 for x in range(10)]

# dict comprehension
a = {x: x*2 for x in range(10)}
# a == {0: 0, 1: 2, 2: 4, 3: 6, 4: 8, 5: 10, 6: 12, 7: 14, 8: 16, 9: 18}
```
What is PEP 8?
PEP 8 is a coding convention(a set of recommendations) how to write your Python code in order to make it more readable and useful for those after you. For more information check PEP 8.
Do you use virtual environments?
I personally and most(by my observation) of the Python developers find the virtual environment tool extremely useful. Yeah, probably you can live without it but this will make the work and support of multiple projects that requires different package versions a living hell.
Can you sum all of the elements in the list, how about to multuply them and get the result?
```
# the basic way
s = 0
for x in range(10):
 s += x

# the right way
s = sum(range(10))


# the basic way
s = 1
for x in range(1, 10):
 s = s * x

# the other way
from operator import mul
reduce(mul, range(1, 10))
 
```
As for the last example, I know Guido van Rossum is not a fan of reduce, more info here, but still for some simple tasks reduce can come quite handy.
Do you know what is the difference between lists and tuples? Can you give me an example for their usage?
First list are mutable while tuples are not, and second tuples can be hashed e.g. to be used as keys for dictionaries. As an example of their usage, tuples are used when the order of the elements in the sequence matters e.g. a geographic coordinates, "list" of points in a path or route, or set of actions that should be executed in specific order. Don't forget that you can use them a dictionary keys. For everything else use lists.
Do you know the difference between range and xrange?
Range returns a list while xrange returns a ~~generator~~ xrange object which takes the same memory no matter of the range size. In the first case you have all items already generated(this can take a lot of time and memory) while in the second you get the elements one by one e.g. only one element is generated and available per iteration. Simple example of generator usage can be find in the problem 2 of the "homework" for my presentation Functions in Python
Tell me a few differences between Python 2.x and 3.x
There are many answers here but for me some of the major changes in Python 3.x are: all strings are now Unicode, print is now function not a statement. There is no range, it has been replaced by xrange which is removed. All classes are new style and the division of integers now returns float.
What are decorators and what is their usage?
According to Bruce Eckel's Introduction to Python Decorators "Decorators allow you to inject or modify code in functions or classes". In other words decorators allow you to wrap a function or class method call and execute some code before or after the execution of the original code. And also you can nest them e.g. to use more than one decorator for a specific function. Usage examples include - logging the calls to specific method, checking for permission(s), checking and/or modifying the arguments passed to the method etc.
The with statement and its usage.
In a few words the with statement allows you to executed code before and/or after a specific set of operations. For example if you open a file for reading and parsing no matter what happens during the parsing you want to be sure that at the end the file is closed. This is normally achieved using the try... finally construction but the with statement simplifies it usin the so called "context management protocol". To use it with your own objects you just have to define __enter__ and __exit__ methods. Some standard objects like the file object automatically support this protocol. For more information you may check Understanding Python's "with" statement.

Well I hope this will be helpful, if you have any question or suggestion feel free to comment.

Update: Due to the lots of comments on Reddit and LinkedIn, I understood that there is some misunderstanding about the post. First, the questions I have published are not the only ones I ask, the interview also includes such related to general programming knowledge and logical thinking. Second the questions above help me get a basic understanding of your Python knowledge but they are not the only think that makes my decision. Not answering some of them does not mean that you won't get the job, but it may show me on which parts we should to work.

Django for Web Prototyping

Mon, 15 Apr 2013 20:46:36 +0000

Or how to use the benefits of Django template system during the PSD to HTML phase

There are two main approaches to start designing a new project - Photoshop mock-up or an HTML prototype. The first one is more traditional and well established in the web industry. The second one is more alternative and (maybe)modern. I remember a video of Jason Fried from 37 Signals where he talks about design and creativity. You can see it at http://davegray.nextslide.com/jason-fried-on-design. There he explains how he stays away from the Photoshop in the initial phase to concetrate on the things that you can interact with instead of focusing on design details.

I am not planning to argue which is the better method, the important thing here is that sooner or later you get to the point where you have to start the HTML coding. Unfortunately frequently this happens in a pure HTML/CSS environment outside of the Django project and then we waste some extra amount of time to convert it to Django templates.

Wouldn't be awesome if you can give the front-end developers something that they can install/run with a simple command and still to allow them to work in the Django environment using all the benefits it provides - templates nesting and including, sekizai tags etc.

I have been planning to do this for a long time and finally it is ready and is available at Django for Prototyping. Currently the default template includes Modernizr, jQuery and jQuery UI but you can easily modify it according to your needs. I would be glad of any feedback and ideas of improvement so feel free to try it and comment.

Introduction to Django - presentation

Mon, 18 Mar 2013 08:31:11 +0000

This presentation shows the basics of Django - what is inside the framework and explains the Model-View-Template system. One of the most important parts is a diagram how the request is processed and the response is generated. Shows the project and the application structure and the basic elements - Models, URLs dispatcher, Views and Templates.

View the slides on SlideShare

Functions in Python Homework

Sat, 16 Mar 2013 09:42:03 +0000

After some delay(please excuse me for it) here are sample solutions of the problems defined in the presentation Functions in Python. Solutions are also available at GitHub.

Functions in Python presentation

Mon, 18 Feb 2013 07:22:19 +0000

Here is my presentation part of the in company Python course.

View the slides on SlideShare

The last slide - "Problem to solve" is something like a simple homework. Sample solutions will be uploaded later this week.

Simple Site Checker and the User-Agent header

Mon, 22 Oct 2012 19:22:15 +0000

Preface: Nine months ago(I can't believe it was that long) I created a script called Simple Site Checker to ease the check of sitemaps for broken links. The script code if publicly available at Github. Yesterday(now when I finally found time to finish this post it must be "A few weeks ago") I decided to run it again on this website and nothing happened - no errors, no warning, nothing. Setting the output level to DEBUG showed the following message "Loading sitemap ..." and exited.
Here the fault was mine, I have missed a corner case in the error catching mechanism i.e. when the sitemap URL returns something different from "200 OK" or "500 internal server error". Just a few second and the mistake was fix.

Problem and Solution: I ran the script again and what a surprise the sitemap URL was returning "403 Forbidden". At the same time the sitemap was perfectly accessible via my browser. After some thinking I remembered about that some security plugins block the access to the website if there is not User-Agent header supplied. The reason for this is to block the access of simple script. In my case even an empty User-Agent did the trick to delude the plugin.


urllib2.urlopen(urllib2.Request(url,
 headers={'User-Agent': USER_AGENT}))

Final words: As a result of the issue mention above one bug in simple site checker was found fixed. At the same time another issue about missing status and progress was raised, more details can be found at Github but in a few words an info message was added to each processed URL to indicate the progress.

If you have any ideas for improvement or anything else feel free to comment, create issues and/or fork the script.

Automation, Fabric and Django - presentation

Wed, 03 Oct 2012 20:12:07 +0000

As a follow up post of Automated deployment with Ubuntu, Fabric and Django here are the slides from my presentation on topic "Automation, Fabric and Django". Unfortunately there is no audio record but if there is interest I can add some comments about each slide as part of this post.

If there is anything that need explanation feel free to ask.

Django project file structure

Mon, 24 Sep 2012 20:33:00 +0000

As I promised in Automated deployment with Ubuntu, Fabric and Django I will use this post to explain the file structure that I use for my Django projects and what I benefit from it. So here is my project directory tree.

The structure

~/workspace/&lt;project_name&gt;/
|-- bin
|-- include
|-- lib
|-- local
|-- src
 |-- .git
 |-- .gitignore
 |-- required_packages.txt
 |-- media
 |-- static
 |-- &lt;project_name&gt;
 | |-- &lt;project_name&gt;
 | | |-- __init__.py
 | | |-- settings
 | | | |-- __init__.py
 | | | |-- &lt;environment_type&gt;.py
 | | | |-- local.py
 | | |-- templates
 | | |-- urls.py
 | | |-- views.py
 | |-- manage.py
 | |-- wsgi.py
 |-- &lt;project_name&gt;.development.nginx.local.conf
 |-- &lt;project_name&gt;.&lt; environment_type&gt;.nginx.uwsgi.conf
 |-- &lt;project_name&gt;.&lt; environment_type&gt;.uwsgi.conf

Explanation

At the top I have a directory named as the project and virtual environment inside of it. The benefit from it is complete isolation of the project from the surrounding projects and python packages installed at OS level and ability to install packages without administrator permissions. It also provides an easy way to transfer the project from one system to another using a requirements file.
The src folder is where I keep everything that is going to enter the version control project source, requirements files, web server configuration etc.
My default .gitignore is made to skip the pyc-files, the PyDev files and everything in the static and media directories.
The media directory is where the MEDIA_ROOT settings point to, respectively static is for the STATIC_ROOT.
All required packages with their version are placed in required_packages.txt so we can install/update them with a single command in the virtual environment.
In a directory with the name of the project is where the python code resides. Inside it the project structure partly follows the new project layout introduced in Django 1.4.
The big difference here is the settings part. It is moved as a separate module where all common/general settings are place in __init__.py. We have one file for each possible environment(development, staging, production) with the environment specific settings like DEBUG and TEMPLATE_DEBUG. In local.py are placed the settings specific for the machine where the project is running. This file should not go into the repository as this is where the access database password and API keys should reside.
I having one local Nginx configuration file because I use the webserver to serve the static files when working locally this is the <project_name>.development.nginx.local.conf.
For each environment there is also a couple of configuration files - one for the web server(<project_name>.< environment_type>.nginx.uwsgi.conf) and one for the uwsgi(<project_name>.< environment_type>.uwsgi.conf). I make symbolic links pointing to these files so any changes made are automatically pushed/pulled via version control, I only have to reload the configuration. There is no option to change something in the configuration at one station and to forget to transfer it to the rest.

Benefits

Complete isolation from other projects using virtual environment
Easy to transfer and update packages on other machines thanks to the pip requirements file
Media/static files outside the source directory for higher security
Web server/uWSGI configuration as part of the repository for easier and error proof synchronization

Probably it have some downside(I can not think any of these now but you never know) so if you think that this can be improved feel free to share your thoughts. Also if there is anything not clear enough, just ask me, I will be happy to clear it.

Automated deployment with Ubuntu, Fabric and Django

Tue, 18 Sep 2012 19:56:18 +0000

A few months ago I started to play with Fabric, and the result was a simple script that automates the creation of a new Django project. In the last months I continued my experiments and extended the script to a full stack for creation and deployment of Django projects.

As the details behind the script like the project structure that I use and the server setup are a bit long, I will keep this post only on the script usage. I will write a follow-up one describing the project structure and server.

So in brief, the setup that I use consists of Ubuntu, Nginx as web server and uWSGI as application server. Upstart controls the last one. The script is available for download at GitHub.
In a wait for more detailed documentation, here is a short description of the main tasks and what they do:

startproject:<project_name>

Creates a new virtual environment
Installs the predefined packages (uWSGI, Django and South))
Creates a new Django project from the predefined template
Creates different configuration files for development and production environment
Initializes a new git repository
Prompts the user to choose a database type. Then installs database required packages, creates a new database and user with the name of the project and generates the local settings file
Runs syncdb(if database is selected) and collectstatic

setup_server

installs the required packages such as Nginx, PIP, GCC, etc.
prompts for a database type and install it
reboots the server as some packages may require it

Once you have a ready server and a project, just use this the following task to deploy it to the server. Please have in mind that it should be used only for initial deployment.

deploy_project:<project_name>,<env_type>,<project_repository>

Creates a new virtual environment
Clones the project from the repository
Installs the required packages
Creates symbolic links to the nginx/uwsgi configuration files
Asks for a database engine, creates a new DB / User with the project name and updates the settings file
Calls the update_project task

update_project:<project_name>,<env_type>

Updates the source code
Installs the required packages(if there are new)
Runs syncdb, migrate and collect static
Restart the Upstart job that runs the uWSGI and the Nginx server

The script is still in development, so use it at your own risk. Also, it reflects my own idea of server/application (I am planning to describe it deeper in a follow-up post) setup, but I would really like if you try it and give feedback. Feel free to fork, comment and advice.

The road to hell is paved with regular expressions ...

Tue, 31 Jul 2012 20:25:37 +0000

... or what is the cost of using regular expressions for simple tasks

Regular expressions are one of the most powerful tools in computing I have ever seen. My previous post about Django compressor and image preloading is a good example how useful they might be. The only limit of their use is your imagination. But "with great power, comes great responsibility" or in this case a great cost. Even the simplest expressions can be quite heavy compared with other methods.

The reason to write about this is a question recently asked in a python group. It was about how to get the elements of a list that match specific string. My proposal was to use comprehension list and simple string comparison while other member proposed using a regular expression. I was pretty sure that the regular expression is slower but not sure exactly how much slower so I made a simple test to find out.


import re
import timeit

my_list = ['abc-123', 'def-456', 'ghi-789', 'abc456', 'abc', 'abd']

def re_check():
 return [i for i in my_list if re.match('^abc$', i)]

t = timeit.Timer(re_check)
print 're_check result >>', re_check()
print "%.2f usec/pass" % (1000000 * t.timeit(number=100000)/100000)

def simple_check():
 return [i for i in my_list if i=='abc']

t = timeit.Timer(simple_check)
print 'simple_check result >>', simple_check()
print "%.2f usec/pass" % (1000000 * t.timeit(number=100000)/100000)

The results was 23.99 vs 1.41 usec/pass respectively for regular expression vs direct comparison i.e. the regexp was 17 times slower. The difference from the example above may be OK in some cases but it rises with the size of the list. This is a simple example how something really quick on local version may take significant time on production and even to broke your application.

So, should you learn and use regular expressions?
Yes! Absolutely!

They are powerful and useful. They will open your mind and allow you to do things you haven't done before. But remember that they are a double-edge razor and should be used cautiously. If you can avoid it with other comparison(one or more) just run a quick test to see whether it will be faster. Of course if you can not avoid it you can also think about caching the results.

Django compressor and image preloading

Mon, 30 Jul 2012 13:26:36 +0000

Preface: Have you noticed how on some websites when you click on a link that opens a lightbox or any overlay for first time it takes some time to display the border/background/button images. Not quite fancy, right?
This is because the load of this images starts at the moment the overlay is rendered on the screen. If this is your first load and these images are not in your browser cache it will take some time for the browser to retrieve them from the server.

Solution: The solution for this is to preload the images i.e. to force the browser to request them from the server before they are actually used. With a simple javascript function and a list of the images URLs this is a piece of cake:


$.preLoadImages = function() {
 var args_len = arguments.length;
 for (var i=0; i < args_len; i++) {
 var cacheImage = document.createElement('img');
 cacheImage.src = arguments[i];
 }
}

$.preLoadImages('/img/img1.png', '/img/img2.png')

Please have in mind that the code above uses the jQuery library. Specialty: Pretty easy, but you have to hardcode the URLs of all images. Also if you are using Django compressor then probably you are aware that it adds extra hash to the URLs of the images in the compressed CSS files. The hash depends from the COMPRESS_CSS_HASHING_METHOD settings and can not be avoided. It is pretty useful cause it forces the client browser to reload the images every time when something has been changed. unfortunately our hardcoded list of URLs does not have this hash. So wouldn't it be much simpler if instead of hardcoding URLs we just read them from the CSS files? Solution 2:


$.preLoadImages = function() {
 $.get($('link[rel="stylesheet"]')[0].href, function(data){
 r = /url\(['|"]?(\S+\.(gif|jpg|jpeg|png)[^'(]*)['|"]?\)/ig;
 while (match = r.exec(data)){
 var cacheImage = document.createElement('img');
 cacheImage.src = match[1];
 }
 });
}

$.preLoadImages()

Now with the help of regular expressions we can read the image URLs directly from the CSS file together with the hash part. Please note the zero index in the css file selector, if your main CSS is not the first declared style-sheet then you will have to change the index according to its position.

I hope you will find this solutions simple and useful. As always feel free to comment, share and propose code improvements.

Python is not a Panacea ...

Mon, 11 Jun 2012 18:29:18 +0000

... neither is any other language or framework

This post was inspired by the serial discussion on the topic "Python vs other language"(in the specific case the other one was PHP, and the question was asked in a Python group so you may guess whether there are any answers in favor of PHP). It is very simple, I believe that every Python developer will tell you that Python is the greatest language ever build, how easy is to learn it, how readable and flexible it is, how much fun it is to work with it and so on. They will tell you that you can do everything with it: web and desktop development, testing, automation, scientific simulations etc. But what most of them will forgot to tell you is that it is not a Panacea.

In the matter of fact you can also build "ugly" and unstable applications in Python too. Most problems come not from the language or framework used, but from bad coding practices and bad understanding of the environment. Python will force you to write readable code but it wont solve all your problems. It is hard to make a complete list of what exactly you must know before starting to build application, big part of the knowledge comes with the experience but here is a small list of some essential things.

Write clean with meaningful variable/class names.
Exceptions are raised, learn how to handle them.
Learn OOP(Object Oriented Programming)
Use functions to granulate functionality and make code reusable
DRY(Don't Repeat Youtself)
If you are going do develop web application learn about the Client-Server relation
Use "layers" to seprate the different parts of your application - database methods, business logic, output etc. MVC is a nice example of such separation
Never store passwords in plain text. Even hashed password are not completely safe, check what Rainbow Tables are.
Comment/Document your code
Write unit test and learn TDD.
Learn how to use version control.
There is a client waiting on the other side - don't make him wait too long.
Learn functional programming.

I hope the above does not sounds as an anti Python talk. This is not its idea. Firstly because there are things that are more important than the language itself(the list above) and secondly because... Python is awesome )))
There are languages that will help you learn the things above faster, Python is one of them - built in documentation features, easy to learn and try and extremely useful. My advice is not to start with PHP as your first programming language it will make you think that mixing variables with different types is OK. It may be fast for some things but most of the times it is not safe so you should better start with more type strict language where you can learn casting, escaping user output etc.

Probably I have missed a few(or more) pointa but I hope I've covered the basics. If you think that anything important is missing, just add it in the comments and I will update the post.

Fabric & Django

Mon, 28 May 2012 12:43:01 +0000

Or how automate the creation of new projects with simple script

Preface: Do you remember all this tiny little steps that you have to perform every time when you start new project - create virtual environment, install packages, start and setup Django project? Kind of annoying repetition, isn't it? How about to automate it a bit.

Solution: Recently I started learning Fabric and thought "What better way to test it in practice than automating a simple, repetitive task?". So, lets mark the tasks that I want the script to perform:

Create virtual environment with the project name
Activate the virtual environment
Download list of packages and install them
Make 'src' directory where the project source will reside
Create new Django project in source directory
Update the settings

Thanks to the local command the first one was easy. The problem was with the second one. Obviously each local command is run autonomously so I had to find some way have activated virtual environment for each task after this. Fortunately the prefix context manager works like a charm. I had some issues making it read and write in the paths I wants and voilà it was working exactly as I want.

The script is too long to place it here but is publicly available at https://gist.github.com/2818562
It is quite simple to use, you only need python, fabric and virtual environment. Then just use the following code.


fab start_project:my_mew_project

To Do: Here are few things that can be improved:

Read the packages from a file
Update urls.py to enable admin
Generate Nginx server block file

So this is my first try with Fabric, I hope that you will like it and find it useful. As always any comments, questions and/or improvement ideas are welcome.

HTTP Status Codes Site

Wed, 01 Feb 2012 22:36:23 +0000

During the development of Simple Site Checker I realised that it would be useful for test purposes if there is a website returning all possible HTTP status codes. Thanks to Google App Engine and webapp2 framework building such website was a piece of cake.

The site can be found at http://httpstatuscodes.appspot.com.

The home page provides a list of all HTTP status codes and their names and if you want to get an HTTP response with a specific status code just add the code after the slash, example:
http://httpstatuscodes.appspot.com/200 - returns 200 OK
http://httpstatuscodes.appspot.com/500 - returns 500 Internal Server Error
Also at the end of each page is located the URL of the HTTP protocol Status Codes Definitions with detailed explanation for each one of them.

The website code is publicly available in github at HTTP Status Codes Site.

If you find it useful feel free to comment and/or share it.

Simple Site Checker

Mon, 30 Jan 2012 22:03:13 +0000

... a command line tool to monitor your sitemap links

I was thinking to make such tool for a while and fortunately I found some time so here it is.

Simple Site Checker is a command line tool that allows you to run a check over the links in you XML sitemap.

How it works: The script requires a single attribute - a URL or relative/absolute path to xml-sitemap. It loads the XML, reads all loc-tags in it and start checking the links in them one by one.
By default you will see no output unless there is an error - the script is unable to load the sitemap or any link check fails.
Using the verbosity argument you can control the output, if you need more detailed information like elapsed time, checked links etc.
You can run this script through a cron-like tool and get an e-mail in case of error.

I will appreciate any user input and ideas so feel free to comment.

Faking attributes in Python classes...

Mon, 30 Jan 2012 08:00:18 +0000

... or how to imitate dynamic properties in a class object

Preface: When you have connections between your application and other systems frequently the data is not in the most useful form for your needs. If you have an API it is awesome but sometimes it just does not act the way you want and your code quickly becomes a series of repeating API calls like api.get_product_property(product_id, property).
Of course it will be easier if you can use objects to represent the data in you code so you can create something like a proxy class to this API:


class Product(object):
 def __init__(self, product_id):
 self.id = product_id

 @property
 def name(self):
 return api_obj.get_product_property(self.id, 'name')

 @property
 def price(self):
 return api_obj.get_product_property(self.id, 'price')

#usage
product = Product(product_id)
print product.name

In my opinion it is cleaner, more pretty and more useful than the direct API calls. But still there is something not quite right. Problem: Your model have not two but twenty properties. Defining 20 method makes the code look not that good. Not to mention that amending the code every time when you need a new property is quite boring. So is there a better way? As I mention at the end of Connecting Django Models with outer applications if you have a class that plays the role of a proxy to another API or other data it may be easier to overwrite the __getattr__ method. Solution:


class Product(object):
 def __init__(self, product_id):
 self.id = product_id

 def __getattr__(self, key):
 return api_obj.get_product_property(self.id, key)

#usage
product = Product(product_id)
print product.name

Now you can directly use the product properties as attribute names of the Product class. Depending from the way that the API works it would be good to raise AttributeError if there is no such property for the product.

Software for business

Tue, 24 Jan 2012 02:53:10 +0000

I am starting a new blog. The reason is that I want to keep this one more technically oriented while the other will be more business and customers oriented. Its name is Software for business and the idea is to show to the business in less technical details how the modern IT technologies like CRM & ERP systems, web sites, e-commerce solutions, online marketing and so on can help their business.
If you find the topic interesting feel free to join.

Connecting Django Models with outer applications

Mon, 23 Jan 2012 22:05:34 +0000

Preface: Sometimes, parts of the data that you have to display in your application reside out of the Django models. Simple example for this is the following case - the client requires that you build them a webshop but they already have CRM solution that holds their products info. Of course they provide you with a mechanism to read this data from their CRM.

Specialty: The problem is that the data in their CRM does not hold some of the product information that you need. For instance it misses SEO-friendly description and product image. So you will have to set up a model at your side and store these images there. It is easy to join them, the only thing that you will need is a simple unique key for every product.

Solution: Here we use the product_id field to make the connection between the CRM data and the Django model.


# in models.py
class Product(models.Model):
 product_id = models.IntegerField(_('Original Product'),
 unique=True)
 description = models.TextField(_('SEO-friendly Description'),
 blank=True)
 pod_image = FilerImageField(verbose_name=_('Product Image'),
 blank=True, null=True)

 @property
 def name(self):
 return crm_api.get_product_name(self.product_id)

# in forms.py
class ProductForm(forms.ModelForm):
 name = forms.CharField(required=False,
 widget=forms.TextInput(attrs={
 'readonly': True,
 'style': 'border: none'}))

 class Meta:
 model = Product
 widgets = {
 'product_id': forms.Select(),
 }
 def __init__(self, *args, **kwargs):
 super(ProductForm, self).__init__(*args, **kwargs)
 self.fields['product_id'].widget.choices = crm_api.get_product_choices()
 if self.instance.id:
 self.fields['name'].initial = self.instance.name

The form here should be used in the admin(add/edit) page of the model. We define that the product_id field will use the select widget and we use a method that connect to the CRM and returns the product choices list.
The "self.instance.id" check is used to fill the name field for product that are already saved.

Final words: This is a very simple example but its idea is to show the basic way to connect your models with another app. I strongly recommend you to use caching if your CRM data is not modified very often in order to save some bandwidth and to speed up your application.
Also if you have multiple field it may be better to overwrite the __getattr__ method instead of defining separate one for each property that you need to pull from the outer application.

P.S. Thanks to Miga for the code error report.

Python os.stat times are OS specific ...

Mon, 21 Nov 2011 07:00:02 +0000

... no problem cause we have stat_float_times

Preface: recently I downloaded PyTDDMon on my office station and what a surprise - the files changes did not affect on the monitor. So I run a quick debug to check what is going on: the monitor was working running a check on every 5 seconds as documented but it does not refresh on file changes.

Problem: it appeared that the hashing algorithm always returns the same value no matter whether the files are changed or not. The calculation is based on three factors: file path, file size and time of last modification. I was changing a single file so the problem was in the last one. Another quick check and another surprise - os.stat(filename).st_mtime returns different value after the files is changed but the total hash stays unchanged. The problem was caused from st_mtime returning floating point value instead of integer.

Solution: Unfortunately the os.stat results appeared to be OS specific. Fortunately the type of the returned result is configurable. Calling os.stat_float_times(False) in the main module force it to return integer values.

Final words: these OS/version/settings things can always surprise you. And most of the time it is not a pleasant surprise. Unfortunately testing the code on hundred of different workstations is not an option. But this is one of the good sides of the open source software. You download it, test it, debug it and have fun.
Also special thanks to Olof(the man behind the pytddmon concept) for the tool, the quick response to my messages and for putting my fix so fast in the repo. I hope that this has saved someone nerves )

P.S. If you are using Python and like TDD - PYTDDMON is the right tool for you.

Django CMS Plugins with selectable template ...

Sun, 09 Oct 2011 21:25:07 +0000

... or how to reuse your plugins inside sections with different design

Problem: Frequently on the websites I am developing I need to display same set of data in several different ways. For example if I have a news box that needs to appear in different sections of the website e.i. in sidebar, main content etc. Using Django CMS plugins make this quite easy.
For simplicity we will take the following case. An image/text tuple with two layout variations - image on left of text and image on right.

Same data but different layout. All you need to do is just to allow your users to change the plugin template according to their needs. If you don't have experience with Django CMS Plugins I advice you to check how to create custom Django CMS Plugins before you continue with solution.

Solution: First you will have to create a tuple holding your templates(and their human readable names) and add a field that will hold the chosen template to the plugin model.


#models.py
PLUGIN_TEMPLATES = (
 ('image_on_left.html', 'Image on left'),
 ('image_on_right.html', 'Image on right'),
)

class SamplePlugin(CMSPlugin):
 # your plugin properties here
 template = models.CharField('Template', max_length=255,
 choices = PLUGIN_TEMPLATES)

Now it is time to tweak the template render method too:


#cms_plugins.py
class CMSSamplePlugin(CMSPluginBase):
 model = SamplePlugin
 name = 'Sample plugin'
 render_template = PLUGIN_TEMPLATES[0][0]
 
 def render(self, context, instance, placeholder):
 if instance and instance.template:
 self.render_template = instance.template
 #your stuff here
 return context

Final words: Yep, this is all. Simple isn't it? It is amazing how sometimes such small things are so useful. If you are having bigger difference in the layout of your templates you will probably have to put a little more stuff in the context that some of your templates may not need but it is OK. Feel free to comment and if you are using this "trick" please add your use case - it will be interesting to see in how many different cases this works.

Neil Gaiman's Anansi Boys...

Sun, 02 Oct 2011 22:00:41 +0000

... an unexpected gift and unexpected dedication

YOU KNOW HOW IT IS. You pick up a book, flip to the dedication, and find that, once again, the author has dedicated a book to someone else and not you.

Not this time.

Because we haven't yet met/have only a glancing acquaintance/are just crazy about each other/haven't seen each other in much too long/are in some way related/will never meet, but will, I trust, despite that, always think fondly of each other...

This one's for you.

With you know what, and you probably know why."

This is how Neil Gaiman's Anansi Boys starts.

About a week ago I spent few days in Greece. One day we were sitting under the shadow of a near bar when something grabbed my attention. Next to the bar there was a book shelf with something special on it:

It was really strange to go in a foreign country, in unknown place and there a few meters away from me standing out with its black cover to see a book of one of my favourite authors. So I just took my camera, go there and take a shot on it.

The bartender asked me what am I shooting so I explained to him that it was a surprise for me to see that book. He took it as he was seeing it for a first time in his life, took a brief look at it and asked me whether I have read it. I said that unfortunately I am not. Then he took another look at the book and gave it to me as a gift. I took and open it and... it was starting with the dedication from the top of this post.

Coincidence? Maybe... or maybe not.

It is always a pleasure for me to discover pieces of art, no matter whether I am travelling or not, no matter is it an ancient castle or a book, it is great.
I haven't still finished the book but it looks amazing. I really recommend it as any other Gaiman's book. Hopefully I'll meet Neil one day till than(and after) I'll keep reading.
So if you are reading this go travel, or read books, or both. Have fun and share your experience.

Language redirects for multilingual sites with Django CMS ...

Sun, 11 Sep 2011 14:15:13 +0000

... or how to avoid duplicate content by keeping the current language in the URL

Preface: Earlier this year I posted about Django CMS 2.2 features that I want to see and one of the things mentioned there was that once you have chosen the language of the site there is no matter whether you will open "/my_page/" or "/en/my_page/" - it just shows the same content. The problem is that this can be considered both duplicate and inconsistent content.
Duplicate because you see the same content with and without the language code in the URL and inconsistent because for the same URL you can get different language versions i.e. different content.

Solution: This can be easy fixed by using a custom middleware that will redirect the URL that does not contain language code. In my case the middleware is stored in "middleware/URLMiddlewares.py"(the path is relative to my project root directory) and contains the following code.


from cms.middleware.multilingual import MultilingualURLMiddleware 
from django.conf import settings
from django.http import HttpResponseRedirect
from django.utils import translation

class CustomMultilingualURLMiddleware(MultilingualURLMiddleware): 
 def process_request(self, request):
 lang_path = request.path.split('/')[1]
 if lang_path in settings.URLS_WITHOUT_LANGUAGE_REDIRECT:
 return None
 language = self.get_language_from_request(request) 
 translation.activate(language) 
 request.LANGUAGE_CODE = language
 if lang_path == '': 
 return HttpResponseRedirect('/%s/' % language)
 if len([z for z in settings.LANGUAGES if z[0] == lang_path]) == 0:
 return HttpResponseRedirect('/%s%s' % (language, request.path))

Now a little explanation on what happens in this middleware. Note: If you are not familiar with how middlewares work go and check Django Middlewares. Back to the code. First we split the URL by '/' and take the second element(this is where our language code should be) and store in lang_path(8). URLS_WITHOUT_LANGUAGE_REDIRECT is just a list of URLs that should not be redirected, if lang_path matches any of the URLs we return None i.e. the request is not changed(9-10). This is used for sections of the site that are not language specific for example media stuff. Then we get language based on the request(11-13). If lang_path is empty then the user has requested the home page and we redirect him to the correct language version of it(14-15). If lang_path does not match any of the declared languages this mean that the language code is missing from the URL and the user is redirected to the correct language version of this page(16-17). To make the middleware above to work you have to update your settings.py. First add the middleware to your MIDDLEWARE_CLASSES - in my case the path is 'middleware.URLMiddlewares.CustomMultilingualURLMiddleware'. Second add URLS_WITHOUT_LANGUAGE_REDIRECT list and place there the URLs that should not be redirected, example:


URLS_WITHOUT_LANGUAGE_REDIRECT = [
 'css',
 'js',
]

Specialties: If the language code is not in the URL and there is no language cookie set your browser settings will be used to determine your preferred language. Unfortunately most of the users do not know about this option and it often stays set to its default value. If you want this setting to be ignored just add the following code after line 10 in the middleware above:


if request.META.has_key('HTTP_ACCEPT_LANGUAGE'):
 del request.META['HTTP_ACCEPT_LANGUAGE']

It removed the HTTP_ACCEPT_LANGUAGE header sent from the browser and Django uses the language set in its settings ad default.

URLS_WITHOUT_LANGUAGE_REDIRECT is extremely useful if you are developing using the built in dev server and serve the media files trough it. But once you put your website on production I strongly encourage you to serve these files directly by the web server instead of using Django static serve.

Final words: In Django 1.4 there will be big changes about multilingual URLs but till then you can use this code will improve your website SEO. Any ideas of improvement will be appreciated.

Retrieving Google Analytics data with Python...

Tue, 06 Sep 2011 15:27:41 +0000

... or how to pull data about page visits instead of implementing custom counter

Preface: OK, so you have a website, right? And you are using Google Analytics to track your page views, visitors and so on?(If not you should reconsider to start using it. It is awesome, free and have lost of features as custom segments, map overlay, AdSense integration and many more.)
So you know how many people have visited your each page of your website, the bounce rate, the average time they spend on the page etc. And this data is only for you or for a certain amount whom you have granted access.

Problem: But what happens if one day you decided to show a public statistic about visitors on your website. For example: How many people have opened the "Product X" page?
Of course you can add a custom counter that increases the views each time when the page is open. Developed, tested and deployed in no time. Everyone is happy until one day someones cat took a nap on his keyboard and "accidentally" kept the F5 button pressed for an hour. The result is simple - one of you pages has 100 times more visits than the other. OK, you can fix this with adding cookies, IP tracking etc. But all this is reinventing the wheel. You already have all this data in your Google Analytics, the only thing you have to do is to stretch hand and take it.

Solution: In our case "the hand" will be an HTTP request via the Google Data API. First you will need to install the Python version of the API:


sudo easy_install gdata

Once you have the API installed you have to build a client and authenticate:


SOURCE_APP_NAME = 'The-name-of-you-app'
my_client = gdata.analytics.client.AnalyticsClient(source=SOURCE_APP_NAME)
my_client.client_login(
 'USERNAME',
 'PASSWORD',
 source=SOURCE_APP_NAME,
 service=my_client.auth_service,
 account_type = 'GOOGLE',
)

token = my_client.auth_token

SOURCE_APP_NAME is the name of the application that makes the request. You can set it to anything you like. After you build the client(2) you must authenticate using your Google account(3-9). If you have both Google and Google APPs account with the same username be sure to provide the correct account type(8). Now you have authenticated and it is time to build the request. Obviously you want to filter the data according some rules. The easiest way is to use the Data Feed Query Explorer to build your filter and test it and then to port it to the code. Here is an example how to get the data about the page views for specific URL for a single month(remember to update the PROFILE_ID according to your profile).


account_query = gdata.analytics.client.AccountFeedQuery()
data_query = gdata.analytics.client.DataFeedQuery({
 'ids': 'ga:PROFILE_ID',
 'dimensions': '', #ga:source,ga:medium
 'metrics': 'ga:pageviews',
 'filters': 'ga:pagePath==/my_url_comes_here/',
 'start-date': '2011-08-06',
 'end-date': '2011-09-06',
 'prettyprint': 'true',
 })

feed = my_client.GetDataFeed(data_query)
result = [(x.name, x.value) for x in feed.entry[0].metric]

Final words: As you see it is relatively easy to get the data from Google but remember that this code makes two request to Google each time it is executed. So you will need to cache the result. The GA data is not real-time so you may automate the process to pull the data(if I remember correctly the data is updated once an hour) and store the results at your side which will really improve the speed. Also have in mind that this is just an example how to use the API instead of pulling the data page by page(as show above) you may pull the results for multiple URLs at once and compute the feed to get your data. It is all in your hands.
You have something to add? Cool I am always open to hear(read) you comments and ideas.

Update: If you are using Django you should consider to use it Memcached to cache these result as shown in Caching websites with Django and Memcached

Foreign key to Django CMS page ...

Thu, 14 Jul 2011 19:24:41 +0000

... how to make usable drop-downs with Django CMS pages

Problem: some times when you create custom applications or plugins for Django CMS you need a property that connects the current item to a page in the CMS. Nothing simple than this - you just add a ForeignKey in your model that points to the Page model and everything is (almost)fine. Example:


from cms.models import Page

class MyModel(models.Model):
 # some model attributes here
 page = models.ForeignKey(Page)

If you registered your model in Django admin or just add a model form to it you will see something like this:

Cool right? Not exactly. The problem is that these pages are in hierarchical structure and listing them in a flat list may be/is little confusing. So let's indent them accordingly to their level in the hierarchy. Solution: The easies way to achieve this indentation is to overwrite the choices list of the ForeignKey field in the ModelForm __init__ method.


class MyModelForm(forms.ModelForm):
 class Meta:
 model = MyModel
 
 def __init__(self, *args, **kwargs):
 super(MyModelForm, self).__init__(*args, **kwargs)
 choices = [self.fields['page'].choices.__iter__().next()]
 for page in self.fields['page'].queryset:
 choices.append(
 (page.id, ''.join(['-'*page.level, page.__unicode__()]))
 )
 self.fields['page'].choices = choices

The magic lies between lines 7 and 11, on line 7 we create a list with one element the default empty option for the drop down. The need to use "__iter__().next()" comes from the fact that the choices attribute of the fields is django.forms.models.ModelChoiceIterator object which is iterable, but not indexable i.e. you can not just use self.fields['url'].choices[0].
After we had the empty choice now it is time to add the real ones, so we iterate over the queryset(8th line) that holds them and for each item we add a tuple to our choices list(10). The first item of the tuple is the page id - nothing special, but the second one... here the python magic comes. We multiple the minus sign('-') by the page level and join the result with the page title. The only thing left is to replace the field choices(line 12) and here is the result:

Final words: For me this is much more usable than the flat list. Of course you can modify the queryset to return only published pages or filter the results in other way and still use the identation code from above.
I'll be happy to hear your thoughts on this.

Caching websites with Django and Memcached...

Tue, 12 Jul 2011 19:46:12 +0000

... memcached installation, django configuration and how to use it for your website

After Caching web sites and web applications and When to use caching for web sites its time for a little sample. This sample shows the usage of Memcached for server-side application cache. The installation part is taken from my Ubuntu so it may differ depending from your OS/distribution.

What is Memcached: Memcached is a tool that allows you to store key-value pairs in you memory. The keys are limited to 250 Bytes and for better performance the value size is limited to 1MB(more details) but this size is fair enough for web usage.

Memcached installation:


apt-get install memcached
apt-get install python-memcache

The first line installs Memcached and the second one install Python API for communication between your application and Memcached daemon. After this the Memcached daemon is up and running. With default configuration it runs on port 11211 on localhost(127.0.0.1). If you want to modify this the configuration file(in my case) is situated in /etc/memcached.conf Django configuration: This one depends from the Django version that you use. For 1.2.5 and prior the next code should by added in your settings file(settings.py):


CACHE_BACKEND = 'memcached://127.0.0.1:11211/'

For 1.3 and development version add:


CACHES = {
 'default': {
 'BACKEND': 'django.core.cache.backends.memcached.MemcachedCache',
 'LOCATION': '127.0.0.1:11211',
 }
}

In both cases if you use different port and/or IP you have to replace them above. More info about cache backend configuration you can find in Django documentation docs. So now you have Memcached running and Django configured. If you have doubts about is this suitable/usable in you case take a look at the posts mentioned above or just add comment with your case and I will be happy to give you an advice. Now it is time to start using it. Cache usage(part I) - how to cache on Python level: If you have some heavy calculations in your view you can cache the result from this and use the calculated one to lower the load. Example:


from django.core.cache import cache

def heavy_view(request):
 cache_key = 'my_heavy_view_cache_key'
 cache_time = 1800 # time to live in seconds
 result = cache.get(cache_key)
 if not result:
 result = # some calculations here
 cache.set(cache_key, result, cache_time)
 return result

The process is simple, you ask the cache for a value corresponding to a given key(line 4). If the result is None you execute the code that generates it(line 8 ) and store it in the cache(line 9). My advice is to declare the key and time as variables cause this will ease their future changes. Cache usage(part II) - how to cache on template level: This is suitable for the cases when you have some heavy processing in the template(as regroup) or you want to cache only part of the template(as latest news section). Example:


{% load cache %}
 ... non cached content here ...
{% cache 1800 latest_news %}
 ... here are latest news - cached ...
{% endcache %}

The basic usage usage is {% cache time_in_seconds key %} ... {% endcache %} You can also cache code fragments based on dynamic properties, for example - current user recent conversations, just pass a 3rd param the uniquely identifies the code to be cached.


{% load cache %}
{% cache 300 recent_conversations request.user.id %}
 ... current user recent conversations - cached ...
{% endcache %}

Final words: as you see from the examples above using Django and Memcached is really easy. Using it correctly will speed up your website and respectively improve your user experience(UX) and SEO. Using it wrong will provide negative results. Just take a moment and think what can be cached, how long can it be cached and is there a reason to be cached. Try to avoid double caching - there is no need to use caching in templates and then cache the rendered template in the view too.

When to use caching for web sites ...

Tue, 12 Jul 2011 05:25:38 +0000

... five major question to ask yourself before using cache

After we learned about Why, Where, What and How of caching web sites now it is time to see when to use it.
The application cache is mainly used for speeding up and/or decreasing the load of frequently executed and(but not necessary) heavy resource using methods. So the first question is:

1) If you have method the consumes lots of CPU time and/or memory can it be optimised?

If you can optimize your code and make the method run faster and consume less resource than do this first and then reconsider whether you still need to cache it..

2) Will caching save you load/wait?

Have in mind that accessing cache has its own load. So caching the result of relatively light operations is pointless. Try to find where your biggest load/wait came from and use cache there.

3) For how long the cache should be valid?

This depend from how often the data is changed. We can split it in 3 major cases.

Case 1: The data changes on equal amounts of time, for example it is updated by cron. In this case you can set the expire time equal to the time interval in which the cron is runned. Example - you are reading a feed with the news from the last hour.

Case 2: The data is changed on random intervals of time - but not real time(i.e. from minutes to hours). In this case you should choose an average amount of time for which you think the data is persistent or even if it is changed you can serve the outdated one to the client. In this case the best way is to be able to invalidate cache when data changes. This is usable if the data is from the same type, for example - news list. Just add a line that invalidates cache on every news affecting operation(add/edit/delete). If you have composite data, for example mix of news, weather cast and currency rates for the day you'll just have to wait the cache to expire by it self.

Case 3: The data is updated real-time(every second, sometimes more than once in a second). If you are watching/playing on the market you really need real-time info for the current rates. But if don`t need the real-time date, for example you are displaying just informative graph with the daily changes you can cache and update it rarely(for example on every few minutes) then it changes.

So you have determined how long your cache should stay valid and the next question is:

4) Will the data be accessed more than once for the cache period?

If not then don't cache it or reconsider to increase cache validity time(question 2).

5) How big is the data to be cached?

Memcached has 1MB limit per cached item. For web(no matter is it site or application) this limit is fair enough for most/all of the cases. If you plan to store something bigger thing again and be sure you are not doing something wrong. If you really need to cache bigger amount of data consider to use another cache storage - database or file system.

If you are an advanced developer you will subconsciously know whether to use cache or not. But this questions may be really helpful if you are a novice.
I am open to hear what you ask yourself before using cache.

Update:

One more case when it is mandatory to use caching(if you can) is when you have frequent calls to an outer API. For example Retrieving Google Analytics Data.

Custom 404 Not Found page with Django CMS...

Sun, 03 Jul 2011 21:21:56 +0000

... or how to make user editable 404 page that stays in the pages tree of the CMS

Basics: Yes you need it! You need 404 page cause you never know what may happen to a link: bad link paste, obsolete or deleted article, someone just playing with your URLs etc. It is better for both you and your website visitors to have a beauty page that follows the website design instead of the webserver default one that usually contains server information which is possible security issue. With Django this is easy, just make a HTML template with file name 404.html, place it in you root template directory and voilà - you are ready. You will also automatically have a request_path variable defined in the context which caries the URL that was not found.

Problem: sometimes clients require to be able to edit their 404 pages. Or other times you need to use some custom context or you want to integrate plug-ins and be able to modify them easy trough the CMS administration. For example: you want do display your brand new awesome "Sitemap Plug-in" on this 404 page.

Solution: Django allows you to specify custom 404 handler view so you just need to define one, set it in urls.py and make it to render the wanted page:


# in urls.py
handler404 = 'site_utils.handler404' 

# in site_utils.py
from cms.views import details 
def handler404(request):
 return details(request, '404-page-url')

Where '404-page-url' is the URL of the page you want to show for 404 errors. So everything seems fine and here is the pitfall. If you use it this way your web page will return "200 OK" instead of "404 Not Found". This could kill your SEO(except if you want your 404 page as first result for your website). So you just need to add a 404 header to the response:


def handler404(request):
 response = details(request, 'novini')
 response.status_code = 404
 return response

Final words: Why are these HTTP status codes so important. The reason is that they tells the search engines and other auto crawling services what is the page status. Is it normal page, redirect, not found, error or something else. Providing incorrect status codes may/will have a negative effect on your website SEO so try to keep them correct, especially when it is easy to achieve as in the example above.

Note: If the code above is not working for you, please check Allan's solution in the comments

Django models ForeignKey and custom admin filters...

Wed, 01 Jun 2011 22:31:29 +0000

... or how to limit dynamic foreign key choices in both edit drop-down and admin filter.

The problem: Having a foreign key between models in django is really simple. For example:


class Question(models.Model):
 # some fields here

class Answer(models.Model):
 # some fields here
 question = models.ForeignKey(Question)

Unfortunately in the real live the choices allowed for the connection are frequently limited by some application logic e.g. you may add answers only to questions created by you. In some simple(rare) cases this can be easy achieved using choices or limit_choices_to attributes in the models.ForeignKey call. In the case of choices you just have to pass list/tuple each element of which contains the value to be stored and human readable name for the choice. Unfortunately this one is computed on server run and is not updated with new items during run-time. If you use limit_choices_to you may pass to it some kind of filter expression e.g. limit_choices_to = {'pub_date__lte': datetime.now} but this not always can do the job. Speciality: So if you want dynamic choices in the admin drop down you have to write a method that will return list with options and bind it to the form(as shown in Django forms ChoiceField with dynamic values…) which is used by admin. This will work great but if you decided to add this column in admin`s list_filter you will see all element from the connected model in filter. How to limit them to the same list used for the form choices? Solution: The simplest solutions is to extend RelatedFilterSpec, overwrite its default choices and add a single row to the model:


from django.contrib.admin.filterspecs import FilterSpec, RelatedFilterSpec

class CustomFilterSpec(RelatedFilterSpec):
 def __init__(self, *args, **kwargs):
 super(CustomFilterSpec, self).__init__(*args, **kwargs) 
 self.lookup_choices = get_questions() #this method returns the dynamic list

FilterSpec.filter_specs.insert(0, (lambda f: bool(f.rel and hasattr(f, 'custom_filter_spec')), CustomFilterSpec))

class Answer(models.Model):
 # some fields here
 question = models.ForeignKey(Question)
 question.custom_filter_spec = True # this is used to identify the fields which use the custom filter

Final word: The solutions is pretty simple and really easy to implement but should be used carefully. If your filtering method(in my case get_questions) is slow/resource consuming it may bring you troubles. Here is the place where and you should think about caching it. This is a place where a application cache can be used. Hope this will help you as much as it helped me.

Caching web sites and web applications...

Wed, 25 May 2011 22:41:37 +0000

...Why, Where, What and How of caching web sites

Basics of web page load: Every time when you open an webpage this results in a series of requests and responses between the client(your browser) and the web server that hosts the requested sites(most of the times this includes more the one servers). Each request tells the server that the client wants specific resource(CSS/JavaScript/image/etc.) and each response carries the resource content.

The main purposes of caching is to decrease bandwidth and to improve website performance.

Where are resources cached: According to the image above there are 2 places where content can be cached: on the client(browser cache) and on the server(server cache). The first one(browser cache) is more suitable when the client often requests single recourse again and again. As an example for this can be pointed CSS-files/JavaScripts/Images and other resources that form the page layout(these are requested on every page load). The second one(server cache) is hold on the server. It also can be split in two major pieces web server cache(when entire resource is cached) and application cache(when data chunks inside the application are cached).

How browser cache works: In order to tell the browser to cache such resources you have to supply each response with the following headers:

Last-Modified - datetime indication of when the content was last modified/updated on the server
Expires - the datetime after which the response will be considered "out of date"
Cache-Control - a directives about whether the resource must be cached and its maximal age of validity
Date - origin servers datetime

When the client request the resource for a first time the response contains both the resource data and the headers mentioned above. This tells the browser to store the response body(data) into its own cache. On the next request for the same resource the browser sends "If-Modified-Since" header with datetime value identical to the "Last-Modified" received from the server on the previous request. If the cache is still valid the web server returns HTTP Status code "304 Not Modified" which tell the browser to retrieve the content from its cache. Otherwise if the resource is modified between the two requests the response contains the new data along with the corresponding date headers.
This one is extremely useful to decrease websites bandwidth and speed up their loading because the common resources are read directly from the client and not pulled again and again from the server over the network.
For static resources this can be set in the webserver configuration. For dynamic ones(generated images or resources pulled from the database) these headers must be provided and checked from the application itself.

How server cache works: As I stated above the server cache can can be split into web server cache and application cache. Both are used when a single resource is requested by multiple of clients. For example the pages HTML is requested by every site visitor. If the content is the same for all of them then the HTML computation(for dynamically generated pages) can be done only for the first request and the latter ones can get the precomputed one from cache. For this you will need a caching proxy. The most common applications that are used for this purpose are Squid, Varnish and NginX

The process is similar to the one on the first figure but here the client communicates with the proxy instead of directly to the web server. If the proxy does not has a copy of the requested resource it gets if from the web server and then serves it to the client. Otherwise it pulls the content from its cache and return it directly.

Application cache: Sometimes caching full page is not an option. For example if you have a news website with two columns: one that is same for all users and one that is customizable by user preferences you can not serve every one with the same page. But you can pre-compute first column HTML, store it in the application cache and retrieve it when need instead of computing it for every user. The most common tool for this is Memcached. Its implementation is application specific but in pseudo code it works the following way:


key = 'cached_item_unique_key'
result = cache.get(key)
if not result:
 # do some computation here
 result = computed_result
 cache.set(key, result, is_valid_period)
return result

Each cached item is represented in the cache with unique key. When requested if there is such data in the cache it is pulled directly from there and server. Otherwise the date is computed, stored in the cache and then served. Server cache will decrease server load and allow your pages to be computed faster.

Final words: Caching is a double-edge razor. Using if carefully and with comprehension will make your life easier. Playing with it without knowing what you do may ruin your application. Do not cache rarely requested content. Do not try to cache everything, cache also has its limits. Use it wise.
If there is something not well explained, or missed, or wrong feel free to ask/comment. Your participation will be appreciated. Also if you find this article helpful share it and help other too.

Internet Explorer, jQuery, AJAX and HTML5...

Mon, 28 Mar 2011 20:59:02 +0000

... or how "invalid" HTML can break your JavaScript without a trace

The problem: A few days ago I had the following case. A script that I wrote was working everywhere except in the Internet Explorer(IE) 8 and 7. To be honest I don`t remember the exact error message but when I use the built-in debugger it point me to a row X in jQuery.min.js(those of you who have ever opened a minified JavaScript file know that it is practically unreadable) and not only this, but also provide me with an useless trace-back totally unrelated to the function called. So I decided to start executing the code that break line by line(good for me that I was activated on a "click" so I knew at least where to start).

Speciality: The extra hardness here came from the fact that the problem code was loaded as a result from an AJAX request so the debugger was powerless. So I started with the simplest approach I could imagine - alerting a text/value after every row to see where the error appears. The problem was in the following code:
[sourcecode type="javascript"] $('#my_element').html(data) Obviously there was something wrong during the execution of the html() method so I started exploring the results from the jQuery selectors. [sourcecode type="javascript"] alert($('container')) // result: [object Object] alert($('container').html) // result: function(a){...} alert($('container').html()) // result: blank (empty string)

I was getting close. The only thing left was to find why the html() method was returning an empty string when I was sure that there was something inside this element(I ran a quick check in FireFox) but I was stuck again.

Investigation: After a few minutes walking in the room later an old issue came to my mind. I ran into it while I was working on a social network project last year. Javascript/jQuery(I didn`t find which one or both) used to act strange when loading invalid HTML with AJAX inside a page under IE.
"Ok, probably a badly closed tag or similar or other fix in a second thing" was my first thought but... unfortunately it turned to be more complex.

Speciality 2: Did I mention it was an HTML5 page? Yeah, I now that IE7&8 does not support HTML5 so we used modernizr but how to recall it on an HTML fragment.

Solution: My favorite StackOverflow come in hand with reloading-modernizr-after-ajax. Following the answer I visited innerShiv, downloaded the plugin and tried again. This time no error but also no reaction on the click - It looked like I was skipping something. Although this it is not explicitly said it looks like that if you don`t pass "false" as second argument of the innerShiv call as in the innershiv jQuery notes the JavaScript code is not executed. I added it and the problem was fixed.

Conclusion: Should I say that I hate IE? Yes, No, Maybe. In the matter of fact I doubt that there is a web developer that uses IE as a primary browser but I have to confess that this time we were both guilty(our developers team and the IE). We used an "invalid code", IE breaks our JavaScript - "eye for an eye" or whatever it is called. The important here is not whose fault it is but to learn from our mistakes. Blaming a two versions old browser is not beneficial to anyone. But this was a good experience and I am sure that I`ll meet this problem again.

Querying ManyToMany fields in Django...

Wed, 23 Mar 2011 19:15:25 +0000

... or how to select items that have one or more categories matching with other item categories

The problem: having a Many To Many relation(ManyToManyField) is really useful when you have to link one object to many other. For example one artist may play in several styles, or a teacher can teach more than one subjects. I`ll use the latter example for the post. If you want to get all teachers that teach (for example) "music" you just have to do something like:


Teacher.objects.filter(subjects = music_obj)

The real problem comes when you want to select all teachers that have subject matching one or more of another teacher subjects. For example "teacher_a" teaches several subject and we want to find all his colleges that teach at least one of his. Solution: to achieve this we have to use the Q object(imported from django.db.models) in a way similar to the making of "OR" query.


from django.db.models import Q
x = Q()
for subject in teacher_a.subjects.all():
 x = x | Q(subjects = subject) 
colegues = Teacher.objects.filter(x).distinct()

Lets see what happens line by line:

1) import the Q class
2) create empty Q object
3) iterate over every subject taught by the specified teacher
4) on every iteration we add an "OR"-clause to the object corresponding to the current subject
5) we use the generated query as a filter

Have in mind that after using the Q object you have to sort your result(manually add order_by clause), because when you use Q the default ordering is not considered.

Final words: The speed of this method is not tested but to be honest I suspect that this can bring performance problems on big tables. In that case maybe you can speed the things up with custom SQL query. So the choice is yours - database independence or speed. Both ways have their pros and cons and its case specific which one you will choose.
If you have found a better way or want to add something feel free to comment.

Merar - The Emerging Markets Investment Network v2.0 ...

Sun, 20 Mar 2011 16:43:25 +0000

... or how we redesigned Merar`s website

If you follow me on Twitter maybe you have noticed that a week ago(to be more precise 11th of March) I tweeted about the new version of Merar. Unfortunately it took me 9 days before I find some time to post about what we do and what we expect. Before to continue with the changes I should mention that in "redesign" I mean some changes in the business model and logic as in the interface design(which I have to say is much simpler and intuitive then the previous one), so lets start with the changes:

The website is now split in two major sections: Investors or as we call it "Provide Funds" and Entrepreneurs or "Get Funds".
We have simplified the registration process so now you are just a few clicks away from becoming a member.
The process of submitting project was made simpler and some helpful hints are added to guide you during it.
There are also small changes in the messaging system, the member profile and some more.

We put big effort on usability and SEO on the design stage of the project and according to the final product and member responses we have done a good job.
My special thanks to all guys from Magic who were part of this project - the credits belong to them.

Now if you are still reading just go and try it. Sell your business(idea), invest your funds or just check it out. I`ll be glad to hear(read) your opinion.

I become a PhD Student

Mon, 14 Mar 2011 20:37:33 +0000

I am happy to say that I successfully passed the entry tests for a PhD student. So now I am part of SULSIT(State University of Library Studies and Information Technologies) PhD program. The program`s name is "Automated Systems for Information processing and information management" and my thesis will be "Research of the current methods and technologies for web sites and web application development". The dissertation have to be in Bulgarian but I will try to POST the table of content translated as well with a short resume for each chapter.
As always any ideas and suggestions are welcome.

Django CMS 2.2 features that I want to see

Mon, 28 Feb 2011 23:07:14 +0000

... this is the "Post on request" for February 2011

Preface: I have to admit that I was expecting a bigger interest in the "Post on request" topic but probably my blog is too young for this but I think I will try again(soon or not). The more important thing is that Jonas Obrist is the indisputable winner of this month "contest".

Features: One of the most useful features in the next Django CMS must be the ability to copy placeholder's content between different language version of one page. For example imagine that you have a home page with several placeholder each with several plugins/snippets inside(latest news, featured products etc.) when creating new language version of the page it is really annoying when you have to add this one by one.

One other feature requested by my colleague Miro is a little opposite of the one I want, he want to be able to add different page templates for each language version of the page. I also find this useful because sometimes your language version are not fully mirrored even on the same page.

Bugs: I am not sure that this is actually a bug but I think it is bad for SEO so I will mark it. If you have a multilingual website, once you have the language set in the cookie the same page is displayed no matter whether you have the language code in the URL. For example, "/en/news/" equals to "/news/". This causes a duplicate content which is considered bad for SEO and also is misleading because every visit of "/news/" with different language in the cookie return different content. I have done some fix for this that will be presented in the next post.

Conclusion: Thanks to all participants and to the guys at Django CMS - you do a really amazing job, I hope that you will like the features I proposed and that we will be able to see them in the next version. Comments and replies are welcomed as ever.

Portfolio

Sat, 19 Feb 2011 15:49:21 +0000

Bellow you can see some of the projects I have worked on.

Merar

I am proud to be a part of the team that created this network for connection between investor and entrepreneurs. Merar currently holds nearly 400 projects with total value of more than 7 billion dolars. Platform: Django.

Bulbrokers

Bulbrokers is the leader of brokerage services on the Bulgarian market. It is a multilingual website built with Django and Django CMS.

State Agency for Child Protection

This is the website of Bulgarian`s State Agency for Child Protection(only bulgarian version). Platform: Django, Django CMS.

Post on Request

Sun, 30 Jan 2011 21:13:22 +0000

In the last few weeks I have been little busy(and suffering from lack of inspiration) so my blog lacks of new posts but I see that there are people visiting it(especially for the Django CMS stuff) so I am willing to start a "Post on Request" practice.

The idea is simple - I will collect post theme requests in the first three weeks(or 21 days) of every month and after a choose based on your votes( and my decision ) I`ll go out with a post on the selected theme by the end of the week. Please keep your question in the following areas - Django, Django CMS, Web Development. Question in other areas will be considered too, but may stand out of my knowledge area. You can place your request here or check the list and vote for another one or just comment this post.
I hope that you will find this initiative interesting and join it.

Django forms ChoiceField and custom HTML output...

Thu, 06 Jan 2011 18:26:03 +0000

... or what to do in case that you need a special design for your choice fields

The problem: Few posts ago I talked(wrote) about Django forms ChoiceField and with dynamic values and now it is time to take a look at the front end and how we display these values to the user. I will use the code from that post:


class MyForm(forms.Form):
 my_choice_field = forms.ChoiceField(choices=MY_CHOICES)

By default Django provide us with a simple drop-down list as visual representation of the choice field. Just create and instance of the form in your view, pass it in the context:


def my_view(request):
 form = MyForm()
 return render_response('template.html',{'form': form})

and then display it in the template using {{form.my_choice_field}}. Speciality: For most cases this drop-down works just fine but sometimes this is not enough. Lets say that instead of drop-down list we need a radio buttons. There are two ways to achieve this - using a custom widget or playing with the template tags and the ChoiceField object. The first solutions is more suitable if you have a repeatable interface for several fields. In my case this was field specific so I chose the second one as faster for implementation. Solution: You can access the field object and its choices calling the field respectively field.choices properties of the form element in the template. Here is an example


<ul>
{% for choice in form.my_choice_field.field.choices %}
 <li>
 <input type="radio" name="my_choice_field" value="{{choice.0}}"
 {% ifequal form.my_choice_field.data choice.0 %} 
 checked="checked"
 {% endifequal %}/>
 <label for="">{{choice.1}}</label>
 </li>
{% endfor %}
</ul>

This allows you to specify custom styles/HTML for every element and also to distinct first and last elements using forloop.first and forloop.last - for more information you can check Django documentation about the for template tag

Final words: I did not have the chance to measure the speed difference between these two methods I mentions but I intend to post a also an example of the first one(using custom widget) and a "benchmark" between them both. If someone has already done this please post the results and your opinion about this.

Extending Django CMS Page model – part III

Tue, 04 Jan 2011 21:57:58 +0000

... or the drawback of Extending Django CMS Page model – part II

First of all please accept my apologies that I couldn't warn you for this drawback earlier but I was really busy in the last few weeks so let's skip directly to the problem.
If you remember in Extending Django CMS Page model – part II we added a method to the NavigationNode class that return an instance of the corresponding Page object. If we use it in the way show here:


from menus.base import NavigationNode
NavigationNode.page_instance = lambda u: Page.objects.filter(pk = u.id)[0]

The lambda function is executed every time when page_instance is called so this definetly increases the number of SQL queries made(+1 for every call of page_instance). Although this query is done using primary key and it is really fast this can bring you troubles. Especially if your hosting provider give you a limit of queries per time.
The good point is that it is not so big problem if you do it once or twice per page and it can be really useful. It all depends from how you use it. Also if you cache the output HTML you will skip the database overhead(but you have to be careful and make a mechanism that will invalidate the cache after page modifications)

(Almost) everything has its strong and weak sides, but using it carefully may bring you what you want. I really hope that someone will find this useful. If you do or you have found another weak/strong side do not hesitate to share your thoughts.

Extending Django CMS Page model - part II

Tue, 14 Dec 2010 22:50:51 +0000

... or how to use the extended model in your templates and especially in the built-in navigation

The problem: As you can see in the previous post there is a very simple way to extend the page model with some custom fields without changing the core code. But how to use your custom fields inside your templates? If you have an instance of the Page model everything is fine. You can just access its properties and everything is fine. Unfortunately there is a problem if you try it inside a custom template used for show_menu template tag.

Speciality: I was really surprised when I tried to access my custom properties inside the mentioned above template and nothing happened. After some investigation I found that the elements in show_menu are not instances of the Page model but of NavigationNode. And even more surprised that NavigationNode does not contain instance of the Page model in its properties. So I was in need to extend the NavigationNode class.

Solution: First of all I decided to make it easier to access the custom properties I created before, so I added the following code to it:


class PageAvatars(models.Model):
 page = models.ForeignKey(Page, unique=True, verbose_name=_("Page"),
 editable=False, related_name='extended_fields')
 big_avatar = FileBrowseField(_(u'Big Avatar'), max_length=255, blank=True)
 small_avatar = FileBrowseField(_(u'Small Avatar'), max_length=255, blank=True)
 
def _big_avatar(obj):
 for e_f in obj.extended_fields.all():
 return e_f.big_avatar
 return None
Page._big_avatar = _big_avatar 
Page.big_avatar = property(lambda u: u._big_avatar())

For simplicity I have skipped the imports, but you can check them in the previous post So now I have a "shortcut" for accessing the big_avatar and it is time to make the Page object available in the NavigationNode.


from menus.base import NavigationNode
NavigationNode.page_instance = lambda u: Page.objects.filter(pk = u.id)[0]

Line 2 defines a property("page_instance") that returns a Page object based on the primary key of the NavigationNode.id that equals to the corresponding Page primary key. Now you can use it in your custom navigation templates in the following way.


{% for child in children %}
 {{child.page_instance.big_avatar}}
{% endfor %}

Final words: I need to confess that I am really, really amazed how simply things can be done in Django and Django CMS. It is really awesome. I hope that you will find this post really helpful and useful. And as always do not be afraid to propose a better solutions or to ask for deeper explanation if something is not clear enough.

Extending Django CMS Page Model

Sun, 12 Dec 2010 18:48:21 +0000

... or how to add some fields to the Page model in the admin without changing Django CMS core

The problem: Some times the Page model just lack of something you need. In my case this was "page avatars". Long story short - I needed an image/avatar for each page in my CMS. So what I have to do was to add a field to the Page model that will keep the info about the avatar path.

Speciality: Of course the simplest solution was just to edit the file(pagemodel.py) that contain the Page model. Unfortunately in the common case(and in my too) several different application resides on a single server and share the same Django CMS. So changing the core code is inconvenient because every change on it will affect all applications that rely on the same CMS core. So what? Copying the whole Django CMS files in my app and modifying them? This does not seems like a good solution too. Fortunately there is a simple way to do that and leave the core code unchanged.

Solution: First you need to create a new model that will hold the new fields you need and a foreign key to the Page model.


from django.db import models
from django.utils.translation import ugettext_lazy as _
from cms.models.pagemodel import Page
from filebrowser.fields import FileBrowseField as FBF

class PageAvatars(models.Model):
 page = models.ForeignKey(Page, unique=True, verbose_name=_("Page"),
 editable=False, related_name='extended_fields')
 big_avatar = FBF(_(u'Big Avatar'), max_length=255, blank=True)
 small_avatar = FBF(_(u'Small Avatar'), max_length=255, blank=True)

After you define the model it is time to make the admin.


from extended_pages.models import PageAvatars
from cms.admin.pageadmin import PageAdmin
from cms.models.pagemodel import Page
from django.contrib import admin

class PageAvatarsAdmin(admin.TabularInline):
 model = PageAvatars
 
PageAdmin.inlines.append(PageAvatarsAdmin)

admin.site.unregister(Page)
admin.site.register(Page, PageAdmin)

The last two lines(11 and 12) are used to force Django to reload the Page model admin.
Now you have your new field as an inline in the Page admin.

Final words: As you can see the solution is simple, fast and the core code is unaffected. There is an unique constraint on the page field in PageAvatars model so you will not be able to add more than one avatar per page.
If there is something unclear in the code or you have a better idea I am ready to hear it.

Django forms ChoiceField with dynamic values...

Sun, 05 Dec 2010 21:28:48 +0000

... or how to get a dynamic drop-down list with Django forms

The problem: Lets say that you need a form with a drop-down list that have dynamic values. With Django this can be done simple and fast, but if you are new you may get yourself into a trap. In a standard form(with static values in the drop-down) your code will be something like this:


MY_CHOICES = (
 ('1', 'Option 1'),
 ('2', 'Option 2'),
 ('3', 'Option 3'),
)
 
class MyForm(forms.Form):
 my_choice_field = forms.ChoiceField(choices=MY_CHOICES)

So if you want the values to be dynamic(or dependent of some logic) you can simply modify your code to something like this:


def get_my_choices():
 # you place some logic here
 return choices_list

class MyForm(forms.Form):
 my_choice_field = forms.ChoiceField(choices=get_my_choices())

and here you will fail(not absolutely). This is a common mistake and sooner or later you will see what you messed but it my be too late. Speciality: the trick is that in this case my_choice_field choices are initialized on server (re)start. Or in other words once you run the server the choices are loaded(calculated) and they will not change until next (re)start. This mean that your logic will be executed only once. This will be OK if your logic is server specific(depends from server settings) or "semi-dynamic" in any other way. But you need this list to be updated on every form load you will need to use something the form __init__ method. Solution: fortunately the form`s class has an __init__ method that is called on every form load. Most of the times you skipped it in the form definition but now you will have to use it.


def get_my_choices():
 # you place some logic here
 return choices_list

class MyForm(forms.Form):
 def __init__(self, *args, **kwargs):
 super(MyForm, self).__init__(*args, **kwargs)
 self.fields['my_choice_field'] = forms.ChoiceField(
 choices=get_my_choices() )

You first call the __init__ method of the parent class(Form) using the super keyword and then declare your dynamic fields(in this case my_choice_field). With this code get_my_choices is called on every form load and you will get your dynamic drop-down.

Final words: have in mind that this method also have one strong drawback. If your application have a heavy load, executing a logic on every form load may(will) bring you troubles. So some caching of the form or the logic result may be useful.
And remember If you have other approach, idea or question I am always ready to hear it.

SEOmoz meetup in Sofia, Bulgaria

Tue, 02 Nov 2010 21:51:50 +0000

First of all I want to thank to Rand Fishkin from SEOmoz and the Webit team for organizing this meetup.

For a long time in the web world there was a feud between the developers and the SEO guys. Both groups claiming that their work is more important, more sophisticated and with bigger value to the client and website visitors. A have to admit that(as a developer) I was also focused only on my work ignoring the SEO part and leaving it to the others. But for the last year I started to give more attention to this field.

Do not get me wrong, I am not going to become a SEO Expert but the truth is that like every other thing the web is complex system. For example - you can hire the best auto engineer but if he is alone you will never get a car that sells. You need a team - engineers, designer, managers, ergonomics experts etc. Everyone of them should focus in his field but also to cooperate with others to provide the optimal solutions.

As a developers our task is to know the basic principles of the semantic web and search engine optimization and to build applications conformable with them. Then the SEO experts come and do their magic. But it is a process that have to be walked hand by hand for both of us. It is always easier if it is made by design(architecture) the to rewrite it later.

As a final word I want to state that I have a really great time at this meeting and I hope that this was my first but not last attendance on event of this kind.

Back to the Tables...

Thu, 30 Sep 2010 22:07:13 +0000

... or how table based layout still lives

When I first started to write HTML (about ten years ago) the web has nothing common to the current one. Modem connections, IE4/IE5 were the major browsers and Netscape was far far away in the competition. These were dark ages for the web. CSS was a new conception that every one mentions but few dared to use. These was the ages of the table based layouts - hard to do and impossible to maintain. Multiple colspans and rowspans, nested table, absolute HTML horror until you(or your boss) decide that "this page need some minor improvement" this was when the real hell started. One less or extra column usually destroys the whole layout. These may seem like a bad coding practice or joke to most of you but I`m sure that everyone that walked this way will agree with me about this.

Years later when I realized the power of CSS and started to build CSS based layouts with floated divisions I was like a new born. It was amazing - simple, clean, easy to build and even easier to maintain. Yes there was some other holes where to fall like box models and other browsers differences but it was a great improvement. Then I learned that table were for table-data only, everything else should be done with CSS and I was happy.

In the last years I was mainly focused on back-end development and the JavaScript part from the front-end so I have to confess that I was little away from all this HTML&CSS stuff. I only write simple things or take deeper when some of the front-end developer faces a problem. But how you all know a man always goes back to its roots so it was my turn now. I have a simple template to code from PSD to HTML/CSS. Header, two fixed width columns, some rounded corners nothing to worry about so I sit down and started. I was pretty happy with the result until I decided to check is there something special that I must have in mind for these e-mail newsletters.

What? Didn`t I tell you? Yes it was a template for a newsletter that is going to be send to users via e-mail. From the time when I started to use CSS layout I have never thought that there will be a time when I will go back and do table based layouts again but it looks like contrasting to the browsers the e-mail clients haven`t evolved in much in the last 6-7 years(at least when we talk about CSS support). So the tables are back in the game and these are some simple rules I found over the net that you must have in mind when you do html e-mails.

Use table to build your layout
Do not declare your CSS in the head tag, use inline styles insted of this
Avoid using background images
Avoid using images at all(just joking but have in mind that you have to check how your mail will look without them cause most of the mail clients did not load images until you confirm that you really want them )
If you are going to use images my personal advice is to use absolute URLs instead of adding the images to the mail. This will save bandwidth of your customers(especially for the ones that use mobile devices), you will be able to change and image every time when you need(if you noticed a problem with am image after the mail is send)

There are several companies that offer e-mail testing services one if them is Campaign Monitor. I have to confess that their price seems a little high to me but it depends from project requirements. The good thing is that the provide absolutely free e-mail clients CSS support table. I found it very useful and that you will do too.

Another solutions is Litmus they have lower prices but their test support less clients.

A free but just for basic tests is Email on Acid

If you know any other(especially free) simiral services please share the with me.
I hope that this will help to someone else, and if you have any questions fee free to ask.

Google Analytics referring sites and https

Thu, 16 Sep 2010 20:16:08 +0000

The problem: recently a client reported the he does not see another site he owns in the list of referring sites in his Google Analytics account(for his main website) but was absolutely sure that he recieves trafic from this website.

The research and the reason: at first I thought that the problem is cause by the 301 redirects the site use to send you to the propper language version of the website(cookies based). But after some reasearch I found that these 301 redirects keep the original referrer so the problem was somewhere else. So I open a page from the second site, click a link to the main one and run "document.referrer" in the console, strange but the answer was empty. I tried it clicking link from other site and everything was ok, the result was the referring site so the problem really was with this one. After some more wondering what is really going on I finally noticed that the problem website was working under secured(https) connection.
According to RFC2616 section 15.1.3

Clients SHOULD NOT include a Referer header field in a (non-secure) HTTP request if the referring page was transferred with a secure protocol.

OK, so the reason for the problem was found but how to solve it?

The solution: Google Analytics allows you to override the referral URL so the only thing you need is to add get parameter to the link url with the url of the pointing page, for example: http://main.com/news/?referral_site=second-website.com
Then place some logic that will check for "referral_site" parameter and use the_setReferrerOverride() gladly provided by the GA team to override the referral address.

If you have another solutions or questions please let me know.

Python comparisons speed depends from the result

Wed, 18 Aug 2010 09:52:39 +0000

Recently I decided to check whether "less than or equal"(<=) is slower than "bigger"(>) and I was surprised from the result. In my case "bigger" was slower. I was amazed, according to the simple logic in the case of less then or equal we need one or two operation(for example we first check for equality and the for then for "is lower"), so I asked at stackoverflow what causes this and here is the answer.

The speed of the operation does not depends from the operation itself but from the result. When the result is false it is calculated faster the if it is true. You can see the disassembler trace here.

Have you found other operations when python acts "strange"?

Google stops Google Wave development

Thu, 05 Aug 2010 10:35:34 +0000

As stated in the official Google blog Google will discontinue Wave development(at least as an independent product). I looks like low user interest is the main reason for this decision.

We have to admin that Wave creates a new point of view on the web technologies and it is little sad that this "new wave" meet its end so quickly but I doubt that Google will completely forgot Wave. I am pretty sure that we will see many of the Wave features in the upcomming Google social network(Google Me).

Also the main parts of the code are availabe as open source so I think that there will be a lot of developers who will continue to use and improve Wave features.

Google and social networks - wins and failures

Tue, 27 Jul 2010 15:26:43 +0000

...will google finaly place itself on the top of social networks?

Have you ever heard of Orkut?

Most of the people will say "No". This was the first try of Google to take place in the world of the social networks. Ok, maybe not the first. I have to admit the the first was in the late 2003rd when Google offered to purchase Friendster, but their offer was declined. So they decided to start own project and in 2004th Orkut was launched with the hope that it will dethrone Facebook from the leading position. Although the big interest in the start it didn`t succeed to take the first place(by registered users) and is now on number 7 with 100 milion users. Some people blame Brazilians for overwhelming it and repulsing other members by using not English but Portuguese in social discussions. Since then Orkut is widely used in Brazil and India but not popular in the other countries.
So this was big failure number one(if you can call failure 100 milion users.)

Google Buzz

This was the second attempt, this time the "attack" was pointed in the area of short messages. Again it couldn`t reach the size of its main competitor Twitter.

Google Me

Digg founder Kevin Rose, posted on Twitter the rumour that Google are going to intriduce a Facebook competitor. SF weekly posted an article that this is true and Google have already allocated many engineers to this project.

The only thing left to us is to wait and see will Google finaly succeed to take the throne of social networks king.

Nokia 5230 Express Music - GPS Navigation

Tue, 25 May 2010 21:33:48 +0000

The Myths about Integrated and Assisted GPS

A week ago I bought Nokia 5230 from M-tel mainly because of the integrated GPS. This is my first meet with this kind of device and after some GPS "problems" I`m ready share my experience(At the end of this post there is explanation how to turn on/off different types of positioning methods).

Using Assisted + Integrated GPS the positioning is completed for a few seconds(outdoors) but(I noticed that later) this results in small GPRS traffic(mainly for the "cold start") which costs some money so I tried to avoid it and turned "Assisted GPS" off.

Later when I try it with "Integrated GPS" only it was unable to connect to satellite, so I started reading forums and post what is going on. In this discussion I learned about the TTFF(Time To First Fix) and after reading that this fix can take an hour I went mad. Fortunately the real tests give better result.

In my case the positioning takes few minutes(less than 5) when I was standing still, on a clear place and holding phone in a way that my hand doesn`t cover the GPS antenna. After this time no other issues and connection loss. For a trip that contains of about 10 minutes in a bus, 10 walking(the phone was in my pocket), 15 indoors(in a shop) and 10 more walking it didn`t loose satellite connection. So don`t worry that you`ll have to a pay a lot of money for GRPS traffic, bluetooth GPS device or else. Just hold you position for a few minutes and wait the phone to connect, after this the device will be working absolutely fine. Have in mind that seeing satellites in the "Satellite status" doesn`t mean that your position is already calculated. Wait until you see the green bars on the bottom of the map(instead of the yellow one). The satelite status can be check in Menu->Location -> GPS data ->Position -> Options -> Satellite Status

How to turn on/off different types of positioning methods: My phone originally had older version of Ovi Maps(maybe 2.x) and I updated it with the latest one(3.03). After installing it I noticed the "Location" application in Menu->Applications(I`m not absolutely sure whether it was there before the update). If you open it you will seen 3 choices "Landmarks", "GPS Data" and "Positioning". From "Positioning" choose "Positioning method" and you will see 4 options - "Asissted GPS", "Integrated GPS", "Bluetooth GPS" and "Network based".

Have a nice navigation and feel free to ask if you have any questions.

GAE, Timezones and weekdays...

Sun, 23 May 2010 08:03:17 +0000

... or how to find the date of week in specified timezone, no matter where your server is.

The problem: I maintain a site targeting user in single country(single timezone) and I have to create administration based on a day of week(show this on Monday, that on Tuesday etc.). The site is based on Google App Engine Platform.

The simplest and most obvious solutions is to take the current time and add/substract the difference between server timezone and your target timezone. Unfortunately this is not going to work in our case.

Speciality: As you may know google have multiple data centers all over the world.

(more info at
royal.pingdom.com.)

So you can not be sure which server is currently responsible for your app. Also you have a DST(daylight saving time) that also must be included in calculation. Fortunately you can use UTC time and
PyTZ for GAE.

Solution:


from datetime import datetime, timedelta
from pytz import timezone
import pytz
bg_zone = timezone('Europe/Sofia')
bg_zone_time = bg_zone.localize(datetime.utcnow())
day_of_week = bg_zone_time.strftime("%w")

Explanation:
#5) create timezone object corresponding to your timezone(in my case this is Europe/Sofia)
#6) get the correct time for the specified timezone calling localize method of the timezone object with current UTC time as parameter
#7) using time formatting to get the day of week. There is simple catch here, at first I try it with weekday() method and get incorrect results. It looks like the method ignores the timezone/DST correction, so you have to use strftime().

Final words: I didn`t test all methods so I couldn`t say which one ignores the timezone and which not, but I hope this article will be helpful for someone facing this or similar problem. Any correction and code improvements will be appreciated.

About Me

Wed, 19 May 2010 21:57:30 +0000

Hi, my name is Ilian Iliev. I love programming especially when it involves Python and Django. I graduated from technical school in 2003 with a speciality "Radio and Television Technologies". Four years later(2007) I achieved bachelor degree in heat engineering, but programming have always been closer to my heart. During my engineering education I was working in the filed of web developing, and here I am now, having master degree in information technologies, professional web developer, but still having some interest in engineering stuff. I also like good music, cooking, fishing, traveling, playing chess, going out with friends and many other. So this is how I live between developing, engineering and real live.

Feel free to contact me at:

Mail: ilian@ilian.io

Linked in: iliansiliev

Or just leave a comment under any article.

Lets start!

Wed, 19 May 2010 21:57:30 +0000

Hi, and welcome to my blog. I`m Ilian Iliev, software engineer(with heat engineering background) and passionate web developer. Here I`ll post articles about programing(python, django, google app engine etc.) as well as my adventures in real world - music, travel, chess and many others. Hope that you will find the useful and interesting, feel free to comment them and contact me. Also English is not my native language so you`re welcome to correct me if I mess it up.

I`m glad you`re here but now go and check my newest posts.