Posts tagged with Technologies

Service Discovery with Smartstack

Published at July 3, 2016 | Tagged with: , ,

A week of innovation

First of all I want to thank Lifesum for having another "Innovation Week", it is a great opportunity and I hope that more companies will start following it. In a few words the idea is to allow everyone from the company to freely pick a project or idea that they want to develop and and work on it for one week. The benefits range from just making people happy because of the break of the routines and the opportunity to work on something a bit different, to seeing some pretty amazing prototypes that can be easily implemented in the company product.

What is Service Discovery?

In summary service discovery is the possibility of the separate services in scalable infrastructure to communicate with each other and to the outside world. In other words - how to route the requests to the corresponding service while providing balanced load on the instances in the pool and monitoring their health. Sounds simple, right?

Well, unfortunately service discovery in the real world is not that simple. In my presentation from the Stockholm Python MeetUp I talked a bit more about the complexity of service discovery, the suboptimal solutions and Smartstack - a solutions invented at AirBnB for simplifying the whole process. You can see more about it in my presentation:

AirBnB service auto discovery from Ilian Iliev

However, the whole idea sounded so awesome that me and my colegue Esma decided to team up on that and try to explore a bit more the opportunities that Smarstack provides for us. In the matter of fact we decided to explore two different approaches: Smartstack and Consul. However we had some issue with the Consul setup and we found that it is not acting exactly the way we so at the end we focused all our attention to Smarstack.

How does it work?

Smartstack consists of two main components - Nerve and Synapse. Never handles the service registration while Synapse reads the information about the available services and configures a local HAProxy that plays as a load balancer for the service pool. For our tests we used Zookeeper as register for the services.

What have we built?

We created a small project consisting of a pool of Zookeeper instances, two node for service A and one node for service B. We tested multiple scenarios of crashes of one or more nodes, both zookepers instances and service instances, how the systems operates during the crashes and how it recovers after the nodes are brought back on.

Results

As a result we created a public repository service-discovery giving details about the whole setup process, our tests and their results. Actually now is the time to praise Esma and her awesome work on conducting the tests and writing the great documentation from the repo above. So, if you have ever wondered about how service discovery works or you just want to test Smartstack, just clone the repo and follow the instruction. We would be happy to hear about your experience.

Python is not a Panacea ...

Published at June 11, 2012 | Tagged with: , , , ,

... neither is any other language or framework

This post was inspired by the serial discussion on the topic "Python vs other language"(in the specific case the other one was PHP, and the question was asked in a Python group so you may guess whether there are any answers in favor of PHP). It is very simple, I believe that every Python developer will tell you that Python is the greatest language ever build, how easy is to learn it, how readable and flexible it is, how much fun it is to work with it and so on. They will tell you that you can do everything with it: web and desktop development, testing, automation, scientific simulations etc. But what most of them will forgot to tell you is that it is not a Panacea.

In the matter of fact you can also build "ugly" and unstable applications in Python too. Most problems come not from the language or framework used, but from bad coding practices and bad understanding of the environment. Python will force you to write readable code but it wont solve all your problems. It is hard to make a complete list of what exactly you must know before starting to build application, big part of the knowledge comes with the experience but here is a small list of some essential things.

  • Write clean with meaningful variable/class names.
  • Exceptions are raised, learn how to handle them.
  • Learn OOP(Object Oriented Programming)
  • Use functions to granulate functionality and make code reusable
  • DRY(Don't Repeat Youtself)
  • If you are going do develop web application learn about the Client-Server relation
  • Use "layers" to seprate the different parts of your application - database methods, business logic, output etc. MVC is a nice example of such separation
  • Never store passwords in plain text. Even hashed password are not completely safe, check what Rainbow Tables are.
  • Comment/Document your code
  • Write unit test and learn TDD.
  • Learn how to use version control.
  • There is a client waiting on the other side - don't make him wait too long.
  • Learn functional programming.

I hope the above does not sounds as an anti Python talk. This is not its idea. Firstly because there are things that are more important than the language itself(the list above) and secondly because... Python is awesome )))
There are languages that will help you learn the things above faster, Python is one of them - built in documentation features, easy to learn and try and extremely useful. My advice is not to start with PHP as your first programming language it will make you think that mixing variables with different types is OK. It may be fast for some things but most of the times it is not safe so you should better start with more type strict language where you can learn casting, escaping user output etc.

Probably I have missed a few(or more) pointa but I hope I've covered the basics. If you think that anything important is missing, just add it in the comments and I will update the post.

Software for business

Published at Jan. 24, 2012 | Tagged with:

I am starting a new blog. The reason is that I want to keep this one more technically oriented while the other will be more business and customers oriented. Its name is Software for business and the idea is to show to the business in less technical details how the modern IT technologies like CRM & ERP systems, web sites, e-commerce solutions, online marketing and so on can help their business.
If you find the topic interesting feel free to join.

Caching websites with Django and Memcached...

Published at July 12, 2011 | Tagged with: , , , , , , ,

... memcached installation, django configuration and how to use it for your website

After Caching web sites and web applications and When to use caching for web sites its time for a little sample. This sample shows the usage of Memcached for server-side application cache. The installation part is taken from my Ubuntu so it may differ depending from your OS/distribution.

What is Memcached: Memcached is a tool that allows you to store key-value pairs in you memory. The keys are limited to 250 Bytes and for better performance the value size is limited to 1MB(more details) but this size is fair enough for web usage.

Memcached installation:

apt-get install memcached
apt-get install python-memcache
The first line installs Memcached and the second one install Python API for communication between your application and Memcached daemon. After this the Memcached daemon is up and running. With default configuration it runs on port 11211 on localhost(127.0.0.1). If you want to modify this the configuration file(in my case) is situated in /etc/memcached.conf Django configuration: This one depends from the Django version that you use. For 1.2.5 and prior the next code should by added in your settings file(settings.py):
CACHE_BACKEND = 'memcached://127.0.0.1:11211/'
For 1.3 and development version add:
CACHES = {
    'default': {
        'BACKEND': 'django.core.cache.backends.memcached.MemcachedCache',
        'LOCATION': '127.0.0.1:11211',
    }
}
In both cases if you use different port and/or IP you have to replace them above. More info about cache backend configuration you can find in Django documentation docs. So now you have Memcached running and Django configured. If you have doubts about is this suitable/usable in you case take a look at the posts mentioned above or just add comment with your case and I will be happy to give you an advice. Now it is time to start using it. Cache usage(part I) - how to cache on Python level: If you have some heavy calculations in your view you can cache the result from this and use the calculated one to lower the load. Example:
from django.core.cache import cache

def heavy_view(request):
    cache_key = 'my_heavy_view_cache_key'
    cache_time = 1800 # time to live in seconds
    result = cache.get(cache_key)
    if not result:
        result = # some calculations here
        cache.set(cache_key, result, cache_time)
    return result
The process is simple, you ask the cache for a value corresponding to a given key(line 4). If the result is None you execute the code that generates it(line 8 ) and store it in the cache(line 9). My advice is to declare the key and time as variables cause this will ease their future changes. Cache usage(part II) - how to cache on template level: This is suitable for the cases when you have some heavy processing in the template(as regroup) or you want to cache only part of the template(as latest news section). Example:
{% load cache %}
 ... non cached content here ...
{% cache 1800 latest_news %}
    ... here are latest news - cached ...
{% endcache %}
The basic usage usage is {% cache time_in_seconds key %} ... {% endcache %} You can also cache code fragments based on dynamic properties, for example - current user recent conversations, just pass a 3rd param the uniquely identifies the code to be cached.
{% load cache %}
{% cache 300 recent_conversations request.user.id %}
    ... current user recent conversations - cached ...
{% endcache %}

Final words: as you see from the examples above using Django and Memcached is really easy. Using it correctly will speed up your website and respectively improve your user experience(UX) and SEO. Using it wrong will provide negative results. Just take a moment and think what can be cached, how long can it be cached and is there a reason to be cached. Try to avoid double caching - there is no need to use caching in templates and then cache the rendered template in the view too.

Caching web sites and web applications...

Published at May 25, 2011 | Tagged with: , , , , , , , , , , ,

...Why, Where, What and How of caching web sites

Basics of web page load: Every time when you open an webpage this results in a series of requests and responses between the client(your browser) and the web server that hosts the requested sites(most of the times this includes more the one servers). Each request tells the server that the client wants specific resource(CSS/JavaScript/image/etc.) and each response carries the resource content.

Client-Server

The main purposes of caching is to decrease bandwidth and to improve website performance.

Where are resources cached: According to the image above there are 2 places where content can be cached: on the client(browser cache) and on the server(server cache). The first one(browser cache) is more suitable when the client often requests single recourse again and again. As an example for this can be pointed CSS-files/JavaScripts/Images and other resources that form the page layout(these are requested on every page load). The second one(server cache) is hold on the server. It also can be split in two major pieces web server cache(when entire resource is cached) and application cache(when data chunks inside the application are cached).

How browser cache works: In order to tell the browser to cache such resources you have to supply each response with the following headers:

  • Last-Modified - datetime indication of when the content was last modified/updated on the server
  • Expires - the datetime after which the response will be considered "out of date"
  • Cache-Control - a directives about whether the resource must be cached and its maximal age of validity
  • Date - origin servers datetime

When the client request the resource for a first time the response contains both the resource data and the headers mentioned above. This tells the browser to store the response body(data) into its own cache. On the next request for the same resource the browser sends "If-Modified-Since" header with datetime value identical to the "Last-Modified" received from the server on the previous request. If the cache is still valid the web server returns HTTP Status code "304 Not Modified" which tell the browser to retrieve the content from its cache. Otherwise if the resource is modified between the two requests the response contains the new data along with the corresponding date headers.
This one is extremely useful to decrease websites bandwidth and speed up their loading because the common resources are read directly from the client and not pulled again and again from the server over the network.
For static resources this can be set in the webserver configuration. For dynamic ones(generated images or resources pulled from the database) these headers must be provided and checked from the application itself.

How server cache works: As I stated above the server cache can can be split into web server cache and application cache. Both are used when a single resource is requested by multiple of clients. For example the pages HTML is requested by every site visitor. If the content is the same for all of them then the HTML computation(for dynamically generated pages) can be done only for the first request and the latter ones can get the precomputed one from cache. For this you will need a caching proxy. The most common applications that are used for this purpose are Squid, Varnish and NginX

The process is similar to the one on the first figure but here the client communicates with the proxy instead of directly to the web server. If the proxy does not has a copy of the requested resource it gets if from the web server and then serves it to the client. Otherwise it pulls the content from its cache and return it directly.

Application cache: Sometimes caching full page is not an option. For example if you have a news website with two columns: one that is same for all users and one that is customizable by user preferences you can not serve every one with the same page. But you can pre-compute first column HTML, store it in the application cache and retrieve it when need instead of computing it for every user. The most common tool for this is Memcached. Its implementation is application specific but in pseudo code it works the following way:

key = 'cached_item_unique_key'
result = cache.get(key)
if not result:
   # do some computation here
   result = computed_result
   cache.set(key, result, is_valid_period)
return result

Each cached item is represented in the cache with unique key. When requested if there is such data in the cache it is pulled directly from there and server. Otherwise the date is computed, stored in the cache and then served. Server cache will decrease server load and allow your pages to be computed faster.

Final words: Caching is a double-edge razor. Using if carefully and with comprehension will make your life easier. Playing with it without knowing what you do may ruin your application. Do not cache rarely requested content. Do not try to cache everything, cache also has its limits. Use it wise.
If there is something not well explained, or missed, or wrong feel free to ask/comment. Your participation will be appreciated. Also if you find this article helpful share it and help other too.

Google stops Google Wave development

Published at Aug. 5, 2010 | Tagged with: , , , , ,

As stated in the official Google blog Google will discontinue Wave development(at least as an independent product). I looks like low user interest is the main reason for this decision.

We have to admin that Wave creates a new point of view on the web technologies and it is little sad that this "new wave" meet its end so quickly but I doubt that Google will completely forgot Wave. I am pretty sure that we will see many of the Wave features in the upcomming Google social network(Google Me).

Also the main parts of the code are availabe as open source so I think that there will be a lot of developers who will continue to use and improve Wave features.

Nokia 5230 Express Music - GPS Navigation

Published at May 25, 2010 | Tagged with: , , , , ,

The Myths about Integrated and  Assisted GPS

A week ago I bought Nokia 5230 from M-tel mainly because of the integrated GPS. This is my first meet with this kind of device and after some GPS "problems" I`m ready share my experience(At the end of this post there is explanation how to turn on/off different types of positioning methods).

Using Assisted + Integrated GPS the positioning is completed for a few seconds(outdoors) but(I noticed that later) this results in small GPRS traffic(mainly for the "cold start") which costs some money so I tried to avoid it and turned "Assisted GPS" off.

Later when I try it with "Integrated GPS" only it was unable to connect to satellite, so I started reading forums and post what is going on. In this discussion I learned about the TTFF(Time To First Fix) and after reading that this fix can take an hour I went mad. Fortunately the real tests give better result.

In my case the positioning takes few minutes(less than 5) when I was standing still, on a clear place and holding phone in a way that my hand doesn`t cover the GPS antenna. After this time no other issues and connection loss. For a trip that contains of about 10 minutes in a bus, 10 walking(the phone was in my pocket), 15 indoors(in a shop) and 10 more walking it didn`t loose satellite connection. So don`t worry that you`ll have to a pay a lot of money for GRPS traffic, bluetooth GPS device or else. Just hold you position for a few minutes and wait the phone to connect, after this the device will be working absolutely fine. Have in mind that seeing satellites in the "Satellite status" doesn`t mean that your position is already calculated. Wait until you see the green bars on the bottom of the map(instead of the yellow one). The satelite status can be check in Menu->Location -> GPS data ->Position -> Options -> Satellite Status

How to turn on/off different types of positioning methods: My phone originally had older version of Ovi Maps(maybe 2.x) and I updated it with the latest one(3.03). After installing it I noticed the "Location" application in Menu->Applications(I`m not absolutely sure whether it was there before the update). If you open it you will seen 3 choices "Landmarks", "GPS Data" and "Positioning". From "Positioning" choose "Positioning method" and you will see 4 options - "Asissted GPS", "Integrated GPS", "Bluetooth GPS" and "Network based".

Have a nice navigation and feel free to ask if you have any questions.