Yet another project : Links

dimanche, 4 janvier 2015

In june, I talked about technical choices for a web-based todo-list management app (article in french). The main goal of this project was to try out redis. It was a success, and I've recently added redis in a most important project.

One of the tools I wanted to test was MongoDB. As I did for Redis, I first give a look to The Little MongoDB Book to get some informations about mongo.

What's MongoDB

The best known concept about MongoDB, is that's a NoSQL storage system. It provides several layers to access data

+------------------------+
|         U S E R        |
+------------------------+
            V   First you connect the server
+------------------------+
|  C O N N E C T I O N   |
+------------------------+
            V   Then you choose a `DB'
+------------------------+
|     D A T A B A S E    |
+------------------------+
            V   Then a `collection'
+------------------------+
|  C O L L E C T I O N   |
+------------------------+
            V   Then you `documents'
+------------------------+
|    D O C U M E N T S   |
+------------------------+

Mongo uses BSON format to store data and is a schema-less system : this means that you can store completly differents documents in the same collection.

Finally, you have that it provides drivers for a lot of languages. For this project and article, I've chosen pymongo which is the recommended driver for python.

About the project

The project itself is quite simple. We want to show a list of interesting URLs that any user can feed.

We'll need 2 pages :

  • the list itself (answering the route /)
  • the form to add a link (route : /new)

Each time a link is clicked, we'll increment a hits counter.

This small app will be enough to talk about most important concepts of MongoDB.

Coding

First of all, let's write the basic code of a BottlePy app :

#!/usr/bin/env python2
#-*- encoding: utf8 -*-

# Links app

import sys
import os
from datetime import datetime
from bson.objectid import ObjectId
import pymongo
from bottle import\
        run, \
        debug, \
        request, \
        static_file, \
        get, \
        post, \
        redirect, \
        jinja2_template as template

# MongoDB db name
DBNAME="links"
# Absolute path to static files
STATIC_ROOT="/home/matael/workspace/learn/python/links/static"

#### Generic Views ####

@get('/static/<filename:path>')
def send_static(filename):
    """ Serves static files """
    return static_file(filename, root=STATIC_ROOT)

# here will go the code

# for debug purpose only
debug(True)
run(reloader=True) # reload everytime a file changes

Looking at this short piece of source code, we can spot a few things :

  • jinja2 will be used for templating
  • datetime is here just for sorting
  • bson will be useful to work with the automatically added _id field.
  • DBNAME identify the database name in mongo

We'll also need a base template :

<!DOCTYPE html>
<html lang="en">
    <head>
        <title>Links</title>
        <meta charset="utf8"/>
        <link href='http://fonts.googleapis.com/css?family=Overlock' rel='stylesheet' type='text/css'>
        <link rel="stylesheet/less" href="static/main.less" type="text/css"/>
        <script type="text/javascript" src="static/less-1.3.0.min.js"></script>
    </head>
    <body>
        <header>
        <h1><a href="/">Links</a></h1>
        </header>
        <section>
        {% block content %}{% endblock %}
        </section>
        <footer>
        <p><a href="http://sam.zoy.org/wtfpl/">WTFPL</a> 2012 | Powered by <a href="https://github.com/Matael/links">Links</a> - <a href="http://blog.matael.org">Matael</a> | <a href="/new">Add</a></p>
        </footer>
    </body>
</html>

Nothing to say here. Just note we'll render our stylesheet using LessCSS

Database connection

Let's write a function make database connection easier :

#### Tools ####
def connect_db():
    db = pymongo.Connection()   # Connect instance
    db = db[DBNAME]             # Select the right DB
    return db.links             # Return a cursor on the right collection

Ok. So we have a base and a easy way to connect the db and collection.

Data structure

Each entry will be recorded a hashtable like this one :

{
    _id: ObjectId('the id'), // automatically generated
    poster: "poster's pseudo",
    url: "http://example.com",
    title: "The title",
    hits: 42, // how many times the link had been clicked
    date: date(the date) // insert date
}

Home

Let's write the handler for / :

@get('/')
def home():
    """ Home page for a GET request """
    db = connect_db() # connection to DB

    # fetch all entries, sorting them by date descending
    result = db.find().sort('date', pymongo.DESCENDING)

    # render the template
    return template("templates/home.html", result=result)

This view is quite simple and easy to understand. find and sort are two standard MongoDB commands. The first one fetch documents and the second... sort them (here, by descending dates).

The corresopnding template is the following :

{% extends 'templates/base.html' %}
{% block content %}
<article>
    {% for r in result %}
    <p class="link">
        <span class="l_hits">{{r['hits']}}</span>
        <span class="l_link"><a href="/goto/{{r['_id']}}">{{r['title']}}</a></span>
        <span class="l_poster">par {{r['poster']}}</span>
    </p>
    {% endfor %}
</article>
{% endblock content %}

First line explain that this code will go into another template. Then we specify the block where to put the HTML (here, content).

On the 7th line, we have a link to the page /goto/<id>, it's in this view we'll count hits.

Count hits

This is the final view we have to write. When a user clicks a link, he goes to /goto/<id> where <id> is the ObjectId of the MongoDB object.

Let's try (there is no template ;) ):

@get("/goto/<id>")
def goto(id):
    """ Increments hits counter and redirect to the link """

    # connect db and collection
    db = connect_db()

    # update the document (ObjectId function is provided by BSON)
    # $inc is a mongoDB helper to increment a fields without fetching the
    # previous value
    db.update({"_id": ObjectId(id)}, {"$inc": {"hits": 1}})

    # Then fetch the object URL
    url = db.find_one({"_id": ObjectId(id)})['url']

    # Close the connection
    db.database.connection.disconnect()

    # and redirect user to his real destination ;)
    redirect(url)

Note the $inc construction. It's one the most interesting MongoDB's feature, and it's explained here

Stylesheet

The last thing I didn't dive yet is the LessCSS stylesheet : you'll find it in the github repo (in static/).

Conclusion

Once again, this app is not perfect. And it wasn't the goal. I wanted just to work with MongoDB to understand better its mechanisms. The experiment is successful : mongo is really interesting to use and could be extremely helpful in some situations.

The entire code of this app is released under the WTFPL and all feedback is welcome.

I hope you've learned things ;)