Originally posted on AppsFlyer blog

One of the coolest things about Gitlab is that it’s built with and for the customers, with a special focus on getting things done — faster and more efficiently.

Did you know that your Gitlab instance can perform HTTP POST requests for almost everything that occurs in the system? — think about the endless possibilities!
You can use it for logging, pulling or changing data in LDAP servers or even to trigger something else!

Sound good? so let’s create a simple, yet powerful Python hook receiver.

First things first

In this example, I use Flask — a microframework for Python based on Werkzeug, Jinja 2 and good intentions, but you can use whatever you feel most comfortable with, in order to receive and manipulate the POST requests that will come from your GitLab instance later on.

Let’s start by creating an instance of the Flask class that we imported, and create our route() decorator in order to tell Flask what URL should trigger our function.
Pay attention that by default, a route only answers to GET requests, but we want to manipulate the HTTP POST requests — so we use the methods argument in the route() decorator. Here’s an examples:

#!/usr/bin/python
from flask import Flask
from flask import request
 
app = Flask(__name__)
 
@app.route('/', methods = ['POST'])
def JsonHandler():
    if request.is_json:
        content = request.get_json()
        print "Just got {0} event!".format(content['event_name'])
        return 'OK'
    else:
        return 'OK'
 
app.run(host='0.0.0.0', port= 8888)

I’m starting by simply checking if the request is in JSON format, and prints the event name, otherwise, i’m dropping it.
Now let’s save, run and test it by sending a simple JSON —

curl -H “Content-Type: application/json” -XPOST -d ‘{“event_name”:”test”,”data”:”wow, such push”}’ http://mycoolhook.com:8888

We should get an ‘OK’ response, and see our printing in the server which our receiver is running on —

* Running on http://0.0.0.0:8888/ (Press CTRL+C to quit)
Just got test event!
127.0.0.1 — — [05/Mar/2018 09:27:27] “POST / HTTP/1.1” 200 -

So, now that we know that our tiny receiver is working, let’s dive in and add more capabilities.

As mentioned before, Gitlab can perform HTTP POST requests for almost everything that occurs in the system, and we can manipulate them by catching the event_name which for most is self-explanatory.
We won’t cover the usage of all of them as it can be different from case to case, but we will cover some of the use cases here at AppsFlyer.

At AppsFlyer, we decided to expose an internal in-house API which will reduce the number of API calls to the built-in Gitlab API, expose only relevant data and avoid manipulating Gitlab API paging again and again. We use it in our building & deployment stages and in other internal services.
But how do we keep up the changes in Gitlab? with system hooks of course!

@app.route('/', methods = ['POST'])
def JsonHandler():
    if request.is_json:
        content = request.get_json()
        # Handle project changes
        if content['event_name'] == "project_create":
            pipe = store.pipeline()
            pipe.hmset(content['name'], {"group": content['owner_name']})
            pipe.hmset(content['name'], {"branches": map_branches })
            pipe.execute()
            logger.info('Project {0} Created.'.format(content['name']))
            return 'OK'
        elif content['event_name'] == "push":
            update_consul_last_push(content)
            return 'OK'
        elif content['event_name'] == "project_destroy":
            store.delete(content['name'])
            logger.info('Project {0} destoried.'.format(content['name']))
            return 'OK'
        elif content['event_name'] == "project_rename":
            store.rename(content['old_path_with_namespace'].split('/')[1], content['name'])
            logger.info('Project renamed from {0} to {1}'.format(content['old_path_with_namespace'].split('/')[1], content['name']))
            return 'OK'
        elif content['event_name'] == "project_transfer":
            store.hmset(content['name'], {"group": content['path_with_namespace'].split('/')[0]})
            logger.info('Project {0} transferd'.format(content['name']))
            return 'OK'
    else:
        return "JSON Only"

store our data in Redis hash, and connect to it via Redis-Py.

The first part you are probably familiar with — getting the request, checking if it’s valid JSON and assigning it to content variable.
Then we start to handle the system hooks by the event_name:
If the event_name is project_create — a new project is created!- so lets create a new hash in Redis. For each Gitlab project we store the group it belongs to and the project’s branches. We use Redis pipelining in order to reduce the number of calls to our Redis instance.
If a push event occurs— We log the pusher name, time, date and project name in Consul. You can see it as “offline backup” in the most chaotic situation when all other backup methods fail — we can still know which R&D member has the latest version of the project that pushed to Gitlab on his own laptop.
If project_destroy / project_rename / project_transfer occurs — we update the relevant data in Redis.

You can continue to do so to each event that Gitlab can post in the same way.

Give me my data!
Now we want to create a new endpoint to retrieve data, and in order to do so we created a new Flask route and use hmget to get the data from Redis —

@app.route('/getgroup/<string:project_name>')
def getGroupName(project_name):
    try:
        return store.hmget(project_name, "group")
    except:
        return 'null'

Retrieves the group of the given project

go on and continue to create more routes to retrieve the data — for project, groups and branches.

Deployment

After your listener is done, we advise you to run it on each of your Gitlab servers using Supervisor, and continue by putting it behind AWS ELB with an SSL Certificate to create a highly available and secure listener.

Gitlab Config

Now, when you have a Python hook receiver that is just waiting for your Gitlab system hooks to come in, you need to configure GitLab to send the hooks to the correct place. Enter the Admin Area -> System Hooks and enter the URL of the ELB that you created, and select what kind of extra hooks you want to enable, like Push / Tag Push events, which will be used later on.

Hit the ‘Add System Hook’ button in order to save it.

Debug

You can easily debug your system hook receiver by going to Admin Area -> System Hooks > Edit, On the bottom of the screen there are “Recent Deliveries” .
There, you can see the status codes (e.g 200,500) of the request, the “Elapsed time” that it took for this specific request to be completed and so on.
You can go to “View details” on each one of the requests in order to get additional debugging information— you can check the “Request headers” & “Response headers” of your request, the “Response body” that came back from your receiver and the JSON itself.
Furthermore, if you face an issue, after fixing it, you can use the “Resend Request” button in order to resend your request back to Gitlab and see the results once more, so no hook will be lost.

---

With just a few lines of code, we created a powerful Python hook receiver that can make our life much easier!
Gitlab is an amazing tool, it gives you the ability to integrate with so many other products and you can see how easy it is to take advantage of such a cool feature for your own needs.