DevOps shouldn't be hard: deployment reports

7 January 2020

Previously, we managed to make our app redeploy on each commit. As you can remember, we get the status of deployment by processing script execution results. So let’s use it to send a notification to ourselves each time we deploy our code.

For that matter, we will create another Node.js server app on our VPS. You can extend the app that we created for continuous deployment, although I would not recommend that. Instead, we can do that Unix way, where each app does its job and does it well. Additionally, we can use the reporting server to notify us about deployments of other parts of our app, e.g. frontend.

As you can guess, we would need to implement both the client and server sides of the app. First, we will need a client (our CD server) that will send request upon successful (or failed) deployment. Second, we will make a server that will listen to those requests and send it further to the message provider of choice.

Speaking of what service to use to send those messages, that is 100% up to. Personally, I use Telegram bot to deliver messages back to me, so I will use that as an example, but this design allows using any method to deliver messages, like SMS, email, Slack, or other.

Client

As with the CD server, we will need a way to check source authenticity. This time, we will use JWT, or JSON Web Tokens to sign our messages. Also, this time we will implement both signing and verification.

Let’s start by writing two helper functions that will deal with JWT generation.

function _getHash(params) {
    const paramString = JSON.stringify(params);
    const hash = crypto
        .createHash('sha256')
        .update(paramString)
        .digest('hex');
    return hash;
}

function _sign(hash, secret) {
    const payload = {
        iss: 'server',
        sha256: hash,
    };
    const token = jwt.sign(payload, secret);
    return token;
}

Here, _getHash creates a SHA256 hash of a message body, and _sign, well, signs it using a secret. Let’s use it in our client.

const axios = require('axios');
const crypto = require('crypto');
const jwt = require('jsonwebtoken');

const client = axios.create({
    baseURL: 'https://our.reporting.server.url',
});

async function sendSuccess(app) {
    const params = {
        success: true,
        app,
    }
    const secret = process.env.SECRET;
    const hash = _getHash(params);
    const token = _sign(hash, secret);
    await client.post('/cd/server', params, {
        headers: {
            'X-Signature': token,
        },
    });
}

Here, we get our secret from .env file, use it to sign a request body, and then send it to our reporting server.

Few things to note:

the URL at which the reporting server is located, replace our.reporting.server.url with yours.
endpoint which we send request to; I use /cd/server as I have other sources like Netlify to receive updates from, but you can use anything, including /.
X-Signature header: again, it can be almost anything, but I would suggest sticking to something similar as this is kinda standard.

And that’s our client. Let’s look at the server now.

Server

Again, we start with a helper function.

function checkSignature(data, signature, secret, issuer) {
    if (signature == undefined) {
        return false;
    }
    let decoded;
    try {
        decoded = jwt.verify(signature, secret);
    } catch(e) {
        return false;
    }
    const dataString = JSON.stringify(data);
    const hash = crypto
        .createHash('sha256')
        .update(dataString)
        .digest('hex');
    const hashMatches = decoded.sha256 == hash;
    const issuerMatches = decoded.iss == issuer;
    if (!hashMatches || !issuerMatches) {
        return false;
    }
    return true;
}

Similar to the one in the article on the CD server, this checkSignature function validates that the signature is authentic.

Here’s the rest of the server code.

const crypto = require('crypto');
const jwt = require('jsonwebtoken');

app.post('/cd/server', async (req, res) {
    const data = req.body;
    const signature = req.header('X-Signature');
    
    const secret = process.env.SERVER_SECRET;
    const issuer = 'server';
    if (!checkSignature(data, signature, secret, issuer)) {
        res.status(403).end();
    }
    const success = data.success;
    const app = data.app;
    const error = data.error;
    bot.cd('Server', app, success);
    res.send('Hello server!');
});

What we do here is check the signature and send a message. A message is sent via the provider of your choice. Here, it’s Telegram bot (bot.cd('Server', app, success);).

Bonus: Netlify

As another example, let’s try sending a message each time our frontend updated on Netlify.

Now, Netlify obviously doesn’t need to hit our CD server, as it does CD itself. Instead, the Netlify webhook will go straight into our reporting server.

Thankfully, here we can reuse most of the code that we wrote before (Netlify uses JWT to sign webhook requests).

app.post('/cd/netlify', async (req, res) {
    const data = req.body;
    const signature = req.header('X-Webhook-Signature');
    const secret = process.env.NETLIFY_SECRET;
    const issuer = 'netlify';
    if (!checkSignature(data, signature, secret, issuer)) {
        res.status(403).end();
    }
    const success = data.state == 'ready';
    const app = data.name;
    bot.cd('Netlify', app, success);
    res.send('Hello Netlify!');
});

Here, we extract the signature from a header, match it with our locally stored key, and send a message if the signature is valid.

Note: NETLIFY_SECRET and SERVER_SECRET don’t have to be different, but I highly recommend making them so. Otherwise, if one key is leaked (say, by a hacker attack on Netlify), another one will be compromised as well, making your stack less secure.

To add webhooks on Netlify, open a project, then click Settings -> Build & Deploy -> Deploy notifications, then press Add notification -> Outgoing webhook. You can add webhooks for successful or failed builds among other events.

Bonus 2: Error handling

Okay, I know you’re tired by now, but there’s another exciting thing I want to share with you: error logging. In other words, it will allow you to be notified each time you have an error in your app.

In essence, it’s very similar to sending a request from the CD server, only this time we will send the error.

In your Node.js app, add custom error handler:

function errorWatcher(err, req, res, next) {
    if (process.env.ENV == 'dev') {
        console.log(err);
    }
    if (process.env.ENV == 'prod') {
        _sendRuntimeFailure(err.toString());
    }
    next(err);
}

async function _sendRuntimeFailure(error) {
    const app = 'my-app';
    const params = {
        app,
        error,
    };
    const hash = _getHash(params);
    const secret = process.env.SECRET;
    const token = _sign(hash, secret);
    await client.post('/runtime', params, {
        headers: {
            'X-Signature': token,
        },
    });
}

Functions _getHash and _sign are the same as we used above. We also use .env to set the ENV variable to dev or prod. That way, only production errors will be sent to you.

The only thing left is to tell express about our handler.

app.use(errorWatcher);

We will also need to wrap our async routes to make sure the error is passed to our handler.

app.get('/endpoint', wrapAsync(router.endpoint));

// Helper function to pass error down the middleware chain
function wrapAsync(fn) {
    return function(req, res, next) {
        fn(req, res, next).catch(next);
    };
}

That’s it. On the reporting server side, it’s 100% identical to what we used for CD server and Netlify: get signature, verify it, and send a message is signature is valid.

Wrapping up

Here, we created another microserver, this time for reporting. The server collects events from multiple sources and routes them into a single place, e.g. Telegram. We managed to send events based on our CD server, Netlify, and express.js app’s error handler.