Azure

Reducing SharePoint Framework Code Smells: 1 – Setting up SonarQube in Azure

Marky Roden Azure, DevOps, SharePoint Framework, SonarQube , , ,

This is a three part series on how to set up SonarQube as a Quality Gate in your SharePoint Framework development process. The end goal is to add SonarQube to your build and release process through DevOps. These three articles will explain:

  1. How to set up a sample SonarQube server in Azure
  2. How to run a code review manually
  3. How to integrate the code review into your Azure DevOps build and release process.

As part of a quality development process not only should developers be linting their code, running unit test and so forth, another step in the process which can be added is a “Code Quality” check using the open source project SonarQube.

In this article we will see how to create a stand alone sample SonarQube server in Azure (and locally if you really want as well).

Introduction

“SonarQube provides the capability to not only show health of an application but also to highlight issues newly introduced. With a Quality Gate in place, you can fix the leak and therefore improve code quality systematically.” 

In practice what it means is an additional tool which developers can use to write better, more maintainable code. This increases quality and reduces overall maintenace costs when implemented as part of a continuous build and deploy process.

There are plugins for JavaScript and TypeScript and thus makes this very applicable to SharePoint Framework development.

Setting up the server

The first step is to create a SonarQube server upon which your code can be reviewed. Some VERY nice person by the name of vanderby has created an ARM template to “Deploy Sonar Cube to Azure“. It is limited by using an embedded database, but it will at least show you the basics before you are ready to scale this properly.

As the github page states – it does take a while to get started but once it is up you can start to use it.

To log into the server I used admin/admin. As this is a sample setup it doesn’t really matter.

Creating a project

Once you are set up and running you can create a project and a key which can then be used to access the server from a command line interface (CLI).

Under the administration server create a new project and once that is complete generate a key for your project

Using these credentals we can test out code from the command line.

Conclusion

Setting up a sample SonarQube server in Azure is pretty simple. As it states though this will not scale and if you are going to use this in an enterprise it will need some better set up. But for the sake of demonstration, it’s just fine.

In the next article we will look at how to apply this to an Azure DevOps build and deploy process for SPFx.

 

Note

You can just as easily set up your own local SonarQube server by following the 2 minute set up installation instructions

Get Started in 2 minutes

 

Calling an external service from your chat bot

Marky Roden Azure, Bot Framework , ,

In this article I will show to how integrate simple commands (intents) into your bot to then integrate with an external service.

Introduction

In previous articles we have looked at how to create a sample Azure bot and in this article we will be looking into how “intents” work. The microsoft documentation on dialogIntents can be found here (for the node bot).

URL Shortener

In an effort to learn more about running a node service in Azure, Azure SQL and to help brand some of my tweets a little better I created my own little URL shortening service running at https://marky.co. The details are not so important for this article but I took the Coligo.io article on create a node url shortner with Mongo and modified it to work with Azure SQL instead of Mongo. The service is called very simply by POSTing the appropriate parameters at the appropriate service on my website. What is returned is a shortened URL. Simple process really.

IntentDialog

“The IntentDialog class lets you listen for the user to say a specific keyword or phrase. We call this intent detection because you are attempting to determine what the user is intending to do. IntentDialogs are useful for building more open ended bots that support natural language style understanding.”

So in a sense I am looking at this as an opportunity to treat my bot as a CLI client for my own laziness. While the “intent” is to allow for natural language understanding I want to look at it as an opportunity to make my bot into a monkey butler of sorts. Do my work for me, I am lazy (or productive).

I am not a fan of CLI in the programming world. Not being a historically Unix kinda guy, never had to and I prefer a point and click tooling approach to things like git and Azure CLI. That said, this is not programming – this is bots 😉

Setting up your dialogIntent to listen for -sh 

The following code snippets is the basis for my simple bot connector – listen for the “-sh ” command then take the following argument and process it.

The http module is used for the communication with the shortener service. Once the response is returned from the service the shortened URL is sent back to the user. The process is simple and the intent of this is just to show an example of how to use a command in a bot to make it do something.

	"use strict";
	var builder = require("botbuilder");
	var botbuilder_azure = require("botbuilder-azure");
	var azure = require('azure-storage');

	var http = require('http');

	var data = "";

	var options = {
		host: 'marky.co',
		port: '80',
		path: thepath,
		method: 'POST',
		headers: {
			'secret': theSercretKeyWhichStopsBots,
			'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8'
		}
	};

	var dotenv = require('dotenv').config();

	var useEmulator = (process.env.NODE_ENV == 'development');

	var connector = useEmulator ? new builder.ChatConnector({
			appId: process.env['MicrosoftAppId'],
			appPassword: process.env['MicrosoftAppPassword']
		}) : new botbuilder_azure.BotServiceConnector({
		appId: process.env['MicrosoftAppId'],
		appPassword: process.env['MicrosoftAppPassword'],
		stateEndpoint: process.env['BotStateEndpoint'],
		openIdMetadata: process.env['BotOpenIdMetadata']
	});

	var bot = new builder.UniversalBot(connector);

	var intents = new builder.IntentDialog();
	bot.dialog('/', intents);

	intents.matches(/-sh /i, function (session, args) {
		var input = args.matched.input
		
		//input will be in the format "-sh http://www.xomino.com"
		//better validation would probably be appropriate at some point

		//split the input into "-sh" and theURL
		data="url="+input.split(" ")[1]; 
		//match the POST length to the incoming URL
		options.headers['Content-Length'] = data.length 
		session.sendTyping(); //...typing

		var req = http.request(options, function(res) {
			res.setEncoding('utf8');
			res.on('data', function (chunk) {
				session.send('Your short URL is: '+JSON.parse(chunk).shortUrl);
			});
		});

		req.on('error', function(e) {
			console.log('problem with request: ' + e.message);
		});

	// write data to request body
		req.write(data);
		req.end();
	});

	intents.onDefault(function (session, args, next) {
		session.send("I'm sorry "+session.dialogData.name+". I didn't understand.");
	});

	if (useEmulator) {
		var restify = require('restify');
		var server = restify.createServer();
		server.listen(3978, function() {
			console.log('test bot endpont at http://localhost:3978/api/messages');
		});
		server.post('/api/messages', connector.listen());    
	} else {
		module.exports = { default: connector.listen() }
	}

 

Using this code I can start to use my bot as a URL shortener rather than having to go to my website and post the URL into a form.

Note – in case you were curious this does not work in the Skype client itself because a URL is automagically transformed into something else for preview. Ah well that’s not quite the point really 🙂

Conclusion

In this article we have see that using the bot IntentDialog class we are able to listen for predetermined commands which will then allow it to take input, act on that input and return an answer. In this case a URL shortener is a simple application but it would be much more useful if I could securely look into my enterprise and extract information relating to my business……

Setting up the sample Azure bot to work locally with the bot emulator

Marky Roden Azure, Bot Framework , ,

In this article I will demonstrate how to configure your local development environment to work with the environmental variables set up within your Azure environment in the sample bot previously discussed,

Introduction 

In the previous article we looked at how to create a sample azure bot and then how to configure it in VSTS for continuous integration. If you want to develop with this sample locally you will need to set it up to work with the local bot emulator. (What is Bot Builder for Node.js and why should I use it?). To be able to do this you will have to configure your local development environment to use the process.env variables which are picked up within the azure runtime environment.

process.env

process.env is how environmental variables are passed into a node development environment. They are especially important when it comes to keeping secret keys secret. We have to make sure that they are not included in the git repository and are not available to the end user. You can learn more about process.env in the nodejs documentation.

Using dotenv

I like to use the dotenv nodejs package to handle local env variables to just read my variables locally from a .env file. If you look at the package.json for the example project, turns out so does microsoft 😉

...
  "dependencies": {
    "azure-storage": "^1.3.2",
    "botbuilder": "^3.4.2",
    "botbuilder-azure": "0.0.1",
    "dotenv": "^4.0.0"
  },

When I clone the VSTS repo locally and run it – nothing seems to happen…..and that’s because I do not have any local env variables….


var useEmulator = (process.env.NODE_ENV == 'development');

......
if (useEmulator) {
    var restify = require('restify');
    var server = restify.createServer();
    server.listen(3978, function() {
        console.log('test bot endpont at http://localhost:3978/api/messages');
    });
    server.post('/api/messages', connector.listen());    
} else {
    module.exports = { default: connector.listen() }
}

So we can use a .env file locally in the root of the project and then require it in my code. The .env file contains nothing more that NODE_ENV = ‘development’ right now.

NOTE: To make sure this is not pushed up to the repo – add *.env to your .gitignore file.

We now have a web server running (nice typo MS 😉 ).

Bot Emulator

Once you download and install the bot emulator you can configure it as per the instructions to run a simple bot without issue. This does not work once you have a bot running in Azure. The sample I created uses multiple Azure services and you can see them being called as the environmental variables in the code.

 

var connector = useEmulator ? new builder.ChatConnector() : new botbuilder_azure.BotServiceConnector({
    appId: process.env['MicrosoftAppId'],
    appPassword: process.env['MicrosoftAppPassword'],
    stateEndpoint: process.env['BotStateEndpoint'],
    openIdMetadata: process.env['BotOpenIdMetadata']
});

Unfortunately when I load up the emulator – as you can see above “new builder.ChatConnector” does not pass in any environmental variables. I then get the following error in the console

The appId is undefined as it is not being passed into the application.

The ApplicationId and secret key were given to you when you created the bot in the first place – if you didn’t write them down, you’re going to be regretting that decision right now….

We need to modify the ChatConnector request to pass in the necessary variables, and we also need to add those variables to the .env file.

 

var connector = useEmulator ? new builder.ChatConnector({
        appId: process.env['MicrosoftAppId'], //new param
        appPassword: process.env['MicrosoftAppPassword']  //new param
    }) : new botbuilder_azure.BotServiceConnector({
    appId: process.env['MicrosoftAppId'],
    appPassword: process.env['MicrosoftAppPassword'],
    stateEndpoint: process.env['BotStateEndpoint'],
    openIdMetadata: process.env['BotOpenIdMetadata']
});

Refreshing the bot emulator, we start to get somewhere

and then it dies when we try and talk to it (doh)

Azure Storage Credentials

The problem is that this chat example uses the Azure Storage Service (to make the Azure Function part of the example work). So we have to add the AzureWebJobsStorage environmental variable to our .env file.

 

var queueSvc = azure.createQueueService(process.env.AzureWebJobsStorage);

You can find this connection key in your Azure portal > Storage Service > Access Keys

The format for the .env file entry  is as follows (Running Azure Functions Locally with the CLI and VS Code): AzureWebJobsStorage=’DefaultEndpointsProtocol=https;AccountName=storagename;AccountKey=secretKey’

Once that is in place – we have a bot running locally talking to the bot framework – unfortunately we do not get a response back from the Azure Function….

Azure Function debugging…..

The problem quite simply is that the message sent to the azure function contains a serviceURL from which it should respond – and in this case it is ” serviceUrl: ‘http://localhost:63136′,” and of course it has no idea what that is.

For the sake of this blog post that is ok – we are at least up and running with the “bot” part of this emulator working, although somewhat disappointing it can’t be fully developed in this environment.

Conclusion

As part of this process we have seen how to connect the local bot emulator to a service running in Azure and how to incorporate a connection to Azure Functions.

 

 

Adding your bot code to VSTS for source control and configuring continuous integration

Marky Roden Azure, Bot Framework , ,

In this article I will walk through the steps to enable source control and continuous integration for your bot

Introduction

In previous articles we have seen how to create a sample bot and how to link it into Skype. In this article we are going to look at how to use Visual Services Team Studio (VSTS) to manage source control and eventually continuous integration. We are going to set up is an environment such that when you check your code into the repository a code deployment is triggered to the nodejs server (our bot in this case) which then finally restarts. In short: code checks in and everything refreshes automagically.

What is going to happen is that we are taking the code out of the Azure sample development web based IDE environment and move it to something a little more robust.

Selecting your Source Control

From your bot interface select settings and scroll down to the continuous integration section.

Note: it is really important to read the instructions – if you do NOT download the zip file of your bot first you will end up with a blank look on your face and a need to rebuild your bot from scratch – take it from this idiot (facepalm).

bot1

Download your zip file and then head over to your favorite source control repo and create one.

In my case I used Visual Studio Team Services (VSTS) to set up a git repo.

bot2I unzipped the file locally and associated it with my new project in VSTS. I use JetBrains Webstorm as my JavaScript development environment but this would work just as well in Visual Studio or VS Code.

bot3

 

Setting up continuous integration

Back within the Azure bot development website click the Continuous Integration button. This will then cause you to move through a number of screens configuring the CI. For this to work you have to have configured your Azure environment to be linked with your VSTS environment. Here’s a great blog post describing how to do just that (Linking your VSTS Account to Azure)

So “Setup”

bot4Select VSTS

bot5

and then your project within VSTS

bot6

Which branch (master just to keep things simple for now)

bot7

et voila

bot8

Configuring a performance test

Following the prompts you can set up the perf test (for once the CI is complete)

bot9

bot10

 

and we now have your Azure bot being deployed from code sitting within VSTS. Cool 🙂

Example

I have my bot hooked up in Skype

bot12

I make a quick change to the message in WebStorm (my JavaScript IDE)

bot11

Commit and push

bot13

bot14

Head back to Skype and within about 60 seconds (it has to restart after all) we have a new message

bot15

What’s really interesting though is that I started my conversation before the restart was ready – but the message was queued and when the server woke up it finally responded – that was very cool

Conclusion

In this article we have looked at how to set up a VSTS source control repo of our new test bot and then configured continuous integration through the bot framework. I check in my new code and within seconds the new functionality is posted out to the live bot.

 

 

 

Adding your Azure bot framework bot into Skype

Marky Roden Azure, Bot Framework , ,

In this article I will show how simple it is to add your newly created bot into your Skype contacts.

Introduction

In the previous article we looked at how to create your first sample Azure bot which uses Azure Functions in the background. This article will continue from where that left off.

Channels

The bot framework has a number of pre-configured “Channels” which allow your bot to be easily added to other chat mediums.

If you aren’t already there, go into your bot and select Channels. Conveniently there is an “Add to Skype” button right there….(can’t imagine what that does)

bot1

Adding to Skype

You click on the link to Add to Contacts

bot2

you then “Add to contacts” – having to log in during the process (I need to get a better logo for my bot)

bot3

about 30 seconds later my bot appeared in my Skype client as a contact (bonus if you recognize the song lyrics)

bot4

Once it appeared in my contacts, I could talk to it – and while this is pretty simple stupid, this is VERY cool 🙂

bot5

I have to say though, the “response” from the Azure functions seems very slow for what should be a trivial task. I have no idea why or how it is even working right now – I’ll get to that 🙂

Conclusion

Adding your bot to the pre-configured channels is as easy as following the instructions and we now have a bot running in Skype! It’s not quite getting me a beer from the fridge yet, but we’ll work on that 🙂


How to create a sample nodejs bot which utilizes Azure Functions in 15 minutes

Marky Roden Azure, Bot Framework , , ,

In this article I will demonstrate how to create a nodejs bot hosted in Azure. The bot will be created from a sample provided from Microsoft. The example bot will use Azure Functions as the trigger for it’s communication. You will need a Microsoft Azure account to be able to follow this process for yourself. This is only an introduction on how to get up and running, more steps in the process will come later.

Introduction 

Looking at the documentation on how to create a test a nodejs bot within the Microsoft Bot Framework can be found in the Microsoft documentation (What is Bot Builder for Node.js and why should I use it?).

I found another more interesting (less CLI) way to do it through Azure portal. As I work my way through figuring out how all this works I expect there will be a number of blog posts as there is a lot to talk about. This post is focused on getting us up and running.

As of February 2017 this Bot Service is in Preview and the process may change before it goes GA later in the year !

Creating a bot service in Azure

Within your Azure portal create a new instance of a bot service from within Intelligence and Analytics.

bot1

Give it a name and resource group

bot2

and create.

Once created you will be able to access your bot from your portal resources pane.

Setting up your service

Opening your new service for the first time, you’ll be presented with the form to create an AppId and secret password for your bot. Follow the prompts and get back to the screen to create a sample bot.

bot3

We are going to select NodeJS and create an azure function bot for the sake of demo

bot4

and then we wait……Ho hum, I wonder what’s going on in the background – probably magic….

bot5

Developer interface

Once the bot has been created you will be presented with a web based IDE showing you the node code and a chat window on the right.


 bot6

You can talk to yourself!! Something I like to do all the time…..apparently….

bot9

Back in Azure Portal

In setting up all this up, Azure auto-magically created a number of new services have been created in the Azure Portal to manage this functionality(which is why we had to wait for the background magic earlier). We can now see the following services making all this happen.

bot8

How it works and how it is happening will come in future blog posts. For the moment we are up and running……and this blog post is getting long enough already.

Conclusion

This entire process took no more than about 15 minutes to complete. There is a lot more to explore around the deployment of the bot and the further development thereof, but for the moment we are up and running with a bot and Azure Functions (at least in development).

PS

I now have a nagging feeling that this is cool and all, but I really should check and understand how much those new magic services are going to cost…….

Setting up a secure, custom domain, node.js site on Azure

Marky Roden Azure, Letsencrypt, node.js , , , ,

In this article I will demonstrate the necessary steps to set up a node.js server running https, hosted in Azure.

Introduction

This article is a combination of my own work and a conglomeration of reference point blog articles which I had to find to achieve all of this.

Creating a node.js site on Azure

If you follow the instructions on this site you should be able to create an azure site (Get started with Node.js web apps in Azure App Service)

a1

Creating a custom domain

Once you have registered your new domain (in my case marky.co) you need to go to the azure portal and follow the instructions posted here (Configuring a custom domain name for an Azure cloud service). You cannot do this on your free tier though and this where you have to chose your plan carefully. To be able to interact with Office Add-Ins I need my service to be SSL enabled.

a2 a3

Once you have selected a Basic plan you should get the following options

a4

Assign your site and as the instructions stated – you can “Bring your domain” by changing the CName within your domain name provider DNS management tools.

a5

Adding SSL

There are a number of methods for getting an SSL certificate but I have taken to doing it for free – you can use the same process I detailed here for exposing your node server to manually collect the letsencrypt certificates  (Using Let’s Encrypt to create an SSL certificate for my Bluemix hosted web site) to create the .pem files.

To turn the .pem files into .pfx files you need to follow the openssl instructions here (How To: Get LetsEncrypt working with IIS manually)

openssl pkcs12 -export -out “certificate.pfx” -inkey “privkey.pem” -in “cert.pem” -certfile chain.pem

a6

The certificate.pfx file can then be loaded into the azure portal. When you import the certificate successfully it is displayed on the main blade automatically.

a7

 

Add the SSL binding

a71

aaaah we love the cloud….

a8

a9

IMPORTANT – Restart your instance and there we go

a10

Conclusion

In this article we have seen how to deploy an instance of node.js on Azure, applied a custom domain to it, created an SSL certificate and added to an azure instance. Once this is complete you should have an SSL secured node.js  instance running which can then be used for Office Add-in deployments.