DevOpsOpenShift

A/B Deployments Made Easy with OpenShift

Who doesn’t get excited about A/B deployments?  I know I do.  With OpenShift, this power becomes accessible to mere mortals.  Moreover, you can try it right now from your own laptop. Let Cavaliers, Charley and Billy show you how. This lab is another in the OpenShift MiniLabs series.

charleybilly

Objective

A/B Hypothesis

A/B deployments are about some business defined hypothesis. In this scenario, we have an application that shows an ordered list of items. The user can rate any item. There are two versions of the application, one that shows a list of “cats” and the other “cities” – hence the name COTD, or “Cat/City of the Day”. So the questions are 1) which of these application versions is more engaging and 2) what should be the ordering of the items. We are going to resolve both questions respectively by tracking 1) the aggregated ratings recorded against each application to determine the favored version and 2) ranking the items by totaling the ratings within each application. The necessary data to do this is mined from the log files of each application. So let’s get started.

screen-shot-2016-10-07-at-3-50-57-pm

screen-shot-2016-10-07-at-4-16-33-pm

Setup

Initial Attempt

This tutorial assumes you have completed the OpenShift MiniLabs installation procedure. Then refresh before continuing.

Repeat Attempt

To reset your environment to repeat this tutorial do the following:

$ cd ~/containersascode
$ ./oc-cluster-wrapper/oc-cluster up containersascode
$ oc login -u developer -p developer
$ oc delete project cotd

Instructions

Create the OpenShift project

$ oc login -u developer -p developer
$ oc new-project cotd --display-name='Cat of the Day' --description='A/B Deployment Example'
$ oc get projects

Create the A application

Wait a minute or two for the curl command to begin working.

$ oc login -u developer -p developer
$ oc project cotd
$ oc new-app --name='cotd1' -l name='cotd' php~https://bitbucket.org/emergile/cotd.git -e SELECTOR=cats
$ oc expose service cotd1 --name=cotd1 -l name='cotd'
$ oc status 
$ curl http://cotd1-cotd.127.0.0.1.nip.io/item.php

Create the B application

Again, wait a minute or two for the curl command to begin working.

$ oc login -u developer -p developer
$ oc project cotd
$ oc new-app --name='cotd2' -l name='cotd' php~https://bitbucket.org/emergile/cotd.git -e SELECTOR=cities
$ oc expose service cotd2 --name=cotd2 -l name='cotd'
$ oc status 
$ curl http://cotd2-cotd.127.0.0.1.nip.io/item.php

Create the AB route

Note that the example below uses a 50/50 split, but you can change the route-backends percentages to anything you want. Indeed, the A/B deployment strategy also supports A, B, C, … and more.

$ oc login -u developer -p developer
$ oc project cotd
$ oc expose service cotd1 --name='ab' -l name='cotd'  
$ oc set route-backends ab cotd1=50 cotd2=50
$ oc annotate route/ab haproxy.router.openshift.io/balance=roundrobin 

Verify Lab Success

Use curl and observe the changing contents.

$ while true; do curl -s http://ab-cotd.127.0.0.1.nip.io/item.php | grep "data/images" | awk '{print $5}'; sleep 2; done

The output should appear similar to:

 data/images/cats/auckland.jpg
 data/images/cities/adelaide.jpg
 data/images/cats/canberra.jpg
 data/images/cities/sydney.jpg
...

Alternatively, launch two different browsers and explore the applications for some more fun. If using the same browser, remove the cookies created to prevent stickiness.

Hypothesis Testing

Now that both applications are running, we can inspect and parse the necessary data from each log file to test our hypothesis. You can automate this process using OpenShift’s log aggregation feature, but for now let’s just illustrate this manually. Do the following for each application “cotd1” and “cotd2”. Use “get pods” to find the container name.

$ oc login -u developer -p developer
$ oc project cotd
$ oc get pods
$ oc tail -f $COTD | grep COTD

The relevant log entries can be harvested by searching for the string COTD over some defined time window. An example of the JSON like extract is shown below. Whenever a user rates an item a corresponding entry appears for the item:rating. This data can be then compiled to change the rankings of the items and the weights of the routing.

 COTD { "user" : "1ro5unuagno293g3hakibqahb4", items" : [ {"adelaide" : "3"}, {"sydney" : "4"}, {"canberra" : "2"}, {"auckland" : "5"}, ] , "client_ip" : "172.17.0.1",  "sydney_time" : "2016:10:08 09:41:00",  } COTD

When you have finished, stop the OpenShift cluster.

$ ./oc-cluster-wrapper/oc-cluster down

Note that when you restart it using the up command above, the system state will be preserved. How good is that!

Trivia

There are various techniques for managing environment variables with OpenShift, including the -e SELECTOR=cats example used above. Other strategies include injecting them into the deploymentconfig (dc) and/or using configmaps. The dc method would be, e.g.

$ oc project cotd
$ oc env dc/cotd SELECTOR=cities

Enjoy.

Leave a Reply