Customer Segmentation Part 3: Network Visualization

Written

This post is the third and final part in the customer segmentation analysis. The first post focused on K-Means Clustering to segment customers into distinct groups based on purchasing habits. The second post takes a different approach, using Pricipal Component Analysis (PCA) to visualize customer groups. The third and final post performs Network Visualization (Graph Drawing) using the igraph and networkD3 libraries as a method to visualize the customer connections and relationship strengths.

3 Part Series

• CUSTOMER SEGMENTATION PART 1: K-MEANS CLUSTERING
• CUSTOMER SEGMENTATION PART 2: PCA FOR SEGMENT VISUALIZATION
• CUSTOMER SEGMENTATION PART 3: NETWORK VISUALIZATION

Customer Segmentation Cheat Sheet

We have a cheat sheet that walks the user through the workflow for segmenting customers. You can download it here.

Download the Segmentation and Clustering Cheat Sheet

10-Minute Overview of Segmentation and Clustering taught in our Business Analysis with R (DS4B 101-R) Course

Table of Contents

• Why Network Visualization?
• Where We Left Off
• Getting Ready for Network Visualization
  • Getting the Data
  • Reading the Data
  • Manipulating the Data
• Developing the Network Visualization
  • Step 1: Create a Cosine Similarity Matrix
  • Step 2: Prune the Tree
  • Step 3: Create the igraph
• Visualizing the Results with networkD3
• Conclusions
• Recap
• Further Reading

Why Network Visualization?

According to Katya Ognyanova, an expert in network theory, Network Visualization is a useful technique for viewing relationships in the data, with the key benefits being identification of the following:

Source: Network Visualization with R

For customer segmentation, we can utilize network visualization to understand both the network communities and the strength of the relationships. Before we jump into network visualization, it’s a good idea to review where we left off in the previous customer segmentation posts.

Where We Left Off

In the first post, we used k-means clustering to analyze the bikes data set, a collection of excel files that contains data for bike shops (customers), bikes (products), and sales orders for the bike manufacturer, Cannondale. The bike shops and sales orders are fictional / simulated (see the orderSimulatoR post for more on this), but the bikes (products) are actual models from Cannondale’s website.

A hypothesis was formed that bike shops purchase bikes based on bike features such as unit price (high end vs affordable), primary category (Mountain vs Road), frame (aluminum vs carbon), etc. The sales orders were combined with the customer and product information and grouped to form a matrix of sales by model and customer. The kmeans() function was run on a range of potential clusters, k, and the silhouette() function from the cluster package was used to determine the optimal number of clusters.

In the second post, we used PCA to visually examine the customer segments using the prcomp() function in base R. PCA visualization allowed us to identify a customer segment that k-means failed to pick up.

Getting Ready for Network Visualization

For those that would like to follow along, rather than run through the previous posts the sections below can be used to get everything ready for network visualization.

Getting the Data

You can access the data here if you would like to follow along. You’ll need to download the following files:

• orders.xlsx: Contains the fictional sales orders for Cannondale. customer.id in the orders.xlsx file relates to bike shop.id in the bikeshops.xlsx file, and product.id in the orders.xlsx file relates to bike.id in the bikes.xlsx file.

• bikes.xlsx: Contains information on products (e.g. bike model, primary category, secondary category, unit price, etc). bike.id is the primary key.

• bikeshops.xlsx: Contains information on customers (e.g. customer name and location). bikeshop.id is the primary key.

The script to load and configure the data into a customer trends matrix is shown below.

Reading the Data

This script will read the data. Make sure you have the excel files in a folder named “data” in your current working directory prior to running the script below.

# Read Cannondale orders data --------------------------------------------------
library(xlsx)   # Used to read bikes data set
customers <- read.xlsx2("./data/bikeshops.xlsx", sheetIndex = 1,
                       colClasses = c("numeric", "character", "character",
                                      "character", "character", "numeric"))
products <- read.xlsx2("./data/bikes.xlsx", sheetIndex = 1,
                        colClasses = c("numeric", "character", "character",
                                       "character", "character", "numeric"))
orders <- read.xlsx2("./data/orders.xlsx", sheetIndex = 1, colIndex = 2:7,
                     colClasses = c("numeric", "numeric", "Date", "numeric",
                                    "numeric", "numeric"))

Manipulating the Data

The script below combines the orders, customers and products data frames into orders.extended, which is a data frame that simulates output we would get from an SQL query of a sales orders database / ERP system. The data is then manipulated to form customerTrends, which has the data structured such that the rows contain products and the columns contain purchase quantity (as percentage of total) by customer.

# Step 1: Combine orders, customers, and products data frames ------------------
library(dplyr)
orders.extended <- merge(orders, customers, by.x = "customer.id", by.y="bikeshop.id")
orders.extended <- merge(orders.extended, products, by.x = "product.id", by.y = "bike.id")
orders.extended <- orders.extended %>%
  mutate(price.extended = price * quantity) %>%
  select(order.date, order.id, order.line, bikeshop.name, model,
         quantity, price, price.extended, category1, category2, frame) %>%
  arrange(order.id, order.line)

# Step 2: Group by model & model features, summarize by quantity purchased -----
library(tidyr)  # For spread function
customerTrends <- orders.extended %>%
  group_by(bikeshop.name, model, category1, category2, frame, price) %>%
  summarise(total.qty = sum(quantity)) %>%
  spread(bikeshop.name, total.qty)
customerTrends[is.na(customerTrends)] <- 0  # Remove NA's

# Step 3: Convert price to binary high/low category ----------------------------
library(Hmisc)  # Needed for cut2 function
customerTrends$price <- cut2(customerTrends$price, g=2)

# Step 4: Convert customer purchase quantity to percentage of total quantity ---
customerTrends.mat <- as.matrix(customerTrends[,-(1:5)])  # Drop first five columns
customerTrends.mat <- prop.table(customerTrends.mat, margin = 2)  # column-wise pct
customerTrends <- bind_cols(customerTrends[,(1:5)], as.data.frame(customerTrends.mat))

Note that if the code above seems confusing, I recommend stepping through each step in the code chunk separately, then stopping to view the data frame that is created. the intention is to build the orders data frame and then convert it into a data frame that relates customers to the product purchase history.

Here’s the first 6 rows of customerTrends:

customerTrends %>% head() %>% knitr::kable() # First 6 rows

Developing the Network Visualization

Going from the customerTrends to a network visualization requires three main steps:

1. Create a Cosine Similarity Matrix
2. Prune the Tree
3. Create the iGraph

Step 1: Create a Cosine Similarity Matrix

The first step to network visualization is to get the data organized into a cosine similarity matrix. A similarity matrix is a way of numerically representing the similarity between multiple variables similar to a correlation matrix. We’ll use Cosine Similarity to measure the relationship, which measures how similar the direction of a vector is to another vector. If that seems complicated, just think of a customer cosine similarity as a number that reflects how closely the direction of buying habits are related. Numbers will range from zero to one with numbers closer to one indicating very similar buying habits and numbers closer to zero indicating dissimilar buying habits.

Implementing the cosine similarity matrix is quite easy since we already have our customerTrends data frame organized relating customers (columns) to product purchases (rows). We’ll use the cosine() function from the lsa library, and this will calculate all of the cosine similarities for the entire matrix of customerTrends.mat. Make sure to use the matrix version of customer trends (customerTrends.mat) because the data frame version needs to be vectorized to work with the cosine() function. Also, set the diagonal of the matrix to zero since showing a customer perfectly related to itself is of no significance (and it will mess up the network visualization).

# Create adjacency matrix using cosine similarity ------------------------------
library(lsa) # for cosine similarity matrix
simMatrix <- cosine(customerTrends.mat)
diag(simMatrix) <- 0 # Remove relationship with self
simMatrix %>% head() %>% knitr::kable() # Show first 6 rows

Step 2: Prune the Tree

It’s a good idea to prune the tree before we move to graphing. The network graphs can become quite messy if we do not limit the number of edges. We do this by reviewing the cosine similarity matrix and selecting an edgeLimit, a number below which the cosine similarities will be replaced with zero. This keeps the highest ranking relationships while reducing the noise. We select 0.70 as the limit, but typically this is a trial and error process. If the limit is too high, the network graph will not show enough detail. Try testing different edge limits to see what looks best in Step 3.

# Prune edges of the tree
edgeLimit <- .70
simMatrix[(simMatrix < edgeLimit)] <- 0

Step 3: Create the iGraph

Creating the igraph is easy with the graph_from_adjacency_matrix() function. Just pass the pruned simMatrix. We can also get the communities by passing the simMatrix to the cluster_edge_betweenness() function.

library(igraph)
simIgraph <- graph_from_adjacency_matrix(simMatrix, 
                                         mode = 'undirected', 
                                         weighted = T)
ceb <- cluster_edge_betweenness(simIgraph) # For community detection

Let’s pause to take a look as the cluster edge betweenness (ceb). We can view a plot of the clusters by passing ceb to the dendPlot() function. Set the mode to “hclust” to view as a hierarchical clustered dendrogram (it uses hclust from the stats library). We can see that the cluster edge betweenness has detected three distinct clusters.

dendPlot(ceb, mode="hclust")

Now we can plot the network graph by passing ceb and simIgraph to the plot() function. See plot.igraph for additional documentation.

plot(x=ceb, y=simIgraph)

Yikes. This is a little difficult to read. With that said, the graph did a nice job of showing the major customer segments. Also notice how the color of the node clusters match the dendrogram groupings. For readability sake, we’ll take a look at what the networkD3 package has to offer.

Visualizing the Results with NetworkD3

The networkD3 library enables D3 JavaScript network graphs from R. It has an igraph_to_networkD3() function that can be used to seamlessly convert an igraph object to network3D. The benefits to D3 are interactivity:

• The names are hidden until the user hovers over the node, improving readability.
• Nodes can be dragged to manipulate the network graph for easier viewing.
• Zoom is enabled, allowing the user to zero-in on specific nodes to more easily see the edges and thus better understand relationships.

# Use igraph ceb to find membership
members <- membership(ceb)

# Convert to object suitable for networkD3
simIgraph_d3 <- igraph_to_networkD3(simIgraph, group = members)

# Create force directed network plot
forceNetwork(Links = simIgraph_d3$links, Nodes = simIgraph_d3$nodes, 
             Source = 'source', Target = 'target', 
             NodeID = 'name', Group = 'group', 
             fontSize = 16, fontFamily = 'Arial', linkDistance = 100,
             zoom = TRUE)

Conclusion

Network visualization is an excellent way to view and understand interrelationships between customers. Network graphs are particularly useful as the data scales, as they can be pruned and viewed interactively to zero-in on complex relationships. We’ve all heard the saying a picture is worth a thousand words. A network graph could be worth many more as customers go from 30 in our example to hundreds, thousands or even millions.

Recap

This post expanded on our customer segmentation methodology by adding network graphing to our tool set. We manipulated our sales order data to obtain a format that relates products to customer purchases. We created a cosine similarity matrix using the cosine() function from the lsa library, and we used this as the foundation for our network graph. We “pruned the tree” to remove edges with low significance. We used the igraph library to create an igraph with communities shown. Finally, we converted the igraph to an interactive graph using the networkD3 package. Pretty impressive, and hopefully you see that it’s not that difficult with a basic knowledge of R programming.

Further Reading

1. Network Visualization with R: This article is an excellent place to start for those that want to understand the details behind Network Visualization in R.

2. CRAN Task View: Cluster Analysis & Finite Mixture Models: While we did not specifically focus on clustering in this post, the CRAN task view covers a wide range of libraries and tools available for clustering that can be used in addition to those covered in the three part series.

3. Data Smart: Using Data Science to Transform Information into Insight: Chapter 5 uses a similar approach on customers that purchase wine deals based on different buying preferences. This is a great book for those that are more familiar with Excel than R programming.

Business Science University

Enjoy data science for business? We do too. This is why we created Business Science University where we teach you how to do Data Science For Busines (#DS4B) just like us!

Our first DS4B course (HR 201) is now available!

Who is this course for?

Anyone that is interested in applying data science in a business context (we call this DS4B). All you need is basic R, dplyr, and ggplot2 experience. If you understood this article, you are qualified.

What do you get it out of it?

You learn everything you need to know about how to apply data science in a business context:

• Using ROI-driven data science taught from consulting experience!

• Solve high-impact problems (e.g. $15M Employee Attrition Problem)

• Use advanced, bleeding-edge machine learning algorithms (e.g. H2O, LIME)

• Apply systematic data science frameworks (e.g. Business Science Problem Framework)

“If you’ve been looking for a program like this, I’m happy to say it’s finally here! This is what I needed when I first began data science years ago. It’s why I created Business Science University.”

Matt Dancho, Founder of Business Science