Static Mapping: TNIC Data#

This notebook provides more details on using evomap for deriving static market maps, especially for large input matrices.


  1. Loading the Data

  2. Mapping the Data

  3. Exploring the Map

  4. Evaluating the Map


Parts of this example are based on

[1] Matthe, M., Ringel, D. M., Skiera, B. (2022), Mapping Market Structure Evolution. Forthcoming in Marketing Science.

[2] Hoberg, G & Phillips, G. (2016), "Text-Based Network Industries and Endogenous Product Differentiation.", Journal of Political Economy 124 (5), 1423-1465.

Last updated: September 2022

Read the full EvoMap paper here (open access):

Contact: For questions or feedback, please get in touch.

Loading the Data#

First, load all required imports for this demonstration and set the seed to ensure reproducibility.

import pandas as pd
import numpy as np
import os

For this demonstration, we use a larger sample of the TNIC data also used in [1]. The original data is provided at

For more background on TNIC data, see [2]. If you intend to use these data, make sure to cite these authors’ original work!

In the TNIC data, each row corresponds to a single firm-firm pair at a specific point in time. Thus, each firm variable appears twice in each row (once for each firm). We provide these data merged with additional firm information as part of this package.

from evomap.datasets import load_tnic_snapshot
tnic_snapshot = load_tnic_snapshot()
dict_keys(['matrix', 'label', 'cluster', 'size'])

The data consist of four parts:

  • a symmetric similarity matrix, containing the pairwise relationships among all firms

  • a label array, containing the name for each firm

  • cluster assignments, based on pre-clustering of the similarity matrix (obtained via a Community Detection algorithm)

  • a size array, containing each firm’s market capitalization (*)

(*) Note, that we cannot provide the original market capitalization for licencing reasons, and therefore report a synthetic derivative which is correlated to the firms’ original market capitalization.

# Load each element of the input data
sim_mat = tnic_snapshot['matrix']
labels = tnic_snapshot['label']
clusters = tnic_snapshot['cluster']
market_caps = tnic_snapshot['size']
display("Total number of firms: {0}, such as ..".format(len(labels)))
for label in labels[:5]:
    display(" .. " + label)
'Total number of firms: 1092, such as ..'
' .. AIR T INC'

For each (firm, firm) pair these data include a measure of (non-negative) pairwise similarity, ranging between 0 and 1:

display("Smallest similarity: {0}".format(np.min(sim_mat)))
display("Highest similarity: {0}".format(np.max(sim_mat)))
'Smallest similarity: 0.0'
'Highest similarity: 0.8652'

Mapping the Data#

To project these relationships onto the 2D plane, evomap implements a range of popular (static) mapping methods.

Here, we will be using and comparing

  • Classic Multidimensional Scaling (CMDS),

  • Sammong Mapping, and

  • t-SNE

and see how they perform on mapping the TNIC data.

Running these methods follows the typical scikit-learn conventions (fit / transform / fit_transform), for instance

output = CMDS().fit_transform(D)

where D is supposed to be a data matrix (here: a matrix of pairwise distances).

As a result, evomap is fully compatible with other methods implemented in the scikit-learn library. For instance, you could easily integrate Isomap into the workflow below, and use evomap for preprocessing, plotting, or evaluation. Alternatively, you could use the scikit-learn implementation of t-SNE, which implements a few additional features (such as the Barnes-Hut approximation). For details, see the sklearn.manifold documentation. Further static mapping methods relying on gradient-based optimization routines can easily be integrated into evomap.

Irrespectively of which method you choose, you might need to transform your data according to the method’s specific requirements. Here, we transform the TNIC data (which represent pariwise similarities) into pairwise distances. A straight-forward way to do so is ‘mirroring’ them. Note that the choice of such transformations will have an impact on the resultant maps.

from evomap.preprocessing import sim2dist
D = sim2dist(sim_mat, transformation= 'inverse') 

For a start, we use Sammon Mapping (a non-linear variant of Multidimensional Scaling) to project this matrix onto the two-dimensional plane.

from evomap.mapping import Sammon
method = Sammon(
    input_type = 'distance', n_iter= 500, verbose = 2)
Y_sammon = method.fit_transform(D)
[SAMMON] Initialization 1/1
[SAMMON] Running Gradient Descent with Backtracking via Halving
[SAMMON] Iteration 50 -- Cost: 0.48 -- Gradient Norm: 840.6893
[SAMMON] Iteration 100 -- Cost: 0.47 -- Gradient Norm: 520.6687
[SAMMON] Iteration 150 -- Cost: 0.46 -- Gradient Norm: 266.8555
[SAMMON] Iteration 200 -- Cost: 0.46 -- Gradient Norm: 118.5507
[SAMMON] Iteration 250 -- Cost: 0.46 -- Gradient Norm: 53.7951
[SAMMON] Iteration 300 -- Cost: 0.46 -- Gradient Norm: 24.8079
[SAMMON] Iteration 350 -- Cost: 0.46 -- Gradient Norm: 11.9917
[SAMMON] Iteration 400 -- Cost: 0.46 -- Gradient Norm: 6.1029
[SAMMON] Iteration 450 -- Cost: 0.46 -- Gradient Norm: 3.2740
[SAMMON] Iteration 500 -- Cost: 0.46 -- Gradient Norm: 1.7979

The output Y always consists of an array containing the map coordinates. Y is shaped as (n_samples, n_dims), where ‘n_dims’ usually equals two.

array([[-4909.17814324,  1470.93232368],
       [ 4426.1732782 , -1138.39395038],
       [ -642.53598261,  5421.13215712],
       [  -77.73716351, -1548.73175579],
       [-2260.6731036 ,  -207.976633  ],
       [  -74.57919906, -1523.64017762]])

Exploring the Map#

While one could simply visualize these coordinates in a 2D scatterplot, evomap provides a lot of functionality to create much richer market maps. All required functions to do so are located within the evomap.printer module.

The baseline function is draw_map:

from evomap.printer import draw_map

You can change the map’s aesthetics and bind additional data to it via draw_map’s keyword arguments. For instance, if class labels are available (e.g., obtained via clustering or additional metadata), they can be added as colors. Here, we can use SIC codes for coloring:

draw_map(Y_sammon, c = clusters)

Likewise, one can use additional data to control each point’s size. Here, we are using market capitalization:

draw_map(Y_sammon, c = clusters, size = market_caps)

One can further annotate the map, using labels (e.g., to identify and highlight the positions of specific firms)

    c = clusters, 
    labels = labels, 
    size = market_caps,
    highlight_labels= ['MICROSOFT CORP', 'WALMART INC', 'PFIZER INC'])

Note that one can also use additional keyword arguments to adjust the plot and its labels further

    c = clusters, 
    size = market_caps,
    labels = labels, 
    title_str = 'TNIC Snapshot (via Sammon)',
    fontdict= {'family': 'Arial', 'size': 8},
    highlight_labels= ['MICROSOFT CORP', 'WALMART INC', 'PFIZER INC'],
    fig_size= (7,7))

Evaluating the Map#

Finally, one should always carefully check how good the resultant map represents the input data. To do so, multiple metrics are available from the corresponding module.

A popular choice, especially for larger markets, is the hitrate of nearest-neighbor recovery.

from evomap.metrics import hitrate_score
    X = sim_mat, 
    Y = Y_sammon, 
    n_neighbors = 10, 
    input_type = 'similarity')

Let’s compare the result to two other methods to generate the map.

Classic Multidimensional Scaling (CMDS):

D_norm = D / np.max(D)
from evomap.mapping import MDS
    input_type = 'distance', 
    verbose = 2, 
    n_iter = 500).fit_transform(D_norm)
[MDS] Initialization 1/1
[MDS] Running Gradient Descent with Backtracking via Halving
[MDS] Iteration 50 -- Cost: 0.48 -- Gradient Norm: 0.2657
[MDS] Iteration 100 -- Cost: 0.45 -- Gradient Norm: 0.2905
[MDS] Iteration 150 -- Cost: 0.44 -- Gradient Norm: 0.1755
[MDS] Iteration 200 -- Cost: 0.43 -- Gradient Norm: 0.0893
[MDS] Iteration 250 -- Cost: 0.43 -- Gradient Norm: 0.0518
[MDS] Iteration 300 -- Cost: 0.43 -- Gradient Norm: 0.0402
[MDS] Iteration 350 -- Cost: 0.43 -- Gradient Norm: 0.0339
[MDS] Iteration 400 -- Cost: 0.43 -- Gradient Norm: 0.0317
[MDS] Iteration 450 -- Cost: 0.43 -- Gradient Norm: 0.0295
[MDS] Iteration 500 -- Cost: 0.43 -- Gradient Norm: 0.0350

t-distributed Stochastic Neighbor Embedding (t-SNE):

from evomap.mapping import TSNE
Y_tsne = TSNE(input_type= 'distance', verbose = 2).fit_transform(D)
[TSNE] Calculating P matrix ...
[TSNE] Gradient descent with Momentum: 0.5
[TSNE] Iteration 50 -- Cost: 15.51 -- Gradient Norm: 0.0230
[TSNE] Iteration 100 -- Cost: 15.13 -- Gradient Norm: 0.0129
[TSNE] Iteration 150 -- Cost: 15.02 -- Gradient Norm: 0.0084
[TSNE] Iteration 200 -- Cost: 14.97 -- Gradient Norm: 0.0081
[TSNE] Iteration 250 -- Cost: 14.95 -- Gradient Norm: 0.0077
[TSNE] Gradient descent with Momentum: 0.8
[TSNE] Iteration 287: gradient norm vanished.

Compare how well they perform:

display("MDS Hitrate: {0:.2f}".format(hitrate_score(D, Y_MDS, n_neighbors=10, input_type='distance')))
display("Sammon Hitrate: {0:.2f}".format(hitrate_score(D, Y_sammon, n_neighbors=10, input_type='distance')))
display("t-SNE Hitrate: {0:.2f}".format(hitrate_score(D, Y_tsne, n_neighbors=10, input_type='distance')))
'MDS Hitrate: 0.23'
'Sammon Hitrate: 0.24'
't-SNE Hitrate: 0.50'

These differences in solution quality also become apparent when looking at the map output:

from matplotlib import pyplot as plt
fig, ax = plt.subplots(1,3, figsize = (15, 5))

    title_str= 'MDS Solution', 
    c = clusters,
    labels = labels,
    highlight_labels=['APPLE INC', 'WALMART INC', 'PFIZER INC'], 
    size = market_caps, 
    ax = ax[0])

    title_str= 'Sammon Solution',
    labels = labels, 
    highlight_labels= ['APPLE INC', 'WALMART INC', 'PFIZER INC'],
    c = clusters,  
    size = market_caps, 
    ax = ax[1])

    title_str= 't-SNE Solution',
    labels = labels,
    highlight_labels=['APPLE INC', 'WALMART INC', 'PFIZER INC'],
    c = clusters, 
    size = market_caps, 
    ax = ax[2])