Tensorflow Applications

Modelling Time Series
Modellling Natural Language

This is the note I took when taking the following two courses,

Modelling Time Series

Introduction

Times seires is defined as

an ordered sequence of values that are usually equally spaced over time

We can use ML for the follwoing applications

Predicting the future (forecasting)
Retracing the past (imputation & Interpolate)
Detection of the anomaly (spikes in time series)
Detection of patterns

Typical Patterns

Stationary

Time series are ideally from a stationary stochastic process, with following patterns.

Trend (linear).
Sensonality (periodicity).
White noise (not learnable).
Auto-correlation (often shown as deterministic decay)

These four basic types are often combined for real data.

Non-stationary

In real life, the data may not be a stationary random process, so everything can change!

These dataset are called none stationary time series.

We chop a small and stable section of the entire time series, to make some predictions.

(more data may not be better!)

We just can not predict drastic changes!

Prepare Data

tf.data.Dataset

Tensorflow offered a nice Dataset class as the container for data.

Here are some examples.

import tensorflow as tf

dataset = tf.data.Dataset.range(10)
for val in dataset:
    print(val.numpy(), end=", ")
print("\n")

""" result
0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 
"""

We can use the window method to generate nested (2D) dataset from 1D arrays.

Notice, the results can not be converted to numpy directly.

import tensorflow as tf

dataset = tf.data.Dataset.range(10)
dataset = dataset.window(5, shift=1)
for data in dataset:
    # data.numpy() -> _VariantDataset' object has no attribute 'numpy'
    for val in data:
        print(val.numpy(), end=", ")
    print()

""" result
0, 1, 2, 3, 4, 
1, 2, 3, 4, 5, 
2, 3, 4, 5, 6, 
3, 4, 5, 6, 7, 
4, 5, 6, 7, 8, 
5, 6, 7, 8, 9, 
6, 7, 8, 9, 
7, 8, 9, 
8, 9, 
9, 
"""

We can use drop_remainder to remove the result, whose length is smaller than widnow size.

import tensorflow as tf

dataset = tf.data.Dataset.range(10)
dataset = dataset.window(5, shift=1, drop_remainder=True)
for data in dataset:
    for val in data:
        print(val.numpy(), end=", ")
    print()

"""result
0, 1, 2, 3, 4, 
1, 2, 3, 4, 5, 
2, 3, 4, 5, 6, 
3, 4, 5, 6, 7, 
4, 5, 6, 7, 8, 
5, 6, 7, 8, 9, 
"""

We can also transofrm a block of array into numpy directly, using the flat_map method.

import tensorflow as tf

dataset = tf.data.Dataset.range(10)
dataset = dataset.window(5, shift=1, drop_remainder=True)
dataset = dataset.flat_map(lambda x : x.batch(5))
for data in dataset:
    print(data.numpy())
    
"""result
[0 1 2 3 4]
[1 2 3 4 5]
[2 3 4 5 6]
[3 4 5 6 7]
[4 5 6 7 8]
[5 6 7 8 9]
"""

Training Data

For the sake of predicting time series, we split the data 1, 2, 3, 4, 5 into

The features 1, 2, 3, 4 (the prediction will base on these numbers)
The label 5, (the prediction will predict this value)

This can be done with the map method

import tensorflow as tf

dataset = tf.data.Dataset.range(10)
dataset = dataset.window(5, shift=1, drop_remainder=True)
dataset = dataset.flat_map(lambda x : x.batch(5))
dataset = dataset.map(lambda x: (x[:-1], x[-1:]))
for x, y in dataset:
    print(x.numpy(), y.numpy())
    
"""result
[0 1 2 3] [4]
[1 2 3 4] [5]
[2 3 4 5] [6]
[3 4 5 6] [7]
[4 5 6 7] [8]
[5 6 7 8] [9]
"""

We can also shuffle the data, to avoid the sequence bias, which is defined as

Sequence bias is when the order of things can impact the selection of things.

import tensorflow as tf

dataset = tf.data.Dataset.range(10)
dataset = dataset.window(5, shift=1, drop_remainder=True)
dataset = dataset.flat_map(lambda x : x.batch(5))
dataset = dataset.map(lambda x: (x[:-1], x[-1:]))
dataset = dataset.shuffle(buffer_size=10)
for x, y in dataset:
    print(x.numpy(), y.numpy())
    
"""result
[0 1 2 3] [4]
[5 6 7 8] [9]
[2 3 4 5] [6]
[1 2 3 4] [5]
[4 5 6 7] [8]
[3 4 5 6] [7]
"""

Finally, we separate the dataset into different batches, for the training of the network.

dataset = tf.data.Dataset.range(10)
dataset = dataset.window(5, shift=1, drop_remainder=True)
dataset = dataset.flat_map(lambda x : x.batch(5))
dataset = dataset.map(lambda x: (x[:-1], x[-1:]))
dataset = dataset.shuffle(buffer_size=10)
dataset = dataset.batch(2).prefetch(1)
for x, y in dataset:
    print(x.numpy().shape, y.numpy().shape)
    
"""result
(2, 4) (2, 1) -> x = [[1, 2, 3, 4], [3, 4, 5, 6]], y = [[5,], [6,]]
(2, 4) (2, 1)
(2, 4) (2, 1)
"""

To generate dataset from an existing time series, we can use the following function, which includes all previous methods.

def windowed_dataset(series, window_size, batch_size, shuffle_buffer):
    """
    Args:
        series (np.ndarray): 1D array
        window_size (int): the length + 1 of the features
        batch_size (int): the number of instances in a batch for training
        suffle_buffer (int): the buffer for randomnise dastaset
    """
    dataset = tf.data.Dataset.from_tensor_slices(series)
    dataset = dataset.window(window_size + 1, shift=1, drop_remainder=True)
    dataset = dataset.flat_map(lambda window: window.batch(window_size + 1))
    dataset = dataset.shuffle(shuffle_buffer).map(
        lambda window: (window[:-1], window[-1])
    )
    dataset = dataset.batch(batch_size).prefetch(1)
    return dataset

Train Models

Baseline Models

There are ways to predict time series with out ML, called the statistical forcasting.

Naive forcasting: $\hat{y}_{i+1} = y_i$ (the result in that course is just wrong.)
Moving average: $\hat{y}i = \frac1S\sum{j=0}^{j=S} \hat{y}_{i-j} $
Differencing: try to predict the difference of the time series, rather than the time series. (Differencing removes the trend and the seasonality)

Linear Regression

We can use linear regression to fit the data, by using a NN with one layer. Mathematically, the calculation is written as

\[y_i = \left( \begin{matrix} w_1 \\ w_2 \\ \vdots \\ w_n \end{matrix} \right) \left( \begin{matrix} y_{i+(0-n)} & y_{i+(1-n)} & \dots & y_{i+(n-1-n)} \end{matrix} \right) + b.\]

The code for such model is,

window_size = 20
batch_size = 32
shuffle_buffer_size = 1000

l0 = tf.keras.layers.Dense(1, input_shape=[window_size])
model = tf.keras.models.Sequential([l0])

MLP

We can use a deeper network to fit the data, with minimum modification. This yields slightly better result.

window_size = 20
batch_size = 32
shuffle_buffer_size = 1000

model = tf.keras.models.Sequential([
  tf.keras.layers.Dense(10, activation="relu", input_shape=[window_size]),
  tf.keras.layers.Dense(10, activation="relu"),
  tf.keras.layers.Dense(1)
])

Simple RNN

Here is the structure of a simple RNN.

                     output (forecasts)
               shape: (batch_size, time_steps, )
                            ▲         
                            │         
                  ┌──────────────────┐
                  │   Dense Layer    │
                  └──────────────────┘
                            ▲         
                            │         
                  ┌──────────────────┐
                  │ Recurrent Layer  │
                  └──────────────────┘
                            ▲         
                            │         
                  ┌──────────────────┐
                  │ Recurrent Layer  │
                  └──────────────────┘
                            ▲         
                            │         
                         input x
      shape (batch_size, time_steps, dimensions)

This is a recurrent layer.

       y(0)              y(1)              y(2)    ...            y(t)
        ▲                 ▲                 ▲                      ▲            
        │                 │                 │                      │            
    ┌──────┐          ┌──────┐          ┌──────┐               ┌──────┐         
    │ Mem  │          │      │          │      │               │      │         
0──▶│ Cell │───H(0)──▶│      │───H(1)──▶│      │─ ... ─H(t-1)─▶│      │──H(t)──▶  
    └──────┘          └──────┘          └──────┘               └──────┘         
        ▲                 ▲                 ▲                      ▲            
        │                 │                 │                      │            
       x(0)              x(1)              x(2)                   x(t)   



──────────────────────────────────Time Steps───────────────────────────────▶

The shapes of different variables are,

x(i): (batch_size, dim), for univariate series, dim = 1
y(i): (batch_size, units_number), the number of output neurons.
H(i): this is the state output, for simple RNN, H(i) = y(i)

The code for a simple RNN model is

model = tf.keras.models.Sequential([
    tf.keras.layers.SimpleRNN(
        units=20,
        return_sequences=True,  # return the { y(i) } sequence
        input_shape=[None, 1]  # None -> input takes arbitrary length
    ),
    tf.keras.layers.SimpleRNN(
        units=20, return_sequences=False,  # y(t)
    ),
    tf.keras.layers.Dense(1),  # forecasting one number
])

The intermediate RNN layers should always have the parameter return_sequence = True.

We can also change the data in the mode, with a Lamda layer.

model = tf.keras.models.Sequential([
    tf.keras.layers.SimpleRNN(
        units=20, return_sequences=True, input_shape=[None, 1]
    ),
    tf.keras.layers.SimpleRNN(units=20, return_sequences=False),
    tf.keras.layers.Dense(1),
    tf.keras.layers.Lambda(
        lambda x : x * 100  # fit the scale of dataset
    )
])

LSTM

We can use an LSTM laryer rather than a SimpleRNN to construct the model.

There is another cell state in LSTM, which carries the information to more time steps.

model = tf.keras.models.Sequential([
    tf.keras.layers.Lambda(
        lambda x: tf.expand_dims(x, axis=-1),
        input_shape=[None]
    ),
    tf.keras.layers.Bidirectional(
        tf.keras.layers.LSTM(32, return_sequences=True)
    ),
    tf.keras.layers.Bidirectional(
        tf.keras.layers.LSTM(32)
    ),
    tf.keras.layers.Dense(1),
    tf.keras.layers.Lambda(lambda x: x * 100.0)
])

We can also add Conv1D layer on top of the LSTM.

model = tf.keras.models.Sequential([
    tf.keras.layers.Conv1D(
        filters=32, kernel_size=3, strides=1,
        padding="causal", activation="relu",
        input_shape=[None, 1]
    ),
    tf.keras.layers.LSTM(32, return_sequences=True),
    tf.keras.layers.LSTM(32, return_sequences=True),
    tf.keras.layers.Dense(1),
    tf.keras.layers.Lambda(lambda x: x * 200)
])

Evaluate Prediction

There are following matrices

\[\begin{aligned} \mathrm{mse} &= \frac1N \sum_i (\hat{y}_i - y_i)^2 \\ \mathrm{rmse} &= \sqrt{\mathrm{mse}} \\ \mathrm{mae} &= \frac1N \sum_i \vert \hat{y}_i - y_i \vert \\ \mathrm{mape} &= \frac1N \sum_i \vert \frac{\hat{y}_i - y_i}{y_i} \vert & \text{p = percentage} \end{aligned}\]

$\hat{y}_i$ is the predicted value at time $i$
$y_i$ is the true value at time point $i$

Modellling Natural Language

Introduction

Natural language processing (NLP) allows us to extract information from languages.

For example, let’s say we want to predict if a movie comment is positive.

Our dataset would contains,

$n$ Comments, and
Each comment will contain $m$ words, and
Each comment will be either positive or negative.

Tokenise

To analyse the natural languages, we need to convert text to numbers. We treat every comment in the same way. And for each comment, we will

Give each word a number/code. This is called encoding.
We decide what to do if we see a new word, known as outer vocabulary (oov).
We will pad the comment so that all of them have equal length, ie word counts.

This is relevant code in tensorflow using the class tokenisor and function pad_sequences.

from tensorflow.keras.preprocessing.text import Tokenizer
from tensorflow.keras.preprocessing.sequence import pad_sequence

vocab_size = 10000  # only give the top 10,000 most frequent words a number/code
max_length = 120    # maximum length (word count) for each comment
trunc_type='post'   # removing excess words at the end of the comments
oov_tok = "<OOV>"   # use <OOV> to represnt words never seen before

comments = ...  # a list of n comments (strings), each comment has different length

# generate the encoding rule that convert words to numbers
tokenizer = Tokenizer(num_words = vocab_size, oov_token=oov_tok)
tokenizer.fit_on_texts(comments)

# convert words to numbers
sequences = tokenizer.texts_to_sequences(comments)

# making the sequences so that they have equal length
padded = pad_sequences(sequences, maxlen=max_length, truncating=trunc_type)

# obtain the map between words and numbers, the dict is {words: number}
word_index = tokenizer.word_index

Learning the Embedding

The tokenised sequences does not make much scence. Word cat may have the code of 00001 and word cats might have 95843 but these two words should somehow be similar.

To get meaningful representations of words, we need to represnt the words as vectores. And the words with similar meanings should be closer in their corresponding space, where the vectors be in. The conversion from code to vector is carried out by embedding.

We get the embedding by training a predictive network. Here is the relevent code.

import tensorflow as tf
from tensorflow.keras.preprocessing.text import Tokenizer
from tensorflow.keras.preprocessing.sequence import pad_sequences

vocab_size = 10000  # only give the top 10,000 most frequent words a number/code
embedding_dim = 16  # each word is represented by 16 numbers
max_length = 120    # maximum length (word count) for each comment
trunc_type='post'   # removing excess words at the end of the comments
oov_tok = "<OOV>"   # use <OOV> to represnt words never seen before

comments = ["comment", "hello world"]  # n comments (strings) with different lengths

# generate the encoding rule that convert words to numbers
tokenizer = Tokenizer(num_words = vocab_size, oov_token=oov_tok)
tokenizer.fit_on_texts(comments)

# convert words to numbers
sequences = tokenizer.texts_to_sequences(comments)

# making the sequences so that they have equal length
padded = pad_sequences(sequences, maxlen=max_length, truncating=trunc_type)

# obtain the map between words and numbers, the dict is {words: number}
word_index = tokenizer.word_index

# construct a model with embedding
model = tf.keras.Sequential([
    tf.keras.layers.Embedding(vocab_size, embedding_dim, input_length=max_length),
    tf.keras.layers.Flatten(),
    tf.keras.layers.Dense(6, activation='relu'),
    tf.keras.layers.Dense(1, activation='sigmoid')
])

model.compile(loss='binary_crossentropy',optimizer='adam',metrics=['accuracy'])
model.summary()

This is the summary of the model.

Model: "sequential"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
embedding (Embedding)        (None, 120, 16)           160000     
# meaning: 120 vectors, each vector is 16 dimensional
_________________________________________________________________
flatten (Flatten)            (None, 1920)              0     
_________________________________________________________________
dense (Dense)                (None, 6)                 11526 
_________________________________________________________________
dense_1 (Dense)              (None, 1)                 7
=================================================================
Total params: 171,533
Trainable params: 171,533
Non-trainable params: 0
_________________________________________________________________

The final “product” of the embeding layer is the weight. In this exampe, the shape of the weight can be obtained via,

vocab_size = 10000  # only give the top 10,000 most frequent words a number/code
embedding_dim = 16  # each word is represented by 16 numbers
.......
embed_layer = model.layers[0]
weights = embed_layer.weights()[0]  # shape (10000, 16), 

The weights will map every word from their code to a vector.

Model the Sequence

We need consider the sequence of the language into consideration.

LSTM

A very popular model called long short term memory (LSTM) can be used to construct an RNN. The structure of the model is different but the idea is similar.

    .───────────.                .───────────.                 .───────────.           
   (    y(0)     )              (    y(1)     )               (    y(2)     )          
    `───────────'                `───────────'                 `───────────'           
          ▲                            ▲                             ▲                 
  ━━━━━━━━╋━━━━━━━Cell State━━━━━━━━━━━╋━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╋━━━━━━━━━━━━▶    
          │                            │                             │                 
┌───────────────────┐        ┌───────────────────┐         ┌───────────────────┐       
│     Function      │───────▶│     Function      │────────▶│     Function      │──────▶
└───────────────────┘        └───────────────────┘         └───────────────────┘       
          ▲                            ▲                             ▲                 
          │                            │                             │                 
          │                            │                             │                 
    .───────────.                .───────────.                 .───────────.           
   (    x(0)     )              (    x(1)     )               (    x(2)     )          
    `───────────'                `───────────'                 `───────────'

LSTM can be think of as an “update” to RNN. There is an additional pipeline called “cell state” being passed from earilier states. The cell state can also be bi-direcitonal, where the future can affect the past.

The following code use the LSTM in the model in tensorflow.

import tensorflow as tf
from tensorflow.keras.preprocessing.text import Tokenizer
from tensorflow.keras.preprocessing.sequence import pad_sequences

vocab_size = 10000  # only give the top 10,000 most frequent words a number/code
embedding_dim = 16  # each word is represented by 16 numbers
max_length = 120    # maximum length (word count) for each comment
trunc_type='post'   # removing excess words at the end of the comments
oov_tok = "<OOV>"   # use <OOV> to represnt words never seen before

comments = ["comment", "hello world"]
tokenizer = Tokenizer(num_words = vocab_size, oov_token=oov_tok)
tokenizer.fit_on_texts(comments)
sequences = tokenizer.texts_to_sequences(comments)
padded = pad_sequences(sequences, maxlen=max_length, truncating=trunc_type)
word_index = tokenizer.word_index

# construct a model with sequence model
model = tf.keras.Sequential([
    tf.keras.layers.Embedding(vocab_size, embedding_dim, input_length=max_length),
    tf.keras.layers.Bidirectional(tf.keras.layers.LSTM(64)),  # use LSTM
    tf.keras.layers.Dense(64, activation='relu'),
    tf.keras.layers.Dense(1, activation='sigmoid')
])

1D CNN

Except for the LSTM, we can also use a 1D convolution layer, like below

# construct a model convolution rather than LSTM
model = tf.keras.Sequential([
    tf.keras.layers.Embedding(vocab_size, embedding_dim, input_length=max_length),
      tf.keras.layers.Conv1D(128, 5, activation='relu'),
    tf.keras.layers.GlobalMaxPooling1D(),
    tf.keras.layers.Dense(64, activation='relu'),
    tf.keras.layers.Dense(1, activation='sigmoid')
])

Generate New Text

We can train a network to generate new text. The idea is to predict the next word based on existing texts.

The following code present an example for this task.

tokenizer = Tokenizer()

data="""
In the town of Athy one Jeremy Lanigan
Battered away til he hadnt a pound.
His father died and made him a man again
Left him a farm and ten acres of ground.
He gave a grand party for friends and relations
Who didnt forget him when come to the wall,
And if youll but listen Ill make your eyes glisten
Of the rows and the ructions of Lanigans Ball.
Myself to be sure got free invitation,
For all the nice girls and boys I might ask,
...
"""

corpus = data.lower().split("\n")

tokenizer.fit_on_texts(corpus)
total_words = len(tokenizer.word_index) + 1

# print(tokenizer.word_index) -> 263
# print(total_words) -> {'and': 1, 'the': 2, 'a': 3, ... }

input_sequences = []
for line in corpus:
    token_list = tokenizer.texts_to_sequences([line])[0]
    for i in range(1, len(token_list)):
        n_gram_sequence = token_list[:i + 1]
        input_sequences.append(n_gram_sequence)

# pad sequences 
max_sequence_len = max([len(x) for x in input_sequences])
input_sequences = np.array(
  pad_sequences(input_sequences, maxlen=max_sequence_len, padding='pre')
)

# create predictors and label
xs, labels = input_sequences[:, :-1], input_sequences[:, -1]

# from numbers to categorical label
ys = tf.keras.utils.to_categorical(labels, num_classes=total_words)

"""
xs[6] -> [ 0  0  0  4  2 66  8 67 68 69] (history)
labels[6] ->  70  (prediction)

categorical label
ys[6].shape -> (263,)
ys[6][70] -> 1
ys[6][65] -> 0
"""

# Train the network to make prediction
model = Sequential()
model.add(Embedding(total_words, 64, input_length=max_sequence_len-1))
model.add(Bidirectional(LSTM(20)))
model.add(Dense(total_words, activation='softmax'))

model.compile(
  loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy']
)
history = model.fit(xs, ys, epochs=500, verbose=1)


# Use the network to generate text, starting with she
seed_text = "She"
next_words = 10

for _ in range(next_words):
    token_list = tokenizer.texts_to_sequences([seed_text])[0]
    token_list = pad_sequences([token_list], maxlen=max_sequence_len-1, padding='pre')
    predicted = model.predict(token_list, verbose=0)
    predicted = np.argmax(predicted, axis=1)
    output_word = ""
    for word, index in tokenizer.word_index.items():
        if index == predicted:
            output_word = word
            break
    seed_text += " " + output_word
print(seed_text)

# -> She stepped out and i stepped in again again again again