Qfyilyi

I start playing with keras and vgg16 recently, and I am using keras.applications.vgg16.

But here I come with a question about what is model.inputs because I saw others using it in https://github.com/keras-team/keras/blob/master/examples/conv_filter_visualization.py although it does not initialize it

    ...

    input_img = model.input

    ...

    layer_output = layer_dict[layer_name].output

    if K.image_data_format() == 'channels_first':

        loss = K.mean(layer_output[:, filter_index, :, :])

    else:

        loss = K.mean(layer_output[:, :, :, filter_index])



    # we compute the gradient of the input picture wrt this loss

    grads = K.gradients(loss, input_img)[0]

I checked the keras site but it only said that is an input tensor with shape (1,224,224,3) But I still don't understand what is that exactly. Is that an image from ImageNet?Or a default image provided by keras for keras model?

I am sorry if I don't have enough understanding of deep learning, but can someone explain it to me please. Thanks

The 4 dimensions of (1,224,224,3) are the batch_size, image_width, image_height and image_channels respectively. (1,224,224,3) means that the VGG16 model accepts a batch size of 1 (one image at a time) of shape 224x224 and three channels (RGB).

For more information on what a batch and therefore a batch size is, you can check this Cross Validated question.

Returning to VGG16, the input of the architecture is (1, 224, 224, 3). What does this mean? That in order to input a image into the network, you will need to:

1. Preprocess it to reach a shape of (224, 224) and 3 channels (RGB)


2. Convert this to an actual matrix of shape (224, 224, 3)


3. Group together various images in a batch of the size that requires the network (in this case, the batch size is 1, but you need to add a dimension to the matrix, in order to obtain the (1, 224, 224, 3)

After doing this, you can input the image to the model.

Keras offers few utilitary functions to do these tasks. Below I present a modified version of the code snippet shown in Extract features with VGG16 from Usage examples for image classification models in the documentation.

In order to have it actually working, you need a jpg of any size named elephant.jpg. You can obtain it running this bash command:

wget https://upload.wikimedia.org/wikipedia/commons/f/f9/Zoorashia_elephant.jpg -O elephant.jpg   

I will split the code in the image preprocesing and the model prediction for clarity:

Load the image

import numpy as np

from keras.preprocessing import image

from keras.applications.vgg16 import preprocess_input



img_path = 'elephant.jpg'

img = image.load_img(img_path, target_size=(224, 224))

x = image.img_to_array(img)

x = np.expand_dims(x, axis=0)

x = preprocess_input(x)

You can add prints along the way to see what's going on, but here is a brief summary:

1. 
   image.load_img() load a PIL image, already in RGB and already reshaping it to (224, 224)


2. 
   image.img_to_array() is translating this image into a matrix of shape (224, 224, 3). If you access, x[0, 0, 0] you will get the red component of the first pixels as a number between 0 and 255


3. 
   np.expand_dims(x, axis=0) is adding the first dimension. x after is has shape (1, 224, 224, 3)
   


4. 
   preprocess_input is doing an extra preprocessing required for imagenet-trained architectures. From its docstring (run help(preprocess_input)) you can see that it:
   
   

   
   

   will convert the images from RGB to BGR, then will zero-center each color channel with respect to the ImageNet dataset, without scaling

   
   

   
   

This seems to be the standard input for ImageNet training set.

That's it for the preprocessing, now you can just input the image in the pretrained model and get a prediction

Predict

y_hat = base_model.predict(x)

print(y_hat.shape) # res.shape (1, 1000)

y_hat contains the probabilities for each of the 1000 imagenet classes the model assigned to this image.

In order to obtain the class names and a readable output, keras provided an utility function too:

from keras.applications.vgg16 import decode_predictions

decode_predictions(y_hat)

Outputs, for the Zoorashia_elephant.jpg image I downloaded before:

[[('n02504013', 'Indian_elephant', 0.48041093),

  ('n02504458', 'African_elephant', 0.47474155),

  ('n01871265', 'tusker', 0.03912963),

  ('n02437312', 'Arabian_camel', 0.0038948185),

  ('n01704323', 'triceratops', 0.00062475674)]]

Which seems pretty good!