Testing Microsoft Azure Custom Vision image recognition using Met Museum subject keywords

This is a test of using the Azure Custom Vision API with a machine learning model that has been trained with tens of thousands of Met Museum images, each of which have been associated with up to five subject keyword tags from a custom art vocabulary determined by the Met (roughly 1,000 unique terms).

The API can take any JPEG URL and provide its predictions in JSON format.

# Query MS Azure CustomVision API for tag predictions, which has been trained 
# with Met subject keywords and tens of thousands of Met images

import json
import requests
import pandas as pd
import matplotlib.pyplot as plot
from IPython.display import Image

# Load Azure Custom Vision API endpoint URL and API key
with open('azurecredentials.json') as f:
    data = json.load(f)
    PREDICTION_KEY = data['predictionkey']
    AZURE_CV_URL   = data['azure_cv_url']
# Provide image URL - string like 'https://foo.org/bar.jpg'

image = '''
http://artimages.clevelandart.org/cma/ump.di?e=D488596F1E639702788EEF15E932E31860AF3A810EC83E293E9806BBB5A1AD19&s=21&se=91419361&v=7&f=1932.313_w.jpg
'''

Send a POST request to the Azure Custom Vision API

The Azure Custom Vision API takes a HTTP POST request and returns a JSON object with the probabilities of what matched, using a prediction model. There are roughly 1,000 art vocabulary terms determined by the Met used as the tags.

# Set POST request header to JSON and our secret key
headers = {'Content-Type' : 'application/json', 'Prediction-Key' : PREDICTION_KEY }
# Set image URL as payload - string like {"url": "https://foo.org/bar.jpg"}
payload = '{"url":"' + image.strip() + '"}'
# Send POST request to our Azure Custom Vision API
r = requests.post(AZURE_CV_URL, headers=headers, data=payload).json()
r
{'error': {'code': '401',
  'message': 'Access denied due to invalid subscription key or wrong API endpoint. Make sure to provide a valid key for an active subscription and use a correct regional API endpoint for your resource.'}}

Use a pandas Dataframe to ingest and plot prediction results

# Turn JSON into a dataframe
df = pd.DataFrame(r['predictions'],columns=['tagName','probability'])

# Show image and horizontal plot anything with a probability greater than a given threshold
df[df['probability']>0.45].plot(kind='barh',x='tagName',y='probability',legend=False)
Image(url=image,width=500)
---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
<ipython-input-6-85eaec2896d5> in <module>
      1 # Turn JSON into a dataframe
----> 2 df = pd.DataFrame(r['predictions'],columns=['tagName','probability'])
      3 
      4 # Show image and horizontal plot anything with a probability greater than a given threshold
      5 df[df['probability']>0.45].plot(kind='barh',x='tagName',y='probability',legend=False)

KeyError: 'predictions'

We are currently working on Met subject keyword to Wikidata mappings, so that these predictions can be used to generate recommended "depicts" statements for Wikidata items.