Checking Met date consistency

This set of scripts checks date consistency

For objects, we check the Object Date, Object Begin Date, Object End Date columns.

  • Object Date - the canonical date or estimate from the curators.
  • Object Begin Date - for search purposes, the earliest date to be considered
  • Object End Date - for search purposes, the latest date to be considered

Import Met CSV database into a dataframe

This is more than 400,000 rows, so it may take 10-20 seconds or more

%%time
import csv
from urllib.request import urlopen
import codecs
# from tqdm import tqdm
# from tqdm._tqdm_notebook import tqdm_notebook
import requests
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from dateutil.parser import parse

# The Met's weekly CSV dump URL in Github is quite big at 250 Mbytes
# Comment out for now, since this is slow to load. 
# url = 'https://media.githubusercontent.com/media/metmuseum/openaccess/master/MetObjects.csv'

# Use local copy of CSV file for speed and read the CSV file into a pandas dataframe
url = 'metmuseum/MetObjects-20190425.csv'
met_df = pd.read_csv(url,low_memory=False)
CPU times: user 8.6 s, sys: 836 ms, total: 9.43 s
Wall time: 9.46 s

Examine the structure of rows

Take a look at some of the rows. NaN means "not a number" or a blank from the CSV file.

met_df.sample(3)
Object Number Is Highlight Is Public Domain Object ID Department Object Name Title Culture Period Dynasty ... Locale Locus Excavation River Classification Rights and Reproduction Link Resource Metadata Date Repository Tags
198871 1999.47.230 False False 318150 Arts of Africa, Oceania, and the Americas Tunic Tunic Aymara NaN NaN ... NaN NaN NaN NaN Textiles-Woven NaN http://www.metmuseum.org/art/collection/search... 4/22/2019 8:00:03 AM Metropolitan Museum of Art, New York, NY Tunics
125362 08.180.925 False False 215274 European Sculpture and Decorative Arts Handkerchief Handkerchief French, Valenciennes NaN NaN ... NaN NaN NaN NaN Textiles-Embroidered NaN http://www.metmuseum.org/art/collection/search... 4/22/2019 8:00:03 AM Metropolitan Museum of Art, New York, NY Clothing and Accessories|Flowers
424394 1972.586.21(25) False False 695129 Drawings and Prints Photograph Still for Will Bradley's Motion Picture "Moong... NaN NaN NaN ... NaN NaN NaN NaN Photographs NaN http://www.metmuseum.org/art/collection/search... 4/22/2019 8:00:03 AM Metropolitan Museum of Art, New York, NY NaN

3 rows × 44 columns

Basic date examination

met_df[['Object ID','Object Name','Title','Department','Object Date','Object Begin Date','Object End Date']].sample(10)
Object ID Object Name Title Department Object Date Object Begin Date Object End Date
308180 438510 Painting The Outskirts of a Village European Paintings ca. 1880 1875 1885
437241 713409 Kylix fragment Terracotta fragment of a kylix (drinking cup) Greek and Roman Art 490–470 B.C. -530 -450
337343 484591 Drawing Large Drawing # 33 Modern and Contemporary Art 1966 1966 1966
359503 552548 Pot sling Pot sling Egyptian Art ca. 945–525 B.C. -945 -945
67106 79270 Evening dress Evening dress Costume Institute probably mid to late 1970s 1973 1979
48453 55831 Print NaN Asian Art 1895 1895 1895
150411 249541 Flask Glass flask Greek and Roman Art 4th–5th century A.D. 300 499
291116 420170 Print Rock Blackfish, from Fish from American Waters... Drawings and Prints 1889 1889 1889
131717 223088 Panel Parrot and squirrel European Sculpture and Decorative Arts mid-18th century 1730 1765
203315 324579 Standard top Top of standard with male figure and flanking ... Ancient Near Eastern Art ca. 1500–1100 B.C. -1500 -1100

Simple, single year date

met_df[met_df['Object Date'].str.match('^(ca.[ ]*)*[0-9]+$',na=False)]['Object Number'].count()
172580
met_df[met_df['Object Date'].str.match('^(ca.[ ]*)*[0-9]+$',na=False)][['Object ID','Object Name','Title','Department','Object Date','Object Begin Date','Object End Date']].sample(10)
                                                             
Object ID Object Name Title Department Object Date Object Begin Date Object End Date
278679 407591 Print West Virginia, from Flags of the States and Te... Drawings and Prints 1888 1888 1888
479011 777646 Baseball card Norm Siebern, First Base-Outfield, Kansas City... Drawings and Prints 1961 1961 1961
436930 712803 Print Shooting, from the Pretty Athletes series (N19... Drawings and Prints 1889 1889 1889
386166 635372 Baseball card, print Miller, Boston, National League, from the Mecc... Drawings and Prints 1911 1911 1911
220156 347570 Book Chushin meimei ga den Drawings and Prints 1848 1848 1848
94050 168598 Hat Hat Costume Institute 1942 1942 1942
98536 180665 Collar Collar Costume Institute ca. 1890 1885 1895
285955 414987 Photograph Card 485, Miss Behr, from the Actors and Actre... Drawings and Prints ca. 1888 1885 1891
91061 156807 Fan Fan Costume Institute ca. 1710 1708 1712
219700 347106 Book Estasi e Rapimento sopra la Luna Drawings and Prints 1763 1763 1763
met_df[met_df['Object Date'].str.match('^(ca.[ ]*)*[0-9]+-[0-9]+$',na=False)][['Object ID','Object Name','Title','Object Date','Object Begin Date','Object End Date']].sample(10)
                                                             
Object ID Object Name Title Object Date Object Begin Date Object End Date
447284 728078 Valentine Valentine 1859-1869 1859 1869
427814 699864 Print Frieze with Ships in a Roadstead ca. 1618-22 1613 1623
187157 296303 Photograph [Unidentified Woman Seated with a Female Spirit] 1862-75 1862 1875
240748 368755 Print The Nativity ca. 1544-48 1539 1549
427823 699873 Print Landscape with a Steep Cliff and a Stream, Sma... ca. 1622-25 1617 1627
427774 699824 Print Landscape with a Steep Cliff and a Stream, Sma... ca. 1622-25 1617 1627
442359 720317 Baseball card, print George Cutshaw, 2nd Base, Pittsburgh, National... 1921-23 1921 1923
440703 718021 Baseball card, print Waite Hoyt, Pitcher, New York Americans, from ... 1921-22 1921 1922
450598 733889 Book\r\nPrint\r\nOrnament & Architecture\r\n Design for a frame after the Galerie de Franço... 1542-45 1542 1545
400918 655274 Drawing A Young Woman Standing with her Hands on her H... ca. 1770-73 1765 1775

Vague terms

met_df[met_df['Object Date'].str.match('([Pp]robably|[Pp]ossibly)',na=False)][['Object ID','Object Name','Title','Object Date','Object Begin Date','Object End Date']].sample(20)
Object ID Object Name Title Object Date Object Begin Date Object End Date
411525 671542 Charger Charger with portrait possibly late 17th century 1685 1699
28770 33662 Shaffron Shaffron in 16th Century German Style probably 18th–19th century 1775 1925
306133 435756 Painting A Classical Landscape probably 1660s 1660 1669
112227 200375 Teapot Teapot probably 1752 1752 1752
308151 438374 Painting A Beach with Fishing Boats probably 1653 1653 1653
313284 447735 Fragment Fragment probably 19th century 1800 1899
18747 22968 Pair of mail sleeves Pair of Mail Sleeves probably 14th to 15th century 1301 1499
321743 459406 Drawing Two Male Nudes probably 18th century 1698 1802
316984 451460 Fragment Fragment probably 18th century 1700 1799
119164 208078 Flagon Flagon probably 18th century 1700 1799
19364 23586 Arrowhead (Yanonē) Arrowhead (Yanonē) probably 16th century 1501 1600
101016 188313 Spoon Figure-top spoon probably mid-18th century 1730 1765
200069 320674 Head Memorial head Possibly 17th century 17 17
455312 741252 Fragment Fragment probably 8th–12th century 800 1300
455032 740972 Fragment Fragment probably 8th–12th century 800 1300
306845 436582 Painting Head of a Young Woman possibly 1780s 1780 1789
455226 741166 Fragment Fragment probably 8th–12th century 800 1300
116518 205099 Cistern Cistern probably 19th century 1800 1899
456280 742220 Fragment Fragment probably 8th–12th century 800 1300
455440 741380 Fragment Fragment probably 8th–12th century 800 1300