Analysing the Results of the Abandonment Controlled Experiment

Notebook Metadata

Author Cristina Sarasua
Last Update 01.08.2018

Purpose Analyse the results of the controlled experiment run for the abandonment project, where the same bacth of HITS has been deployed several times in CrowdFlower(?), controlling for two variables:

  1. the reward (B1 reward \$0.10 per HIT B2 reward \$ 0.30 per HIT)
  2. the length (B1 3 docs \$ 0.15 per HIT B2 6 docs \$ 0.30 per HIT) 5 cents per document

Code

import pandas as pd
import matplotlib.pyplot as plt
import datetime
import numpy as np
%matplotlib inline
# Data Directories
dataDir = ""
taskDir = dataDir + "tasks/"
logDir = dataDir + "logs/"


#dataDir = "/Users/sarasua/Documents/RESEARCH/collab_Abandonment/controlledexperiment/results/"
#taskDir = dataDir + "task_res/"
#logDir = dataDir + "logs/"

# Concrete Task Files
fExp1A = taskDir + "CONTROLLED_EXPFIXED_REWARD_PAY15_DOCSET1.json"
fExp1B = taskDir + "CONTROLLED_EXPFIXED_REWARD_PAY30_DOCSET2.json"
fExp2A = taskDir + "CONTROLLED_EXPFIXED_LEN_PAY10_DOCSET1.json"
fExp2B = taskDir + "CONTROLLED_EXPFIXED_LEN_PAY30_DOCSET2.json"

# Concrete Logged Events Files
fLog1A = logDir + "CONTROLLED_EXPFIXED_REWARD_PAY15_DOCSET1.json"
fLog1B = logDir + "CONTROLLED_EXPFIXED_REWARD_PAY30_DOCSET2.json"
fLog2A = logDir + "CONTROLLED_EXPFIXED_LEN_PAY10_DOCSET1.json"
fLog2B = logDir + "CONTROLLED_EXPFIXED_LEN_PAY30_DOCSET2.json"

Data Preprocessing

TODO: Data Preprocessing Tasks

  • Load data
  • Separate abandoned and submit people like Lei did
  • If we want to compare with the other batch of tasks (aka experiment in the wild) then we need to rescale the relevance judgements because in the first experiment it was a 4-level scale and in this one is 2)
  • Merging the files into a DF that I am interested in
  • Do the split of groups in a similar way - but how to analyse later?
  • In cross-check: did they ensure that in experiments A and B there are always disjoint workers? that would be in-between subject experiment design
exp1A = pd.read_json(path_or_buf = fExp1A, lines = True, encoding = 'utf-8', orient = "records")
exp1B = pd.read_json(path_or_buf = fExp1B, lines = True, encoding = 'utf-8', orient = "records")
exp2A = pd.read_json(path_or_buf = fExp2A, lines = True, encoding = 'utf-8', orient = "records")
exp2B = pd.read_json(path_or_buf = fExp2B, lines = True, encoding = 'utf-8', orient = "records")

log1A = pd.read_json(path_or_buf = fLog1A, lines = True, encoding = 'utf-8', orient = "records")
log1B = pd.read_json(path_or_buf = fLog1B, lines = True, encoding = 'utf-8', orient = "records")
log2A = pd.read_json(path_or_buf = fLog2A, lines = True, encoding = 'utf-8', orient = "records")
log2B = pd.read_json(path_or_buf = fLog2B, lines = True, encoding = 'utf-8', orient = "records")
# Data Format & Content Exploration - with example exp1A 
exp1A.head()
agreement created_at data gold_pool id job_id judgments_count missed_count results state updated_at
0 1 2018-07-25 22:54:27 {'unit_id': '1'} NaN 1830562407 1286845 1 0 {'judgments': [{'id': 3895462947, 'created_at'... finalized 2018-07-26 01:40:32
1 1 2018-07-25 22:54:27 {'unit_id': '2'} NaN 1830562408 1286845 1 0 {'judgments': [{'id': 3895409444, 'created_at'... finalized 2018-07-26 01:03:17
2 1 2018-07-25 22:54:27 {'unit_id': '3'} NaN 1830562409 1286845 1 0 {'judgments': [{'id': 3895402529, 'created_at'... finalized 2018-07-26 00:59:09
3 1 2018-07-25 22:54:27 {'unit_id': '4'} NaN 1830562410 1286845 1 0 {'judgments': [{'id': 3895398081, 'created_at'... finalized 2018-07-26 00:56:45
4 1 2018-07-25 22:54:27 {'unit_id': '5'} NaN 1830562411 1286845 1 0 {'judgments': [{'id': 3895417305, 'created_at'... finalized 2018-07-26 01:08:23
# Create the colum workerid extracted from the dict
def extractUnitId(row):
    resDic = row['data']    
    unitId = resDic['unit_id']
    return unitId
exp1A['unit_id'] = exp1A.apply(extractUnitId,axis=1)
exp1B['unit_id'] = exp1B.apply(extractUnitId,axis=1)
exp2A['unit_id'] = exp2A.apply(extractUnitId,axis=1)
exp2B['unit_id'] = exp2B.apply(extractUnitId,axis=1)
# Create the colum workerid extracted from the dict
def extractWorkerId(row):
    resDic = row['results']
    workerId = resDic['judgments'][0]['worker_id']
    if(len(resDic['judgments']) > 1):
        print('One worker with more than one judgment! '+ str(workerId))
    
    return workerId
    
exp1A['worker_id'] = exp1A.apply(extractWorkerId,axis=1)
exp1B['worker_id'] = exp1B.apply(extractWorkerId,axis=1)
exp2A['worker_id'] = exp2A.apply(extractWorkerId,axis=1)
exp2B['worker_id'] = exp2B.apply(extractWorkerId,axis=1)
# Data Format & Content Exploration - with example log1A 
log1A.head()
judgments message pay server_time session_id step steps task_id time times worker_id
0 [None, -1, -1, -1] Start 15 2018-07-26 00:53:33.683495 0.ldhmu5i2 1 [1] 1286845 1532566410102 [None, 0, 0, 0] 43978656
1 [None, -1, -1, -1] MW Worker Rejected:43978656 15 2018-07-26 00:53:33.911709 0.ldhmu5i2 1 [1] 1286845 1532566410115 [None, 0, 0, 0] 43978656
2 [None, -1, -1, -1] Start 15 2018-07-26 00:53:42.309351 0.2g5zq7ff 1 [1] 1286845 1532566414517 [None, 0, 0, 0] 41202032
3 [None, -1, -1, -1] Start 15 2018-07-26 00:54:02.042663 0.er79jbbc 1 [1] 1286845 1532566439356 [None, 0, 0, 0] 39404795
4 [None, -1, -1, -1] MW Worker Rejected:39404795 15 2018-07-26 00:54:02.229070 0.er79jbbc 1 [1] 1286845 1532566439373 [None, 0, 0, 0] 39404795

Explanation by Kevin: var final_log = { “session_id”: session_id, // unique session id (to capture page refresh) “message”: String(message), // message that triggered the log -- see below “worker_id”: worker_id, // worker id “task_id”:task_id, // task_id “time”: Date.now(), //time of sending log “step”: step, // step into the task (i.e., 1,2,3,4... no_docs, paystep) “judgments”: judgments, //array of judgments -- start at 1 (0 is null) “times”: times, //array of times for the judgments -- start at 1 (0 is null) “steps”: steps, // array of steps in to the task; e.g., if the worker pressed back at step 2 the array is 1,2,1,2,3,... };

the message-set is:

  • nextButton
  • backButton
  • Final_OK --> task concluded succesfully
  • paying --> paying the worker
  • Start --> start task
  • MW Worker Rejected:’+worker_id --> worker blacklisted that tried to start the task
  • MWorker ok --> opposite of the last (not sure if present)
log1A.message.unique()
array(['Start', 'MW Worker Rejected:43978656',
       'MW Worker Rejected:39404795', 'MW Worker Rejected:43726491',
       'MW Worker Rejected:39718779', 'nextButton', 'Final_OK', 'paying',
       'MW Worker Rejected:44239317', 'backButton',
       'MW Worker Rejected:44103483', 'MW Worker Rejected:44179475',
       'MW Worker Rejected:28009209', 'MW Worker Rejected:6476374',
       'MW Worker Rejected:37101159', 'MW Worker Rejected:40073351',
       'MW Worker Rejected:44003641', 'MW Worker Rejected:31822324',
       'MW Worker Rejected:34949514', 'MW Worker Rejected:38968301',
       'MW Worker Rejected:33313161', 'MW Worker Rejected:44204171',
       'MW Worker Rejected:44361126', 'MW Worker Rejected:11001780',
       'MW Worker Rejected:44433928', 'MW Worker Rejected:42074684',
       'MW Worker Rejected:44259527', 'MW Worker Rejected:18495435',
       'MW Worker Rejected:33453966', 'MW Worker Rejected:43967749'],
      dtype=object)
sessions = log1A[['worker_id', 'session_id']]
sessions.groupby(['worker_id']).size().unique()
array([ 2,  6,  8,  1,  7,  3,  9,  4, 10, 14,  5, 11])
import json
def getJudgments(row):
    text_result = row['results']['judgments'][0]['data']['text_result']
    textrjson = json.loads(text_result)
    judgments = textrjson['judgments']
     # return pd.Series(judgments) OK but just the array expects the shape of the original data frame calling the apply
    return len(judgments)
# Helpers
def countJudgments(row):
    #return len(row['judgments'])
    return row['judgments_count'] # it's wrapped in the judgments - data
# Cross checks & Basic stats - units per people etc. Global and separating people? 
def checkTask(taskDf):
    
    # checking published config
    print('total number of HITs:' + str(len(taskDf)))
    # KO print('number of judgments per HIT' + str(taskDf.results.map(lambda x: len(x)).max()))   
    nulls = pd.isnull(taskDf)
    
    # missing values
       
    print('Empty value in data column: ' + str(len(nulls.loc[nulls['data'] == True])) + ' out of '+ str(len(nulls['data'])))
    print('Empty value in results column: ' + str(len(nulls.loc[nulls['results'] == True])) + ' out of '+ str(len(nulls['results'])))
    print('Empty value in created_at column: ' + str(len(nulls.loc[nulls['created_at'] == True])) + ' out of '+ str(len(nulls['created_at'])))
    print('Empty value in updated_at column: ' + str(len(nulls.loc[nulls['updated_at'] == True])) + ' out of '+ str(len(nulls['updated_at'])))
    print('Empty value in id column: ' + str(len(nulls.loc[nulls['id'] == True])) + 'out of '+ str(len(nulls['id'])))
    print('Empty value in job_id column: ' + str(len(nulls.loc[nulls['job_id'] == True])) + ' out of '+ str(len(nulls['job_id'])))
    print('Empty value in worker_id column: ' + str(len(nulls.loc[nulls['worker_id'] == True])) + ' out of '+ str(len(nulls['worker_id'])))
    print('Empty value in unit_id column: ' + str(len(nulls.loc[nulls['unit_id'] == True])) + ' out of '+ str(len(nulls['unit_id'])))
    
    
    
    # counts
    print('Total number of workers: ' + str(taskDf['worker_id'].nunique()))
    print('Total number of units - they are judgments: ' + str(taskDf['unit_id'].nunique())) 
    print('AVG Number of units per worker: '+ str(taskDf.groupby(['worker_id'])['unit_id'].nunique().mean()) + ' Max Number of units per worker: '+ str(taskDf.groupby(['worker_id'])['unit_id'].nunique().max()) )
    print('Number of judgments per worker: ' )
    judgmentsCount = pd.Series()
    # when returning an array it takes the length of the DF here! Pandas - print(len(taskDf.columns))
    judgmentsCount = taskDf.apply(getJudgments,axis=1)
    print(judgmentsCount.describe())
        
   
exp1A['results'][0]['judgments'][0]['data']['text_result'] # this one gives an array of 4?
'{"session_id":"0.4nbus5f6","message":"paying","worker_id":33017350,"task_id":1286845,"time":1532569214713,"step":5,"judgments":[null,"2","0","0"],"times":[null,77.2,70.998,41.138],"steps":[1,2,3,4,5],"pay":15}'
checkTask(exp1A) # is the title of the files misleading? From the number of judgments sent by workers it looks like exp1A is the one of the lenth
checkTask(exp1B)
checkTask(exp2A)
checkTask(exp2B)
total number of HITs:100
Empty value in data column: 0 out of 100
Empty value in results column: 0 out of 100
Empty value in created_at column: 0 out of 100
Empty value in updated_at column: 0 out of 100
Empty value in id column: 0out of 100
Empty value in job_id column: 0 out of 100
Empty value in worker_id column: 0 out of 100
Empty value in unit_id column: 0 out of 100
Total number of workers: 100
Total number of units - they are judgments: 100
AVG Number of units per worker: 1.0 Max Number of units per worker: 1
Number of judgments per worker: 
count    100.0
mean       4.0
std        0.0
min        4.0
25%        4.0
50%        4.0
75%        4.0
max        4.0
dtype: float64
total number of HITs:100
Empty value in data column: 0 out of 100
Empty value in results column: 0 out of 100
Empty value in created_at column: 0 out of 100
Empty value in updated_at column: 0 out of 100
Empty value in id column: 0out of 100
Empty value in job_id column: 0 out of 100
Empty value in worker_id column: 0 out of 100
Empty value in unit_id column: 0 out of 100
Total number of workers: 100
Total number of units - they are judgments: 100
AVG Number of units per worker: 1.0 Max Number of units per worker: 1
Number of judgments per worker: 
count    100.0
mean       7.0
std        0.0
min        7.0
25%        7.0
50%        7.0
75%        7.0
max        7.0
dtype: float64
total number of HITs:100
Empty value in data column: 0 out of 100
Empty value in results column: 0 out of 100
Empty value in created_at column: 0 out of 100
Empty value in updated_at column: 0 out of 100
Empty value in id column: 0out of 100
Empty value in job_id column: 0 out of 100
Empty value in worker_id column: 0 out of 100
Empty value in unit_id column: 0 out of 100
Total number of workers: 100
Total number of units - they are judgments: 100
AVG Number of units per worker: 1.0 Max Number of units per worker: 1
Number of judgments per worker: 
count    100.0
mean       7.0
std        0.0
min        7.0
25%        7.0
50%        7.0
75%        7.0
max        7.0
dtype: float64
total number of HITs:100
Empty value in data column: 0 out of 100
Empty value in results column: 0 out of 100
Empty value in created_at column: 0 out of 100
Empty value in updated_at column: 0 out of 100
Empty value in id column: 0out of 100
Empty value in job_id column: 0 out of 100
Empty value in worker_id column: 0 out of 100
Empty value in unit_id column: 0 out of 100
Total number of workers: 100
Total number of units - they are judgments: 100
AVG Number of units per worker: 1.0 Max Number of units per worker: 1
Number of judgments per worker: 
count    100.0
mean       7.0
std        0.0
min        7.0
25%        7.0
50%        7.0
75%        7.0
max        7.0
dtype: float64
def checkLog(logDf):
    # missing values
    nulls = pd.isnull(logDf)
    print('Empty value in data column: ' + str(len(nulls.loc[nulls['message'] == True])) + 'out of '+ str(len(nulls['message'])))
    print('Empty value in session_id column: ' + str(len(nulls.loc[nulls['session_id'] == True])) + 'out of '+ str(len(nulls['session_id'])))
    print('Empty value in task_id column: ' + str(len(nulls.loc[nulls['task_id'] == True])) + 'out of '+ str(len(nulls['task_id'])))
    print('Empty value in time column: ' + str(len(nulls.loc[nulls['time'] == True])) + 'out of '+ str(len(nulls['time'])))
    print('Empty value in times column: ' + str(len(nulls.loc[nulls['times'] == True])) + 'out of '+ str(len(nulls['times'])))
    print('Empty value in worker_id column: ' + str(len(nulls.loc[nulls['worker_id'] == True])) + 'out of '+ str(len(nulls['worker_id'])))
    print('Empty value in pay column: ' + str(len(nulls.loc[nulls['pay'] == True])) + 'out of '+ str(len(nulls['pay'])))
    print('Empty value in judgmens column: ' + str(len(nulls.loc[nulls['judgments'] == True])) + 'out of '+ str(len(nulls['judgments'])))

    # counts
    print('Total number of workers: ' + str(logDf['worker_id'].nunique()))
    print('Total number of tasks: ' + str(logDf['task_id'].nunique())) # task = unit
    print('AVG Number of sessions per worker: '+ str(logDf.groupby(['worker_id'])['session_id'].nunique().mean()) + ' Max Number of sessions per worker: '+ str(logDf.groupby(['worker_id'])['session_id'].nunique().max()) )
    print('AVG Number of tasks per worker: '+ str(logDf.groupby(['worker_id'])['task_id'].nunique().mean()) + ' Max Number of tasks per worker: '+ str(logDf.groupby(['worker_id'])['task_id'].nunique().max()) )
checkLog(log1A)
checkLog(log1B)
checkLog(log2A)
checkLog(log2B)
Empty value in data column: 0out of 790
Empty value in session_id column: 0out of 790
Empty value in task_id column: 0out of 790
Empty value in time column: 0out of 790
Empty value in times column: 0out of 790
Empty value in worker_id column: 0out of 790
Empty value in pay column: 0out of 790
Empty value in judgmens column: 0out of 790
Total number of workers: 209
Total number of tasks: 1
AVG Number of sessions per worker: 1.1961722488038278 Max Number of sessions per worker: 6
AVG Number of tasks per worker: 1.0 Max Number of tasks per worker: 1
Empty value in data column: 0out of 1129
Empty value in session_id column: 0out of 1129
Empty value in task_id column: 0out of 1129
Empty value in time column: 0out of 1129
Empty value in times column: 0out of 1129
Empty value in worker_id column: 0out of 1129
Empty value in pay column: 0out of 1129
Empty value in judgmens column: 0out of 1129
Total number of workers: 202
Total number of tasks: 1
AVG Number of sessions per worker: 1.2227722772277227 Max Number of sessions per worker: 8
AVG Number of tasks per worker: 1.0 Max Number of tasks per worker: 1
Empty value in data column: 0out of 1089
Empty value in session_id column: 0out of 1089
Empty value in task_id column: 1out of 1089
Empty value in time column: 0out of 1089
Empty value in times column: 0out of 1089
Empty value in worker_id column: 0out of 1089
Empty value in pay column: 0out of 1089
Empty value in judgmens column: 0out of 1089
Total number of workers: 207
Total number of tasks: 1
AVG Number of sessions per worker: 1.1159420289855073 Max Number of sessions per worker: 6
AVG Number of tasks per worker: 0.9951690821256038 Max Number of tasks per worker: 1
Empty value in data column: 0out of 1146
Empty value in session_id column: 0out of 1146
Empty value in task_id column: 1out of 1146
Empty value in time column: 0out of 1146
Empty value in times column: 0out of 1146
Empty value in worker_id column: 0out of 1146
Empty value in pay column: 0out of 1146
Empty value in judgmens column: 0out of 1146
Total number of workers: 178
Total number of tasks: 1
AVG Number of sessions per worker: 1.1629213483146068 Max Number of sessions per worker: 4
AVG Number of tasks per worker: 0.9943820224719101 Max Number of tasks per worker: 1
def checkTaskJobJointly(taskDf, logDf):
    
    abandonedDf = logDf[~logDf['worker_id'].isin(taskDf['worker_id'])]
    completedDf = logDf[logDf['worker_id'].isin(taskDf['worker_id'])]
    
    # all the answers in the task completion report are also in the log data set
    print('Number of people who abandoned: ' + str(len(logDf['worker_id'][~logDf['worker_id'].isin(taskDf['worker_id'])].unique())) ) #+ ' and they are IDs: '+  str(logDf['worker_id'][~logDf['worker_id'].isin(taskDf['worker_id'])])
    print('Number of people who submitted: ' + str(len(logDf['worker_id'][logDf['worker_id'].isin(taskDf['worker_id'])].unique())) ) #+ ' and they are IDs: '+  str(logDf['worker_id'][logDf['worker_id'].isin(taskDf['worker_id'])])
    print('Number of people who were not logged: ' + str(len(taskDf['worker_id'][~taskDf['worker_id'].isin(logDf['worker_id'])].unique())) )
    print('*Total number of workers in Task*: '+ str(taskDf['worker_id'].nunique()))
    print('*Total number of workers in Log*: '+ str(logDf['worker_id'].nunique()))
    
    return abandonedDf, completedDf
   
    

print('--- Experiment 1  ------------------------')
print('--- (A) ------------------------')
aban_1A, complet_1A = checkTaskJobJointly(exp1A, log1A)
print('--- (B) ------------------------')
aban_1B, complet_1B = checkTaskJobJointly(exp1B, log1B)
print('--- Experiment 2  ------------------------')
print('--- (A) ------------------------')
aban_2A, complet_2A = checkTaskJobJointly(exp2A, log2A)
print('--- (B) ------------------------')
aban_2B, complet_2B = checkTaskJobJointly(exp2B, log2B)
--- Experiment 1  ------------------------
--- (A) ------------------------
Number of people who abandoned: 109
Number of people who submitted: 100
Number of people who were not logged: 0
*Total number of workers in Task*: 100
*Total number of workers in Log*: 209
--- (B) ------------------------
Number of people who abandoned: 102
Number of people who submitted: 100
Number of people who were not logged: 0
*Total number of workers in Task*: 100
*Total number of workers in Log*: 202
--- Experiment 2  ------------------------
--- (A) ------------------------
Number of people who abandoned: 107
Number of people who submitted: 100
Number of people who were not logged: 0
*Total number of workers in Task*: 100
*Total number of workers in Log*: 207
--- (B) ------------------------
Number of people who abandoned: 78
Number of people who submitted: 100
Number of people who were not logged: 0
*Total number of workers in Task*: 100
*Total number of workers in Log*: 178

Building the 4 groups of people:

Focus is on the log files, filtering in one way or the other.

# Get the two subgroups for abandoned workes, who either abandones right away or abandoned after restarting -- more than one session
def abandSpec(df):
    # (!!) Pandas passes through the first twice
    dfG = df.groupby(['worker_id'])
    abanA = dfG.filter(lambda x: len(x['session_id'].unique()) == 1)
    abanB = dfG.filter(lambda x: len(x['session_id'].unique()) > 1)
    return abanA,abanB
# Get the two subgroups for completed workes, who either submitted answers right away or submitted after restarting -- more than one session
# Coded in a different method for extensibility reasons
def completSpec(df):
    # (!!) Pandas passes through the first twice
    dfG = df.groupby(['worker_id'])
    complA = dfG.filter(lambda x: len(x['session_id'].unique()) == 1)
    complB = dfG.filter(lambda x: len(x['session_id'].unique()) > 1)
    return complA,complB
    
    
# Get all the concrete subsets for all versions of the two controlled experiments.

# Experiment 1 (A,B settings)
abanA_1A,abanB_1A = abandSpec(aban_1A)
completA_1A,completB_1A = completSpec(complet_1A)

abanA_1B,abanB_1B = abandSpec(aban_1B)
completA_1B,completB_1B = completSpec(complet_1B)

# Experiment 2 (A,B settings)
abanA_2A,abanB_2A = abandSpec(aban_2A)
completA_2A,completB_2A = completSpec(complet_2A)

abanA_2B,abanB_2B = abandSpec(aban_2B)
completA_2B,completB_2B = completSpec(complet_2B)
# Cross-check - CORRECT
print('abandoned subgroups 1A')
print(abanA_1A.worker_id.nunique() + abanB_1A.worker_id.nunique())
print('abandoned subgroups 1B')
print(abanA_1B.worker_id.nunique() + abanB_1B.worker_id.nunique())
print('abandoned subgroups 2A')
print(abanA_2A.worker_id.nunique() + abanB_2A.worker_id.nunique())
print('abandoned subgroups 2B')
print(abanA_2B.worker_id.nunique() + abanB_2B.worker_id.nunique())

print('completed subgroups 1A')
print(completA_1A.worker_id.nunique() + completB_1A.worker_id.nunique())
print('completed subgroups 1B')
print(completA_1B.worker_id.nunique() + completB_1B.worker_id.nunique())
print('completed subgroups 2A')
print(completA_2A.worker_id.nunique() + completB_2A.worker_id.nunique())
print('completed subgroups 2B')
print(completA_2B.worker_id.nunique() + completB_2B.worker_id.nunique())
abandoned subgroups 1A
109
abandoned subgroups 1B
102
abandoned subgroups 2A
107
abandoned subgroups 2B
78
completed subgroups 1A
100
completed subgroups 1B
100
completed subgroups 2A
100
completed subgroups 2B
100
# ----- Testing Pandas
#log1A[log1A['worker_id']==41202032]
#d = log1A.sort_values(by=['worker_id'])
#d.head(100)
#log1Ag = log1A.groupby(['worker_id'])
#abb = log1Ag.filter(lambda x: len(x['session_id'].unique()) > 1)
#abb
#abb.groupby(['worker_id']).get_group(41202032)
#abb.groupby(['worker_id']).get_group(6476374) #- does not find it - it's correct
# --
# a = [1,2,3]
# b = [2,3,4]
# data = pd.DataFrame()
# data['a'] = pd.Series(a)
# data['b'] = pd.Series(b)
# data.head()
# data['a'][~data['a'].isin(data['b'])]
# data['a'][data['a'].isin(data['b'])]
# data['a'].isin(data['b'])
# data[~data['a'].isin(data['b'])]
# ----------- end of testing Pandas

Experiment-based Hypotheses

Compare per group, compare per experiment controlled.

from scipy.stats import shapiro
from scipy.stats import normaltest
from scipy.stats import anderson

# Input: series has the sample whose distribution we want to test
# Output: gaussian boolean True if it is normal distribution and False otherwise.
def testNormality(series):
    
    alpha = 0.05
    gaussian = False
    
    # only if the three tests give normal will be normal. If we find one that is not passed, then it is NOT normal. 
    
    # Shapiro-Wilk Test - for smaller data sets around thousands of records
    print('length of series in Shapiro is: '+ str(len(series)))
    stats1, p1 = shapiro(series)
    print('Statistics Shapiro-Wilk Test =%.3f, p=%.3f' % (stats1, p1))
    if p1 > alpha:
        gaussian = True
    print('Shapiro.Wilk says it is Normal '+ str(gaussian))
    
    gaussian = False # because of intermediate printing, reinitialize
    # D'Agostino and Pearson's Test
    stats2, p2 = normaltest(series) #dataw.humid
    print('Statistics D\'Agostino and Pearson\'s Test=%.3f, p=%.3f' % (stats2, p2))
    if p2 > alpha:
        gaussian = False
        print('D\'Agostino and Pearson\'s says it is Normal '+ str(gaussian))
    
    # Anderson-Darling Test
    '''result = anderson(series) 
    print('Statistic: %.3f' % result.statistic)
    for i in range(len(result.critical_values)):
        sl, cv = result.significance_level[i], result.critical_values[i]
        if result.statistic > result.critical_values[i]:
            gaussian = False'''
        
    
    return gaussian
    
from scipy.stats import ttest_ind
from scipy.stats import mannwhitneyu

# Input:
# series1 is the series with the set of measurements for every single worker in case A of controlled experiment
# series2 is the series with the set of measurements for every single worker in case B of controlled experiment
# gaussian is the boolean value indicating if the samples have passed the test of normality or not (True is apply parametric test)
# Output:
# stats of statistical test 
# p-value 
# acceptHo (True if we fail to reject it and False if we reject it) 
# See also all tables for picking the tests (e.g., https://help.xlstat.com/customer/en/portal/articles/2062457-which-statistical-test-should-you-use-)
def compareTwoSamples(series1,series2, gaussian):
    # Tests to compare two samples (H0: they have equal distribution; H1: they have different distribution)
    
    alpha = 0.05
    acceptH0 = False
    
    if (gaussian == True):
        # Run Student's T-test
        stats, p = ttest_ind(series1, series2)
        print('Statistics=%.3f, p=%.3f' % (stats, p))
        
    else:
        
        # Run Mann-Whitney; Kruskal-Wallis test is for more samples.
        stats, p = mannwhitneyu(series1, series2)
        print('Statistics=%.3f, p=%.3f' % (stats, p))
        
        # result - hypothesis testing
   
    if p > alpha:
        acceptH0 = True
    
    print('The two samples have the same distribution (meanA = meanB): ' + str(acceptH0))
    return stats,p,acceptH0        
        
    
    
    

Work Done

Just compare per experiment controlled.

Comparing the means of sessions done by workers in the two populations

Idea: In those that abandoned, the higher the reward and the longer the set of documents the more value the HIT has, the more they try:

H0R means of session count are equal in both populations (reward)

H1R not equal

H0L means of session count are equal in both populations (length)

H1L not equal

# Functions to compute the measurements
def getSessionCount(df):
    dfG = df.groupby(['worker_id'])
    print(dfG.head(10))
    sessionCounts = dfG.apply(lambda x: len(x['session_id'].unique()))
    sessionCountsRI = sessionCounts.reset_index()
    del(sessionCountsRI['worker_id'])
    sessionCountsRI.columns=['sessionCount']
    return sessionCountsRI
def getMessageCount(df):
    dfG = df.groupby(['worker_id'])
    sessionCounts = dfG.apply(lambda x: len(x['message']))
 
print(len(abanA_1A))
sessionC_abanA_1A = getSessionCount(abanA_1A)
sessionC_abanB_1A = getSessionCount(abanB_1A)
sessionC_abanA_1B = getSessionCount(abanA_1B)
sessionC_abanB_1B = getSessionCount(abanB_1B)
sessionC_abanA_2A = getSessionCount(abanA_2A)
sessionC_abanB_2A = getSessionCount(abanB_2A)
sessionC_abanA_2B = getSessionCount(abanA_2B)
sessionC_abanB_2B = getSessionCount(abanB_2B)
115
              judgments                      message  pay  \
0    [None, -1, -1, -1]                        Start   15   
1    [None, -1, -1, -1]  MW Worker Rejected:43978656   15   
3    [None, -1, -1, -1]                        Start   15   
4    [None, -1, -1, -1]  MW Worker Rejected:39404795   15   
7    [None, -1, -1, -1]  MW Worker Rejected:43726491   15   
8    [None, -1, -1, -1]                        Start   15   
9    [None, -1, -1, -1]                        Start   15   
10   [None, -1, -1, -1]                        Start   15   
11   [None, -1, -1, -1]                        Start   15   
12   [None, -1, -1, -1]                        Start   15   
20   [None, -1, -1, -1]                        Start   15   
41   [None, -1, -1, -1]                        Start   15   
44   [None, -1, -1, -1]                        Start   15   
47   [None, -1, -1, -1]                        Start   15   
56   [None, -1, -1, -1]                        Start   15   
58   [None, -1, -1, -1]                        Start   15   
59   [None, -1, -1, -1]  MW Worker Rejected:44239317   15   
60   [None, -1, -1, -1]                        Start   15   
67   [None, -1, -1, -1]                        Start   15   
69   [None, -1, -1, -1]                        Start   15   
80   [None, -1, -1, -1]                        Start   15   
89   [None, -1, -1, -1]                        Start   15   
91   [None, -1, -1, -1]                        Start   15   
92   [None, -1, -1, -1]                        Start   15   
93   [None, -1, -1, -1]                        Start   15   
94   [None, -1, -1, -1]                        Start   15   
105  [None, -1, -1, -1]                        Start   15   
114  [None, -1, -1, -1]                        Start   15   
125  [None, -1, -1, -1]                        Start   15   
132  [None, -1, -1, -1]                        Start   15   
..                  ...                          ...  ...   
498  [None, -1, -1, -1]                        Start   15   
500  [None, -1, -1, -1]  MW Worker Rejected:11001780   15   
501  [None, -1, -1, -1]                        Start   15   
502  [None, -1, -1, -1]                        Start   15   
503   [None, 1, -1, -1]                   nextButton   15   
507  [None, -1, -1, -1]  MW Worker Rejected:44433928   15   
508  [None, -1, -1, -1]                        Start   15   
510  [None, -1, -1, -1]                        Start   15   
512  [None, -1, -1, -1]                        Start   15   
539  [None, -1, -1, -1]                        Start   15   
549  [None, -1, -1, -1]                        Start   15   
550  [None, -1, -1, -1]  MW Worker Rejected:44259527   15   
595  [None, -1, -1, -1]                        Start   15   
596  [None, -1, -1, -1]                        Start   15   
604  [None, -1, -1, -1]                        Start   15   
613  [None, -1, -1, -1]  MW Worker Rejected:18495435   15   
614  [None, -1, -1, -1]                        Start   15   
615  [None, -1, -1, -1]                        Start   15   
633   [None, 2, -1, -1]                   nextButton   15   
642  [None, -1, -1, -1]                        Start   15   
643  [None, -1, -1, -1]                        Start   15   
648  [None, -1, -1, -1]                        Start   15   
656  [None, -1, -1, -1]                        Start   15   
659  [None, -1, -1, -1]                        Start   15   
703  [None, -1, -1, -1]                        Start   15   
705  [None, -1, -1, -1]                        Start   15   
706  [None, -1, -1, -1]                        Start   15   
707  [None, -1, -1, -1]  MW Worker Rejected:43967749   15   
727  [None, -1, -1, -1]                        Start   15   
745  [None, -1, -1, -1]                        Start   15   

                   server_time  session_id  step steps  task_id  \
0   2018-07-26 00:53:33.683495  0.ldhmu5i2     1   [1]  1286845   
1   2018-07-26 00:53:33.911709  0.ldhmu5i2     1   [1]  1286845   
3   2018-07-26 00:54:02.042663  0.er79jbbc     1   [1]  1286845   
4   2018-07-26 00:54:02.229070  0.er79jbbc     1   [1]  1286845   
7   2018-07-26 00:54:48.864196  0.uza6eu1t     1   [1]  1286845   
8   2018-07-26 00:54:48.979677  0.uza6eu1t     1   [1]  1286845   
9   2018-07-26 00:54:51.969131  0.p0e9h57p     1   [1]  1286845   
10  2018-07-26 00:54:55.876113  0.xjujh1cr     1   [1]  1286845   
11  2018-07-26 00:54:58.654957  0.4x4x9szv     1   [1]  1286845   
12  2018-07-26 00:54:59.442455  0.lkid6bky     1   [1]  1286845   
20  2018-07-26 00:55:39.703202  0.8v1raaiq     1   [1]  1286845   
41  2018-07-26 00:56:33.265282  0.8sll1a5a     1   [1]  1286845   
44  2018-07-26 00:56:43.319008  0.65oeoit6     1   [1]  1286845   
47  2018-07-26 00:56:49.032120  0.8rutsahu     1   [1]  1286845   
56  2018-07-26 00:57:11.288635  0.47shrmue     1   [1]  1286845   
58  2018-07-26 00:57:14.329126  0.2c22qjdm     1   [1]  1286845   
59  2018-07-26 00:57:14.597636  0.2c22qjdm     1   [1]  1286845   
60  2018-07-26 00:57:17.418025  0.gbyq0td7     1   [1]  1286845   
67  2018-07-26 00:57:32.346766  0.6s3ix0ut     1   [1]  1286845   
69  2018-07-26 00:57:34.610276  0.ata55qh8     1   [1]  1286845   
80  2018-07-26 00:58:10.689900  0.7ja6sxqg     1   [1]  1286845   
89  2018-07-26 00:58:24.589581  0.mjikvt75     1   [1]  1286845   
91  2018-07-26 00:58:26.618296  0.gjo5nbol     1   [1]  1286845   
92  2018-07-26 00:58:27.665664  0.qs0vox8v     1   [1]  1286845   
93  2018-07-26 00:58:30.823847  0.g6j3eyiw     1   [1]  1286845   
94  2018-07-26 00:58:31.177673  0.2e3bdlz1     1   [1]  1286845   
105 2018-07-26 00:58:57.316368  0.oomb7u8j     1   [1]  1286845   
114 2018-07-26 00:59:29.055777  0.wmhst2rz     1   [1]  1286845   
125 2018-07-26 01:00:06.280091  0.65sgrdoq     1   [1]  1286845   
132 2018-07-26 01:00:17.502234  0.45k7iziu     1   [1]  1286845   
..                         ...         ...   ...   ...      ...   
498 2018-07-26 01:35:55.443811  0.vyh3q8g3     1   [1]  1286845   
500 2018-07-26 01:36:02.598126  0.04sdyaev     1   [1]  1286845   
501 2018-07-26 01:36:02.598126  0.04sdyaev     1   [1]  1286845   
502 2018-07-26 01:36:14.214697  0.4ygobvb9     1   [1]  1286845   
503 2018-07-26 01:36:33.212053  0.gssqpg9n     1   [1]  1286845   
507 2018-07-26 01:37:04.662651  0.0fd1wif2     1   [1]  1286845   
508 2018-07-26 01:37:04.682952  0.0fd1wif2     1   [1]  1286845   
510 2018-07-26 01:37:10.309994  0.s8hx1j3u     1   [1]  1286845   
512 2018-07-26 01:37:11.557914  0.qfhr9iru     1   [1]  1286845   
539 2018-07-26 01:40:06.756334  0.7z5air2c     1   [1]  1286845   
549 2018-07-26 01:40:34.565832  0.2t4nkmyo     1   [1]  1286845   
550 2018-07-26 01:40:34.607752  0.2t4nkmyo     1   [1]  1286845   
595 2018-07-26 01:56:32.693144  0.crk42iu9     1   [1]  1286845   
596 2018-07-26 01:57:15.678638  0.29holi1v     1   [1]  1286845   
604 2018-07-26 01:58:31.382180  0.a8vlnt88     1   [1]  1286845   
613 2018-07-26 01:59:46.286735  0.2xt8xi0l     1   [1]  1286845   
614 2018-07-26 01:59:46.336816  0.2xt8xi0l     1   [1]  1286845   
615 2018-07-26 01:59:54.825963  0.n27j7nve     1   [1]  1286845   
633 2018-07-26 02:01:34.068846  0.n27j7nve     1   [1]  1286845   
642 2018-07-26 02:06:37.185105  0.mcwlw4j7     1   [1]  1286845   
643 2018-07-26 02:06:48.251492  0.maareiu6     1   [1]  1286845   
648 2018-07-26 02:07:29.951751  0.6c8vqe91     1   [1]  1286845   
656 2018-07-26 02:08:26.948052  0.nrp5v0wl     1   [1]  1286845   
659 2018-07-26 02:08:36.552379  0.gq8nf1u4     1   [1]  1286845   
703 2018-07-26 02:29:51.201131  0.3lgix2xa     1   [1]  1286845   
705 2018-07-26 02:30:09.151958  0.yt8sdspq     1   [1]  1286845   
706 2018-07-26 02:30:13.151716  0.l1jo6nfa     1   [1]  1286845   
707 2018-07-26 02:30:13.595330  0.l1jo6nfa     1   [1]  1286845   
727 2018-07-26 02:36:44.949902  0.sml82me3     1   [1]  1286845   
745 2018-07-26 02:38:59.587652  0.dmnnoi37     1   [1]  1286845   

              time            times  worker_id  
0    1532566410102  [None, 0, 0, 0]   43978656  
1    1532566410115  [None, 0, 0, 0]   43978656  
3    1532566439356  [None, 0, 0, 0]   39404795  
4    1532566439373  [None, 0, 0, 0]   39404795  
7    1532566500887  [None, 0, 0, 0]   43726491  
8    1532566500868  [None, 0, 0, 0]   43726491  
9    1532566490959  [None, 0, 0, 0]   33712172  
10   1532566493564  [None, 0, 0, 0]   44068769  
11   1532566509288  [None, 0, 0, 0]   44113607  
12   1532566497485  [None, 0, 0, 0]   43840260  
20   1532566535538  [None, 0, 0, 0]   44712145  
41   1532566613898  [None, 0, 0, 0]   43718452  
44   1532566599226  [None, 0, 0, 0]   44618730  
47   1532566603633  [None, 0, 0, 0]   44514533  
56   1532566628330  [None, 0, 0, 0]   34968890  
58   1532566627714  [None, 0, 0, 0]   44239317  
59   1532566627777  [None, 0, 0, 0]   44239317  
60   1532566636801  [None, 0, 0, 0]   43702983  
67   1532566650813  [None, 0, 0, 0]   43667456  
69   1532566652434  [None, 0, 0, 0]   36769251  
80   1532566743943  [None, 0, 0, 0]   40928327  
89   1532566663491  [None, 0, 0, 0]   41841779  
91   1532566705605  [None, 0, 0, 0]   44489666  
92   1532566703402  [None, 0, 0, 0]   44334102  
93   1532566629732  [None, 0, 0, 0]   44258615  
94   1532566709516  [None, 0, 0, 0]   34837948  
105  1532566735468  [None, 0, 0, 0]   43920794  
114  1532566763898  [None, 0, 0, 0]   44158351  
125  1532566799543  [None, 0, 0, 0]   43852389  
132  1532566814173  [None, 0, 0, 0]   42836146  
..             ...              ...        ...  
498  1532568951139  [None, 0, 0, 0]   44171284  
500  1532568965157  [None, 0, 0, 0]   11001780  
501  1532568965150  [None, 0, 0, 0]   11001780  
502  1532568964944  [None, 0, 0, 0]   43999875  
503  1532568967681  [None, 0, 0, 0]   43425397  
507  1532569156644  [None, 0, 0, 0]   44433928  
508  1532569156594  [None, 0, 0, 0]   44433928  
510  1532569027590  [None, 0, 0, 0]   44633075  
512  1532569012025  [None, 0, 0, 0]   41503745  
539  1532569201190  [None, 0, 0, 0]   39697270  
549  1532569233513  [None, 0, 0, 0]   44259527  
550  1532569233527  [None, 0, 0, 0]   44259527  
595  1532570086539  [None, 0, 0, 0]   38585736  
596  1532570234190  [None, 0, 0, 0]   44165114  
604  1532570308353  [None, 0, 0, 0]   43744819  
613  1532570382511  [None, 0, 0, 0]   18495435  
614  1532570382420  [None, 0, 0, 0]   18495435  
615  1532548736096  [None, 0, 0, 0]   44085613  
633  1532548835550  [None, 0, 0, 0]   44085613  
642  1532570733801  [None, 0, 0, 0]   43636336  
643  1532570801043  [None, 0, 0, 0]   44619684  
648  1532570846802  [None, 0, 0, 0]   44507993  
656  1532570899315  [None, 0, 0, 0]   44395248  
659  1532570903785  [None, 0, 0, 0]   43843151  
703  1532572189403  [None, 0, 0, 0]   44160341  
705  1532572207651  [None, 0, 0, 0]   43531815  
706  1532572209309  [None, 0, 0, 0]   43967749  
707  1532572209321  [None, 0, 0, 0]   43967749  
727  1532572608100  [None, 0, 0, 0]   43644467  
745  1532572740443  [None, 0, 0, 0]   43348892  

[115 rows x 11 columns]
              judgments                      message  pay  \
5    [None, -1, -1, -1]                        Start   15   
6    [None, -1, -1, -1]                        Start   15   
18   [None, -1, -1, -1]                        Start   15   
19   [None, -1, -1, -1]  MW Worker Rejected:39718779   15   
21   [None, -1, -1, -1]                        Start   15   
23   [None, -1, -1, -1]                        Start   15   
24   [None, -1, -1, -1]  MW Worker Rejected:39718779   15   
25   [None, -1, -1, -1]                        Start   15   
46   [None, -1, -1, -1]                        Start   15   
57    [None, 3, -1, -1]                   nextButton   15   
64     [None, 3, 3, -1]                   backButton   15   
65     [None, 3, 3, -1]                   nextButton   15   
68   [None, -1, -1, -1]                        Start   15   
77   [None, -1, -1, -1]                        Start   15   
98   [None, -1, -1, -1]                        Start   15   
99   [None, -1, -1, -1]                        Start   15   
100  [None, -1, -1, -1]  MW Worker Rejected:44103483   15   
102  [None, -1, -1, -1]                        Start   15   
103  [None, -1, -1, -1]  MW Worker Rejected:44103483   15   
108  [None, -1, -1, -1]                        Start   15   
109  [None, -1, -1, -1]  MW Worker Rejected:44103483   15   
142  [None, -1, -1, -1]                        Start   15   
189  [None, -1, -1, -1]                        Start   15   
229  [None, -1, -1, -1]                        Start   15   
230  [None, -1, -1, -1]                        Start   15   
399  [None, -1, -1, -1]                        Start   15   
407  [None, -1, -1, -1]                        Start   15   
410  [None, -1, -1, -1]                        Start   15   
415  [None, -1, -1, -1]                        Start   15   
416  [None, -1, -1, -1]                        Start   15   
437  [None, -1, -1, -1]                        Start   15   
443  [None, -1, -1, -1]                        Start   15   
457  [None, -1, -1, -1]                        Start   15   
459  [None, -1, -1, -1]                        Start   15   
521  [None, -1, -1, -1]                        Start   15   
522  [None, -1, -1, -1]  MW Worker Rejected:42074684   15   
565  [None, -1, -1, -1]                        Start   15   
566  [None, -1, -1, -1]  MW Worker Rejected:42074684   15   
590  [None, -1, -1, -1]                        Start   15   
591  [None, -1, -1, -1]                        Start   15   
603  [None, -1, -1, -1]                        Start   15   
609  [None, -1, -1, -1]                        Start   15   
610  [None, -1, -1, -1]                        Start   15   
612  [None, -1, -1, -1]                        Start   15   
616  [None, -1, -1, -1]                        Start   15   
641  [None, -1, -1, -1]                        Start   15   
646  [None, -1, -1, -1]                        Start   15   
649  [None, -1, -1, -1]                        Start   15   
655  [None, -1, -1, -1]                        Start   15   
664  [None, -1, -1, -1]                        Start   15   
665  [None, -1, -1, -1]  MW Worker Rejected:33453966   15   

                   server_time  session_id  step      steps  task_id  \
5   2018-07-26 00:54:19.809559  0.ezd4zyge     1        [1]  1286845   
6   2018-07-26 00:54:40.916040  0.3s4jpneg     1        [1]  1286845   
18  2018-07-26 00:55:34.665218  0.1dczio8o     1        [1]  1286845   
19  2018-07-26 00:55:34.719920  0.1dczio8o     1        [1]  1286845   
21  2018-07-26 00:55:44.430293  0.9hix52z5     1        [1]  1286845   
23  2018-07-26 00:55:47.809914  0.13ydycyv     1        [1]  1286845   
24  2018-07-26 00:55:48.220153  0.13ydycyv     1        [1]  1286845   
25  2018-07-26 00:55:52.895621  0.vvwhho2j     1        [1]  1286845   
46  2018-07-26 00:56:48.354278  0.mw6u1int     1        [1]  1286845   
57  2018-07-26 00:57:12.286479  0.mw6u1int     1        [1]  1286845   
64  2018-07-26 00:57:18.471016  0.mw6u1int     2     [1, 2]  1286845   
65  2018-07-26 00:57:21.560898  0.mw6u1int     1  [1, 2, 1]  1286845   
68  2018-07-26 00:57:34.236681  0.91go4epu     1        [1]  1286845   
77  2018-07-26 00:58:03.969373  0.joh4hruw     1        [1]  1286845   
98  2018-07-26 00:58:50.759971  0.p1qx6123     1        [1]  1286845   
99  2018-07-26 00:58:51.210970  0.4hrhhctb     1        [1]  1286845   
100 2018-07-26 00:58:51.236033  0.4hrhhctb     1        [1]  1286845   
102 2018-07-26 00:58:55.966820  0.rncq0jn4     1        [1]  1286845   
103 2018-07-26 00:58:56.046833  0.rncq0jn4     1        [1]  1286845   
108 2018-07-26 00:59:00.605240  0.ayvblgjk     1        [1]  1286845   
109 2018-07-26 00:59:00.637345  0.ayvblgjk     1        [1]  1286845   
142 2018-07-26 01:00:50.586665  0.g3aog05i     1        [1]  1286845   
189 2018-07-26 01:03:53.504948  0.rnbygium     1        [1]  1286845   
229 2018-07-26 01:06:02.177337  0.1mgogsxu     1        [1]  1286845   
230 2018-07-26 01:06:07.629112  0.x83290ci     1        [1]  1286845   
399 2018-07-26 01:24:04.226865  0.1qlo48fi     1        [1]  1286845   
407 2018-07-26 01:25:27.976612  0.h2lpvzcq     1        [1]  1286845   
410 2018-07-26 01:25:53.868302  0.blkch70c     1        [1]  1286845   
415 2018-07-26 01:26:16.174124  0.z0e2gx0i     1        [1]  1286845   
416 2018-07-26 01:26:21.227184  0.z8ettr9s     1        [1]  1286845   
437 2018-07-26 01:28:45.474319  0.yu3764uq     1        [1]  1286845   
443 2018-07-26 01:29:32.933503  0.bcaf74eb     1        [1]  1286845   
457 2018-07-26 01:30:05.139422  0.ms8b8u2n     1        [1]  1286845   
459 2018-07-26 01:30:10.768175  0.yl7zcdzz     1        [1]  1286845   
521 2018-07-26 01:38:29.844541  0.a2gm2hhu     1        [1]  1286845   
522 2018-07-26 01:38:31.653428  0.a2gm2hhu     1        [1]  1286845   
565 2018-07-26 01:42:38.011429  0.wqr2p2ut     1        [1]  1286845   
566 2018-07-26 01:42:38.022094  0.wqr2p2ut     1        [1]  1286845   
590 2018-07-26 01:54:53.692398  0.itrb8d8p     1        [1]  1286845   
591 2018-07-26 01:54:57.352300  0.aqm79ers     1        [1]  1286845   
603 2018-07-26 01:58:28.367405  0.afb6cl09     1        [1]  1286845   
609 2018-07-26 01:59:26.791216  0.sus054a6     1        [1]  1286845   
610 2018-07-26 01:59:42.289318  0.ew3ogdm4     1        [1]  1286845   
612 2018-07-26 01:59:44.749753  0.kdf7wlgm     1        [1]  1286845   
616 2018-07-26 01:59:59.347242  0.k55g9f71     1        [1]  1286845   
641 2018-07-26 02:06:19.068064  0.qskkf0n3     1        [1]  1286845   
646 2018-07-26 02:07:19.760134  0.1vujdij4     1        [1]  1286845   
649 2018-07-26 02:07:30.943385  0.tck1qg51     1        [1]  1286845   
655 2018-07-26 02:08:13.858146  0.zcvoaf73     1        [1]  1286845   
664 2018-07-26 02:10:05.054644  0.0eno63a1     1        [1]  1286845   
665 2018-07-26 02:10:05.065815  0.0eno63a1     1        [1]  1286845   

              time                     times  worker_id  
5    1532568328538           [None, 0, 0, 0]   31931633  
6    1532566199281           [None, 0, 0, 0]   44403422  
18   1532566566250           [None, 0, 0, 0]   39718779  
19   1532566566266           [None, 0, 0, 0]   39718779  
21   1532566545745           [None, 0, 0, 0]   44331661  
23   1532566579088           [None, 0, 0, 0]   39718779  
24   1532566579098           [None, 0, 0, 0]   39718779  
25   1532566543775           [None, 0, 0, 0]   44074657  
46   1532568555139           [None, 0, 0, 0]   42914834  
57   1532568580779           [None, 0, 0, 0]   42914834  
64   1532568587148      [None, 27.111, 0, 0]   42914834  
65   1532568591203  [None, 27.111, 6.365, 0]   42914834  
68   1532566646839           [None, 0, 0, 0]   44074657  
77   1532566675136           [None, 0, 0, 0]   44464384  
98   1532566697289           [None, 0, 0, 0]   43744112  
99   1532568531174           [None, 0, 0, 0]   44103483  
100  1532568531199           [None, 0, 0, 0]   44103483  
102  1532568535991           [None, 0, 0, 0]   44103483  
103  1532568536018           [None, 0, 0, 0]   44103483  
108  1532568541197           [None, 0, 0, 0]   44103483  
109  1532568541213           [None, 0, 0, 0]   44103483  
142  1532566826344           [None, 0, 0, 0]   43744112  
189  1532567021201           [None, 0, 0, 0]   44464384  
229  1532569110618           [None, 0, 0, 0]   42914834  
230  1532569037477           [None, 0, 0, 0]   31931633  
399  1532589833925           [None, 0, 0, 0]   43989219  
407  1532568324755           [None, 0, 0, 0]   44284614  
410  1532568354491           [None, 0, 0, 0]   42787196  
415  1532568371323           [None, 0, 0, 0]   44056968  
416  1532568383005           [None, 0, 0, 0]   42787196  
437  1532568521443           [None, 0, 0, 0]   44056968  
443  1532568568817           [None, 0, 0, 0]   44482026  
457  1532568601161           [None, 0, 0, 0]   44482026  
459  1532568611727           [None, 0, 0, 0]   44331661  
521  1532569114133           [None, 0, 0, 0]   42074684  
522  1532569114153           [None, 0, 0, 0]   42074684  
565  1532569362590           [None, 0, 0, 0]   42074684  
566  1532569362608           [None, 0, 0, 0]   42074684  
590  1532569811103           [None, 0, 0, 0]   44403422  
591  1532569815999           [None, 0, 0, 0]   44403422  
603  1532570332825           [None, 0, 0, 0]   34457264  
609  1532572156422           [None, 0, 0, 0]   44111905  
610  1532572171951           [None, 0, 0, 0]   44111905  
612  1532570381005           [None, 0, 0, 0]   44284614  
616  1532570421966           [None, 0, 0, 0]   34457264  
641  1532570776319           [None, 0, 0, 0]   33453966  
646  1532570840320           [None, 0, 0, 0]   44011256  
649  1532592441566           [None, 0, 0, 0]   43989219  
655  1532570895159           [None, 0, 0, 0]   44011256  
664  1532571002226           [None, 0, 0, 0]   33453966  
665  1532571002241           [None, 0, 0, 0]   33453966  
                           judgments                      message  pay  \
0     [None, -1, -1, -1, -1, -1, -1]                        Start   30   
1     [None, -1, -1, -1, -1, -1, -1]  MW Worker Rejected:39718779   30   
3     [None, -1, -1, -1, -1, -1, -1]                        Start   30   
4     [None, -1, -1, -1, -1, -1, -1]  MW Worker Rejected:39404795   30   
6     [None, -1, -1, -1, -1, -1, -1]                        Start   30   
7     [None, -1, -1, -1, -1, -1, -1]                        Start   30   
8     [None, -1, -1, -1, -1, -1, -1]                        Start   30   
9     [None, -1, -1, -1, -1, -1, -1]  MW Worker Rejected:43702983   30   
16    [None, -1, -1, -1, -1, -1, -1]                        Start   30   
17    [None, -1, -1, -1, -1, -1, -1]                        Start   30   
20    [None, -1, -1, -1, -1, -1, -1]                        Start   30   
21     [None, 0, -1, -1, -1, -1, -1]                   nextButton   30   
23      [None, 0, 0, -1, -1, -1, -1]                   nextButton   30   
28    [None, -1, -1, -1, -1, -1, -1]                        Start   30   
39    [None, -1, -1, -1, -1, -1, -1]                        Start   30   
40    [None, -1, -1, -1, -1, -1, -1]  MW Worker Rejected:43978656   30   
69    [None, -1, -1, -1, -1, -1, -1]                        Start   30   
84    [None, -1, -1, -1, -1, -1, -1]                        Start   30   
87    [None, -1, -1, -1, -1, -1, -1]                        Start   30   
106   [None, -1, -1, -1, -1, -1, -1]                        Start   30   
129   [None, -1, -1, -1, -1, -1, -1]                        Start   30   
168   [None, -1, -1, -1, -1, -1, -1]                        Start   30   
169   [None, -1, -1, -1, -1, -1, -1]  MW Worker Rejected:28009209   30   
178   [None, -1, -1, -1, -1, -1, -1]                        Start   30   
179   [None, -1, -1, -1, -1, -1, -1]  MW Worker Rejected:44003641   30   
183   [None, -1, -1, -1, -1, -1, -1]  MW Worker Rejected:37101159   30   
184   [None, -1, -1, -1, -1, -1, -1]                        Start   30   
186   [None, -1, -1, -1, -1, -1, -1]                        Start   30   
187   [None, -1, -1, -1, -1, -1, -1]   MW Worker Rejected:6476374   30   
199   [None, -1, -1, -1, -1, -1, -1]                        Start   30   
...                              ...                          ...  ...   
698   [None, -1, -1, -1, -1, -1, -1]                        Start   30   
699   [None, -1, -1, -1, -1, -1, -1]                        Start   30   
701   [None, -1, -1, -1, -1, -1, -1]                        Start   30   
702   [None, -1, -1, -1, -1, -1, -1]  MW Worker Rejected:35722748   30   
704   [None, -1, -1, -1, -1, -1, -1]                        Start   30   
712   [None, -1, -1, -1, -1, -1, -1]                        Start   30   
867    [None, 0, -1, -1, -1, -1, -1]                   nextButton   30   
892   [None, -1, -1, -1, -1, -1, -1]                        Start   30   
904   [None, -1, -1, -1, -1, -1, -1]  MW Worker Rejected:18495435   30   
905   [None, -1, -1, -1, -1, -1, -1]                        Start   30   
911   [None, -1, -1, -1, -1, -1, -1]                        Start   30   
925   [None, -1, -1, -1, -1, -1, -1]                        Start   30   
927   [None, -1, -1, -1, -1, -1, -1]                        Start   30   
936   [None, -1, -1, -1, -1, -1, -1]                        Start   30   
937   [None, -1, -1, -1, -1, -1, -1]  MW Worker Rejected:33453966   30   
938   [None, -1, -1, -1, -1, -1, -1]                        Start   30   
955   [None, -1, -1, -1, -1, -1, -1]                        Start   30   
963   [None, -1, -1, -1, -1, -1, -1]                        Start   30   
990   [None, -1, -1, -1, -1, -1, -1]                        Start   30   
1038  [None, -1, -1, -1, -1, -1, -1]                        Start   30   
1039  [None, -1, -1, -1, -1, -1, -1]  MW Worker Rejected:44225018   30   
1040  [None, -1, -1, -1, -1, -1, -1]                        Start   30   
1070  [None, -1, -1, -1, -1, -1, -1]                        Start   30   
1072  [None, -1, -1, -1, -1, -1, -1]                        Start   30   
1073  [None, -1, -1, -1, -1, -1, -1]  MW Worker Rejected:38439462   30   
1088  [None, -1, -1, -1, -1, -1, -1]                        Start   30   
1089  [None, -1, -1, -1, -1, -1, -1]  MW Worker Rejected:44395725   30   
1090  [None, -1, -1, -1, -1, -1, -1]                        Start   30   
1107  [None, -1, -1, -1, -1, -1, -1]                        Start   30   
1119  [None, -1, -1, -1, -1, -1, -1]                        Start   30   

                    server_time  session_id  step   steps  task_id  \
0    2018-07-26 00:56:23.468218  0.9pvh44a2     1     [1]  1286857   
1    2018-07-26 00:56:23.877502  0.9pvh44a2     1     [1]  1286857   
3    2018-07-26 00:56:39.012594  0.vj53i7bh     1     [1]  1286857   
4    2018-07-26 00:56:39.058721  0.vj53i7bh     1     [1]  1286857   
6    2018-07-26 00:56:47.495885  0.5a5co902     1     [1]  1286857   
7    2018-07-26 00:56:50.513140  0.pnviue0j     1     [1]  1286857   
8    2018-07-26 00:57:00.337368  0.p1in1y83     1     [1]  1286857   
9    2018-07-26 00:57:00.552425  0.p1in1y83     1     [1]  1286857   
16   2018-07-26 00:57:31.574168  0.tb8mblbn     1     [1]  1286857   
17   2018-07-26 00:57:32.983979  0.7hgfbfsp     1     [1]  1286857   
20   2018-07-26 00:57:52.020695  0.nw2lyvql     1     [1]  1286857   
21   2018-07-26 00:57:53.098314  0.tb8mblbn     1     [1]  1286857   
23   2018-07-26 00:57:57.362887  0.tb8mblbn     2  [1, 2]  1286857   
28   2018-07-26 00:58:06.792362  0.6hzmfy8e     1     [1]  1286857   
39   2018-07-26 00:58:53.172930  0.zhouvxk7     1     [1]  1286857   
40   2018-07-26 00:58:53.384045  0.zhouvxk7     1     [1]  1286857   
69   2018-07-26 00:59:51.797652  0.i5q9toga     1     [1]  1286857   
84   2018-07-26 01:00:11.973500  0.sobj59p0     1     [1]  1286857   
87   2018-07-26 01:00:24.015034  0.oadhfuti     1     [1]  1286857   
106  2018-07-26 01:01:18.517557  0.uuy5belt     1     [1]  1286857   
129  2018-07-26 01:01:59.434842  0.nloaoivf     1     [1]  1286857   
168  2018-07-26 01:02:51.923794  0.pko97rej     1     [1]  1286857   
169  2018-07-26 01:02:52.149492  0.pko97rej     1     [1]  1286857   
178  2018-07-26 01:03:08.353859  0.qjct36zl     1     [1]  1286857   
179  2018-07-26 01:03:08.437629  0.qjct36zl     1     [1]  1286857   
183  2018-07-26 01:03:15.366523  0.dyj9hwz9     1     [1]  1286857   
184  2018-07-26 01:03:15.425938  0.dyj9hwz9     1     [1]  1286857   
186  2018-07-26 01:03:27.713376  0.5qqjfslq     1     [1]  1286857   
187  2018-07-26 01:03:28.016201  0.5qqjfslq     1     [1]  1286857   
199  2018-07-26 01:03:46.541814  0.rvc88ba2     1     [1]  1286857   
...                         ...         ...   ...     ...      ...   
698  2018-07-26 01:37:56.147008  0.oerxqaz9     1     [1]  1286857   
699  2018-07-26 01:37:57.871061  0.wraqio7e     1     [1]  1286857   
701  2018-07-26 01:37:59.463735  0.hmsgqpk5     1     [1]  1286857   
702  2018-07-26 01:37:59.708032  0.hmsgqpk5     1     [1]  1286857   
704  2018-07-26 01:38:01.244031  0.ifqrofrj     1     [1]  1286857   
712  2018-07-26 01:38:12.610875  0.rixy3h1a     1     [1]  1286857   
867  2018-07-26 01:51:33.475646  0.i9kbdaq2     1     [1]  1286857   
892  2018-07-26 01:58:18.505944  0.jtu3wr0a     1     [1]  1286857   
904  2018-07-26 01:59:25.151637  0.bhbf2i8s     1     [1]  1286857   
905  2018-07-26 01:59:25.203270  0.bhbf2i8s     1     [1]  1286857   
911  2018-07-26 02:01:06.172988  0.vrtfu8mx     1     [1]  1286857   
925  2018-07-26 02:02:40.696752  0.o5lnc2ze     1     [1]  1286857   
927  2018-07-26 02:02:50.619003  0.w5svbvsk     1     [1]  1286857   
936  2018-07-26 02:03:58.400747  0.h1lu9d7s     1     [1]  1286857   
937  2018-07-26 02:03:58.517923  0.h1lu9d7s     1     [1]  1286857   
938  2018-07-26 02:04:04.464288  0.nx8os8ij     1     [1]  1286857   
955  2018-07-26 02:05:29.119675  0.0udjx15q     1     [1]  1286857   
963  2018-07-26 02:06:23.610075  0.iklfncbn     1     [1]  1286857   
990  2018-07-26 02:08:36.694053  0.9u8uw7dv     1     [1]  1286857   
1038 2018-07-26 02:28:24.853622  0.ljt0vwkh     1     [1]  1286857   
1039 2018-07-26 02:28:24.979977  0.ljt0vwkh     1     [1]  1286857   
1040 2018-07-26 02:28:46.003915  0.0hr9qlcn     1     [1]  1286857   
1070 2018-07-26 02:38:21.104203  0.jkln6cza     1     [1]  1286857   
1072 2018-07-26 02:39:04.874327  0.qhextoha     1     [1]  1286857   
1073 2018-07-26 02:39:04.875078  0.qhextoha     1     [1]  1286857   
1088 2018-07-26 02:58:34.827394  0.jz5gxnnx     1     [1]  1286857   
1089 2018-07-26 02:58:35.326461  0.jz5gxnnx     1     [1]  1286857   
1090 2018-07-26 02:59:12.179987  0.js4xyzgr     1     [1]  1286857   
1107 2018-07-26 03:29:39.775113  0.yuyo19j7     1     [1]  1286857   
1119 2018-07-26 03:41:15.391353  0.iol1cwx8     1     [1]  1286857   

               time                         times  worker_id  
0     1532566615116      [None, 0, 0, 0, 0, 0, 0]   39718779  
1     1532566615135      [None, 0, 0, 0, 0, 0, 0]   39718779  
3     1532566596355      [None, 0, 0, 0, 0, 0, 0]   39404795  
4     1532566596373      [None, 0, 0, 0, 0, 0, 0]   39404795  
6     1532566599172      [None, 0, 0, 0, 0, 0, 0]   33056318  
7     1532566611461      [None, 0, 0, 0, 0, 0, 0]   44525804  
8     1532566619668      [None, 0, 0, 0, 0, 0, 0]   43702983  
9     1532566619681      [None, 0, 0, 0, 0, 0, 0]   43702983  
16    1532566648435      [None, 0, 0, 0, 0, 0, 0]   36769251  
17    1532566720782      [None, 0, 0, 0, 0, 0, 0]   43693574  
20    1532566668142      [None, 0, 0, 0, 0, 0, 0]   39920459  
21    1532566670939      [None, 0, 0, 0, 0, 0, 0]   36769251  
23    1532566675699  [None, 24.45, 0, 0, 0, 0, 0]   36769251  
28    1532566683982      [None, 0, 0, 0, 0, 0, 0]   36957773  
39    1532566730402      [None, 0, 0, 0, 0, 0, 0]   43978656  
40    1532566730416      [None, 0, 0, 0, 0, 0, 0]   43978656  
69    1532566790723      [None, 0, 0, 0, 0, 0, 0]   43876059  
84    1532566809173      [None, 0, 0, 0, 0, 0, 0]   42836146  
87    1532566878440      [None, 0, 0, 0, 0, 0, 0]   40928327  
106   1532566871777      [None, 0, 0, 0, 0, 0, 0]   44034041  
129   1532566913322      [None, 0, 0, 0, 0, 0, 0]   44041056  
168   1532566920861      [None, 0, 0, 0, 0, 0, 0]   28009209  
169   1532566920869      [None, 0, 0, 0, 0, 0, 0]   28009209  
178   1532566988978      [None, 0, 0, 0, 0, 0, 0]   44003641  
179   1532566988989      [None, 0, 0, 0, 0, 0, 0]   44003641  
183   1532566993466      [None, 0, 0, 0, 0, 0, 0]   37101159  
184   1532566993428      [None, 0, 0, 0, 0, 0, 0]   37101159  
186   1532566999530      [None, 0, 0, 0, 0, 0, 0]    6476374  
187   1532566999542      [None, 0, 0, 0, 0, 0, 0]    6476374  
199   1532567022918      [None, 0, 0, 0, 0, 0, 0]   39939687  
...             ...                           ...        ...  
698   1532569075249      [None, 0, 0, 0, 0, 0, 0]   42669204  
699   1532569075594      [None, 0, 0, 0, 0, 0, 0]   44633075  
701   1532569070254      [None, 0, 0, 0, 0, 0, 0]   35722748  
702   1532569070278      [None, 0, 0, 0, 0, 0, 0]   35722748  
704   1532569080352      [None, 0, 0, 0, 0, 0, 0]   44110677  
712   1532569086464      [None, 0, 0, 0, 0, 0, 0]   44044017  
867   1532569787037      [None, 0, 0, 0, 0, 0, 0]   38585736  
892   1532570296689      [None, 0, 0, 0, 0, 0, 0]   44258646  
904   1532570359170      [None, 0, 0, 0, 0, 0, 0]   18495435  
905   1532570359137      [None, 0, 0, 0, 0, 0, 0]   18495435  
911   1532570449719      [None, 0, 0, 0, 0, 0, 0]   44137269  
925   1532570551276      [None, 0, 0, 0, 0, 0, 0]   35174948  
927   1532570558580      [None, 0, 0, 0, 0, 0, 0]   39142745  
936   1532570635618      [None, 0, 0, 0, 0, 0, 0]   33453966  
937   1532570635641      [None, 0, 0, 0, 0, 0, 0]   33453966  
938   1532570642608      [None, 0, 0, 0, 0, 0, 0]   44220444  
955   1532570923697      [None, 0, 0, 0, 0, 0, 0]   44526082  
963   1532570781732      [None, 0, 0, 0, 0, 0, 0]   44295488  
990   1532570915745      [None, 0, 0, 0, 0, 0, 0]   31336490  
1038  1532572103263      [None, 0, 0, 0, 0, 0, 0]   44225018  
1039  1532572103274      [None, 0, 0, 0, 0, 0, 0]   44225018  
1040  1532572121590      [None, 0, 0, 0, 0, 0, 0]   44181201  
1070  1532572697443      [None, 0, 0, 0, 0, 0, 0]   44548069  
1072  1532572743438      [None, 0, 0, 0, 0, 0, 0]   38439462  
1073  1532572743455      [None, 0, 0, 0, 0, 0, 0]   38439462  
1088  1532573911509      [None, 0, 0, 0, 0, 0, 0]   44395725  
1089  1532573911549      [None, 0, 0, 0, 0, 0, 0]   44395725  
1090  1532573347921      [None, 0, 0, 0, 0, 0, 0]   44020940  
1107  1532575778405      [None, 0, 0, 0, 0, 0, 0]   35743341  
1119  1532576474415      [None, 0, 0, 0, 0, 0, 0]   44032553  

[118 rows x 11 columns]
                          judgments                      message  pay  \
10   [None, -1, -1, -1, -1, -1, -1]                        Start   30   
12   [None, -1, -1, -1, -1, -1, -1]                        Start   30   
14   [None, -1, -1, -1, -1, -1, -1]                        Start   30   
26   [None, -1, -1, -1, -1, -1, -1]                        Start   30   
27   [None, -1, -1, -1, -1, -1, -1]  MW Worker Rejected:44334102   30   
36   [None, -1, -1, -1, -1, -1, -1]                        Start   30   
37   [None, -1, -1, -1, -1, -1, -1]  MW Worker Rejected:44103483   30   
54   [None, -1, -1, -1, -1, -1, -1]                        Start   30   
55   [None, -1, -1, -1, -1, -1, -1]  MW Worker Rejected:44103483   30   
134  [None, -1, -1, -1, -1, -1, -1]                        Start   30   
135  [None, -1, -1, -1, -1, -1, -1]  MW Worker Rejected:44179475   30   
155  [None, -1, -1, -1, -1, -1, -1]                        Start   30   
156  [None, -1, -1, -1, -1, -1, -1]  MW Worker Rejected:44179475   30   
157  [None, -1, -1, -1, -1, -1, -1]                        Start   30   
158  [None, -1, -1, -1, -1, -1, -1]  MW Worker Rejected:44135638   30   
177  [None, -1, -1, -1, -1, -1, -1]                        Start   30   
254  [None, -1, -1, -1, -1, -1, -1]                        Start   30   
256  [None, -1, -1, -1, -1, -1, -1]  MW Worker Rejected:44204171   30   
266  [None, -1, -1, -1, -1, -1, -1]                        Start   30   
268  [None, -1, -1, -1, -1, -1, -1]  MW Worker Rejected:44204171   30   
279  [None, -1, -1, -1, -1, -1, -1]                        Start   30   
280  [None, -1, -1, -1, -1, -1, -1]  MW Worker Rejected:44204171   30   
396  [None, -1, -1, -1, -1, -1, -1]                        Start   30   
397  [None, -1, -1, -1, -1, -1, -1]  MW Worker Rejected:38968301   30   
416  [None, -1, -1, -1, -1, -1, -1]                        Start   30   
417  [None, -1, -1, -1, -1, -1, -1]  MW Worker Rejected:38968301   30   
460  [None, -1, -1, -1, -1, -1, -1]                        Start   30   
461  [None, -1, -1, -1, -1, -1, -1]  MW Worker Rejected:44334102   30   
548  [None, -1, -1, -1, -1, -1, -1]                        Start   30   
549  [None, -1, -1, -1, -1, -1, -1]  MW Worker Rejected:44361126   30   
552  [None, -1, -1, -1, -1, -1, -1]                        Start   30   
553  [None, -1, -1, -1, -1, -1, -1]  MW Worker Rejected:44361126   30   
554  [None, -1, -1, -1, -1, -1, -1]                        Start   30   
555  [None, -1, -1, -1, -1, -1, -1]  MW Worker Rejected:44361126   30   
556  [None, -1, -1, -1, -1, -1, -1]                        Start   30   
557  [None, -1, -1, -1, -1, -1, -1]                        Start   30   
558  [None, -1, -1, -1, -1, -1, -1]  MW Worker Rejected:44135638   30   
559  [None, -1, -1, -1, -1, -1, -1]  MW Worker Rejected:44361126   30   
560  [None, -1, -1, -1, -1, -1, -1]  MW Worker Rejected:44361126   30   
561  [None, -1, -1, -1, -1, -1, -1]                        Start   30   
571  [None, -1, -1, -1, -1, -1, -1]                        Start   30   
632  [None, -1, -1, -1, -1, -1, -1]                        Start   30   
641  [None, -1, -1, -1, -1, -1, -1]                        Start   30   
642  [None, -1, -1, -1, -1, -1, -1]  MW Worker Rejected:11001780   30   
659  [None, -1, -1, -1, -1, -1, -1]                        Start   30   
716  [None, -1, -1, -1, -1, -1, -1]                        Start   30   
717  [None, -1, -1, -1, -1, -1, -1]  MW Worker Rejected:42074684   30   
799  [None, -1, -1, -1, -1, -1, -1]                        Start   30   
800  [None, -1, -1, -1, -1, -1, -1]  MW Worker Rejected:42074684   30   
980  [None, -1, -1, -1, -1, -1, -1]  MW Worker Rejected:11001780   30   
981  [None, -1, -1, -1, -1, -1, -1]                        Start   30   

                   server_time  session_id  step steps  task_id  \
10  2018-07-26 00:57:02.807410  0.savrhlud     1   [1]  1286857   
12  2018-07-26 00:57:17.875092  0.500c7lt7     1   [1]  1286857   
14  2018-07-26 00:57:26.761145  0.z7vghwid     1   [1]  1286857   
26  2018-07-26 00:58:04.830216  0.h1qpk7b7     1   [1]  1286857   
27  2018-07-26 00:58:04.900047  0.h1qpk7b7     1   [1]  1286857   
36  2018-07-26 00:58:50.573708  0.cswje2ya     1   [1]  1286857   
37  2018-07-26 00:58:50.594633  0.cswje2ya     1   [1]  1286857   
54  2018-07-26 00:59:25.507527  0.v8nxihmx     1   [1]  1286857   
55  2018-07-26 00:59:25.636390  0.v8nxihmx     1   [1]  1286857   
134 2018-07-26 01:02:12.054903  0.ysfixt8o     1   [1]  1286857   
135 2018-07-26 01:02:12.334572  0.ysfixt8o     1   [1]  1286857   
155 2018-07-26 01:02:31.309206  0.s5asp1sn     1   [1]  1286857   
156 2018-07-26 01:02:31.399012  0.s5asp1sn     1   [1]  1286857   
157 2018-07-26 01:02:32.176881  0.46bj9nnt     1   [1]  1286857   
158 2018-07-26 01:02:32.377405  0.46bj9nnt     1   [1]  1286857   
177 2018-07-26 01:03:07.679506  0.8br95rkz     1   [1]  1286857   
254 2018-07-26 01:05:44.597324  0.0mjyvryt     1   [1]  1286857   
256 2018-07-26 01:05:44.908383  0.0mjyvryt     1   [1]  1286857   
266 2018-07-26 01:05:59.551779  0.qvfjngaz     1   [1]  1286857   
268 2018-07-26 01:05:59.830603  0.qvfjngaz     1   [1]  1286857   
279 2018-07-26 01:06:03.693186  0.eeutgts0     1   [1]  1286857   
280 2018-07-26 01:06:03.985545  0.eeutgts0     1   [1]  1286857   
396 2018-07-26 01:08:14.666297  0.3kgr1rta     1   [1]  1286857   
397 2018-07-26 01:08:15.274516  0.3kgr1rta     1   [1]  1286857   
416 2018-07-26 01:08:38.513094  0.7dn2goxo     1   [1]  1286857   
417 2018-07-26 01:08:38.526214  0.7dn2goxo     1   [1]  1286857   
460 2018-07-26 01:10:45.994166  0.bkvqq8vx     1   [1]  1286857   
461 2018-07-26 01:10:46.088457  0.bkvqq8vx     1   [1]  1286857   
548 2018-07-26 01:27:01.284615  0.3oj6rw1p     1   [1]  1286857   
549 2018-07-26 01:27:02.649240  0.3oj6rw1p     1   [1]  1286857   
552 2018-07-26 01:27:17.496301  0.gupnhx56     1   [1]  1286857   
553 2018-07-26 01:27:18.586150  0.gupnhx56     1   [1]  1286857   
554 2018-07-26 01:27:25.542168  0.ywwcrrjf     1   [1]  1286857   
555 2018-07-26 01:27:25.952087  0.ywwcrrjf     1   [1]  1286857   
556 2018-07-26 01:27:30.264062  0.hbxsaxsd     1   [1]  1286857   
557 2018-07-26 01:27:30.536571  0.yzoadyo0     1   [1]  1286857   
558 2018-07-26 01:27:30.806877  0.yzoadyo0     1   [1]  1286857   
559 2018-07-26 01:27:30.837087  0.hbxsaxsd     1   [1]  1286857   
560 2018-07-26 01:27:35.119747  0.jf28b06j     1   [1]  1286857   
561 2018-07-26 01:27:38.216141  0.jf28b06j     1   [1]  1286857   
571 2018-07-26 01:28:13.775582  0.gwbwfftp     1   [1]  1286857   
632 2018-07-26 01:34:59.161530  0.fpjj3vop     1   [1]  1286857   
641 2018-07-26 01:35:50.828913  0.h7pjop5c     1   [1]  1286857   
642 2018-07-26 01:35:50.828913  0.h7pjop5c     1   [1]  1286857   
659 2018-07-26 01:36:44.357583  0.a6dweblu     1   [1]  1286857   
716 2018-07-26 01:38:26.715565  0.qmzuass7     1   [1]  1286857   
717 2018-07-26 01:38:26.896473  0.qmzuass7     1   [1]  1286857   
799 2018-07-26 01:42:20.479395  0.rmtact52     1   [1]  1286857   
800 2018-07-26 01:42:20.952400  0.rmtact52     1   [1]  1286857   
980 2018-07-26 02:07:49.014847  0.g8fa8f3c     1   [1]  1286857   
981 2018-07-26 02:07:49.014847  0.g8fa8f3c     1   [1]  1286857   

              time                     times  worker_id  
10   1532566621175  [None, 0, 0, 0, 0, 0, 0]   43667456  
12   1532566636383  [None, 0, 0, 0, 0, 0, 0]   43667456  
14   1532566645277  [None, 0, 0, 0, 0, 0, 0]   43667456  
26   1532566680358  [None, 0, 0, 0, 0, 0, 0]   44334102  
27   1532566680388  [None, 0, 0, 0, 0, 0, 0]   44334102  
36   1532568530361  [None, 0, 0, 0, 0, 0, 0]   44103483  
37   1532568530380  [None, 0, 0, 0, 0, 0, 0]   44103483  
54   1532568565569  [None, 0, 0, 0, 0, 0, 0]   44103483  
55   1532568565591  [None, 0, 0, 0, 0, 0, 0]   44103483  
134  1532545301504  [None, 0, 0, 0, 0, 0, 0]   44179475  
135  1532545301637  [None, 0, 0, 0, 0, 0, 0]   44179475  
155  1532545323396  [None, 0, 0, 0, 0, 0, 0]   44179475  
156  1532545323496  [None, 0, 0, 0, 0, 0, 0]   44179475  
157  1532568708162  [None, 0, 0, 0, 0, 0, 0]   44135638  
158  1532568708190  [None, 0, 0, 0, 0, 0, 0]   44135638  
177  1532566985443  [None, 0, 0, 0, 0, 0, 0]   43181135  
254  1532567156714  [None, 0, 0, 0, 0, 0, 0]   44204171  
256  1532567156726  [None, 0, 0, 0, 0, 0, 0]   44204171  
266  1532567171511  [None, 0, 0, 0, 0, 0, 0]   44204171  
268  1532567171532  [None, 0, 0, 0, 0, 0, 0]   44204171  
279  1532567176477  [None, 0, 0, 0, 0, 0, 0]   44204171  
280  1532567176484  [None, 0, 0, 0, 0, 0, 0]   44204171  
396  1532567337582  [None, 0, 0, 0, 0, 0, 0]   38968301  
397  1532567337610  [None, 0, 0, 0, 0, 0, 0]   38968301  
416  1532567361642  [None, 0, 0, 0, 0, 0, 0]   38968301  
417  1532567361690  [None, 0, 0, 0, 0, 0, 0]   38968301  
460  1532567441780  [None, 0, 0, 0, 0, 0, 0]   44334102  
461  1532567441793  [None, 0, 0, 0, 0, 0, 0]   44334102  
548  1532568416627  [None, 0, 0, 0, 0, 0, 0]   44361126  
549  1532568416643  [None, 0, 0, 0, 0, 0, 0]   44361126  
552  1532568432481  [None, 0, 0, 0, 0, 0, 0]   44361126  
553  1532568432495  [None, 0, 0, 0, 0, 0, 0]   44361126  
554  1532568440705  [None, 0, 0, 0, 0, 0, 0]   44361126  
555  1532568440727  [None, 0, 0, 0, 0, 0, 0]   44361126  
556  1532568446188  [None, 0, 0, 0, 0, 0, 0]   44361126  
557  1532570204952  [None, 0, 0, 0, 0, 0, 0]   44135638  
558  1532570204978  [None, 0, 0, 0, 0, 0, 0]   44135638  
559  1532568446197  [None, 0, 0, 0, 0, 0, 0]   44361126  
560  1532568450251  [None, 0, 0, 0, 0, 0, 0]   44361126  
561  1532568450237  [None, 0, 0, 0, 0, 0, 0]   44361126  
571  1532568490503  [None, 0, 0, 0, 0, 0, 0]   43181135  
632  1532568893634  [None, 0, 0, 0, 0, 0, 0]   37957892  
641  1532568952885  [None, 0, 0, 0, 0, 0, 0]   11001780  
642  1532568952895  [None, 0, 0, 0, 0, 0, 0]   11001780  
659  1532568989011  [None, 0, 0, 0, 0, 0, 0]   37957892  
716  1532569111141  [None, 0, 0, 0, 0, 0, 0]   42074684  
717  1532569111158  [None, 0, 0, 0, 0, 0, 0]   42074684  
799  1532569345030  [None, 0, 0, 0, 0, 0, 0]   42074684  
800  1532569345057  [None, 0, 0, 0, 0, 0, 0]   42074684  
980  1532570871676  [None, 0, 0, 0, 0, 0, 0]   11001780  
981  1532570871669  [None, 0, 0, 0, 0, 0, 0]   11001780  
                           judgments     message  pay  \
0     [None, -1, -1, -1, -1, -1, -1]       Start   10   
1     [None, -1, -1, -1, -1, -1, -1]       Start   10   
2     [None, -1, -1, -1, -1, -1, -1]       Start   10   
3     [None, -1, -1, -1, -1, -1, -1]       Start   10   
4     [None, -1, -1, -1, -1, -1, -1]       Start   10   
6     [None, -1, -1, -1, -1, -1, -1]       Start   10   
7     [None, -1, -1, -1, -1, -1, -1]       Start   10   
9     [None, -1, -1, -1, -1, -1, -1]       Start   10   
12    [None, -1, -1, -1, -1, -1, -1]       Start   10   
13    [None, -1, -1, -1, -1, -1, -1]       Start   10   
15     [None, 1, -1, -1, -1, -1, -1]  nextButton   10   
21    [None, -1, -1, -1, -1, -1, -1]       Start   10   
31    [None, -1, -1, -1, -1, -1, -1]       Start   10   
32    [None, -1, -1, -1, -1, -1, -1]       Start   10   
35    [None, -1, -1, -1, -1, -1, -1]       Start   10   
38    [None, -1, -1, -1, -1, -1, -1]       Start   10   
44    [None, -1, -1, -1, -1, -1, -1]       Start   10   
57    [None, -1, -1, -1, -1, -1, -1]       Start   10   
59    [None, -1, -1, -1, -1, -1, -1]       Start   10   
60    [None, -1, -1, -1, -1, -1, -1]       Start   10   
75    [None, -1, -1, -1, -1, -1, -1]       Start   10   
86    [None, -1, -1, -1, -1, -1, -1]       Start   10   
95    [None, -1, -1, -1, -1, -1, -1]       Start   10   
97    [None, -1, -1, -1, -1, -1, -1]       Start   10   
98    [None, -1, -1, -1, -1, -1, -1]       Start   10   
105   [None, -1, -1, -1, -1, -1, -1]       Start   10   
108   [None, -1, -1, -1, -1, -1, -1]       Start   10   
120   [None, -1, -1, -1, -1, -1, -1]       Start   10   
123   [None, -1, -1, -1, -1, -1, -1]       Start   10   
135   [None, -1, -1, -1, -1, -1, -1]       Start   10   
...                              ...         ...  ...   
584   [None, -1, -1, -1, -1, -1, -1]       Start   10   
586   [None, -1, -1, -1, -1, -1, -1]       Start   10   
590   [None, -1, -1, -1, -1, -1, -1]       Start   10   
591   [None, -1, -1, -1, -1, -1, -1]       Start   10   
593   [None, -1, -1, -1, -1, -1, -1]       Start   10   
595   [None, -1, -1, -1, -1, -1, -1]       Start   10   
614   [None, -1, -1, -1, -1, -1, -1]       Start   10   
622   [None, -1, -1, -1, -1, -1, -1]       Start   10   
628   [None, -1, -1, -1, -1, -1, -1]       Start   10   
647   [None, -1, -1, -1, -1, -1, -1]       Start   10   
665   [None, -1, -1, -1, -1, -1, -1]       Start   10   
681   [None, -1, -1, -1, -1, -1, -1]       Start   10   
705   [None, -1, -1, -1, -1, -1, -1]       Start   10   
717   [None, -1, -1, -1, -1, -1, -1]       Start   10   
756   [None, -1, -1, -1, -1, -1, -1]       Start   10   
828   [None, -1, -1, -1, -1, -1, -1]       Start   10   
834   [None, -1, -1, -1, -1, -1, -1]       Start   10   
841   [None, -1, -1, -1, -1, -1, -1]       Start   10   
877   [None, -1, -1, -1, -1, -1, -1]       Start   10   
895   [None, -1, -1, -1, -1, -1, -1]       Start   10   
899   [None, -1, -1, -1, -1, -1, -1]       Start   10   
904   [None, -1, -1, -1, -1, -1, -1]       Start   10   
911   [None, -1, -1, -1, -1, -1, -1]       Start   10   
913    [None, 1, -1, -1, -1, -1, -1]  nextButton   10   
955   [None, -1, -1, -1, -1, -1, -1]       Start   10   
960   [None, -1, -1, -1, -1, -1, -1]       Start   10   
970   [None, -1, -1, -1, -1, -1, -1]       Start   10   
1028  [None, -1, -1, -1, -1, -1, -1]       Start   10   
1032  [None, -1, -1, -1, -1, -1, -1]       Start   10   
1060  [None, -1, -1, -1, -1, -1, -1]       Start   10   

                    server_time  session_id  step steps    task_id  \
0    2018-07-25 11:27:40.008932  0.f03mqh9l     1   [1]  1286626.0   
1    2018-07-25 11:28:02.781684  0.euhmtag5     1   [1]  1286626.0   
2    2018-07-25 11:28:09.870873  0.ir8afh08     1   [1]  1286626.0   
3    2018-07-25 11:28:13.668804  0.mlxv7rso     1   [1]  1286626.0   
4    2018-07-25 11:28:18.000698  0.mfy9iuvp     1   [1]  1286626.0   
6    2018-07-25 11:28:39.392138  0.5kqk57oo     1   [1]  1286626.0   
7    2018-07-25 11:28:40.951520  0.7ac8aeaj     1   [1]  1286626.0   
9    2018-07-25 11:28:44.592548  0.qgbg7ojf     1   [1]  1286626.0   
12   2018-07-25 11:29:00.241205  0.6pf06c7j     1   [1]  1286626.0   
13   2018-07-25 11:29:10.184443  0.ed7h5uxz     1   [1]  1286626.0   
15   2018-07-25 11:29:20.433313  0.euhmtag5     1   [1]  1286626.0   
21   2018-07-25 11:29:41.284045  0.n4hlf9oz     1   [1]  1286626.0   
31   2018-07-25 11:30:26.270202  0.eh7fp16u     1   [1]  1286626.0   
32   2018-07-25 11:30:31.929070  0.u8ouubys     1   [1]  1286626.0   
35   2018-07-25 11:30:42.550574  0.2de4z93h     1   [1]  1286626.0   
38   2018-07-25 11:30:45.913089  0.jwllm9al     1   [1]  1286626.0   
44   2018-07-25 11:30:57.496745  0.r9letw66     1   [1]  1286626.0   
57   2018-07-25 11:31:19.358011  0.3jbqinsr     1   [1]  1286626.0   
59   2018-07-25 11:31:22.721323  0.xvhq5cld     1   [1]  1286626.0   
60   2018-07-25 11:31:23.129925  0.9wxobcv0     1   [1]  1286626.0   
75   2018-07-25 11:31:54.350702  0.w57icwca     1   [1]  1286626.0   
86   2018-07-25 11:32:17.992922  0.qnwednq0     1   [1]  1286626.0   
95   2018-07-25 11:32:29.150461  0.yzc43vtg     1   [1]  1286626.0   
97   2018-07-25 11:32:36.078783  0.6rk7vjrj     1   [1]  1286626.0   
98   2018-07-25 11:32:40.449738  0.73nhheza     1   [1]  1286626.0   
105  2018-07-25 11:32:45.126817  0.loae89k3     1   [1]  1286626.0   
108  2018-07-25 11:32:56.751100  0.679hfp60     1   [1]  1286626.0   
120  2018-07-25 11:33:20.202637  0.uvbxcyp3     1   [1]  1286626.0   
123  2018-07-25 11:33:24.171808  0.pws0u296     1   [1]  1286626.0   
135  2018-07-25 11:33:42.897811  0.w2l8aqyg     1   [1]  1286626.0   
...                         ...         ...   ...   ...        ...   
584  2018-07-25 12:06:23.085660  0.x9m7ou9x     1   [1]  1286626.0   
586  2018-07-25 12:06:31.948556  0.favuee3k     1   [1]  1286626.0   
590  2018-07-25 12:06:56.321868  0.35uqw4qm     1   [1]  1286626.0   
591  2018-07-25 12:07:14.975039  0.et767wo8     1   [1]  1286626.0   
593  2018-07-25 12:07:17.098182  0.77ettpjs     1   [1]  1286626.0   
595  2018-07-25 12:07:22.628941  0.q4vgp8kq     1   [1]  1286626.0   
614  2018-07-25 12:08:20.742892  0.xkoe2enf     1   [1]  1286626.0   
622  2018-07-25 12:09:11.815471  0.w1u9j81d     1   [1]  1286626.0   
628  2018-07-25 12:09:52.726621  0.8wgb3uby     1   [1]  1286626.0   
647  2018-07-25 12:11:21.457932  0.z3oxp3h3     1   [1]  1286626.0   
665  2018-07-25 12:12:10.624448  0.61ypdozn     1   [1]  1286626.0   
681  2018-07-25 12:13:14.267538  0.4xdqzi1x     1   [1]  1286626.0   
705  2018-07-25 12:14:44.842988  0.cj0tr3lj     1   [1]  1286626.0   
717  2018-07-25 12:15:20.338195  0.bh81iz2b     1   [1]  1286626.0   
756  2018-07-25 12:18:40.149159  0.zaybk39v     1   [1]  1286626.0   
828  2018-07-25 12:29:21.074882  0.sfrdcm5z     1   [1]  1286626.0   
834  2018-07-25 12:29:51.298569  0.swvqldff     1   [1]  1286626.0   
841  2018-07-25 12:30:42.457023  0.wac4kzbf     1   [1]  1286626.0   
877  2018-07-25 12:33:55.929727  0.zuoo4379     1   [1]  1286626.0   
895  2018-07-25 12:36:10.471356  0.kkkogsvl     1   [1]  1286626.0   
899  2018-07-25 12:36:41.443344  0.y8mzf8rs     1   [1]  1286626.0   
904  2018-07-25 12:37:17.459536  0.o1uswa16     1   [1]  1286626.0   
911  2018-07-25 12:37:43.014972  0.esac9rl6     1   [1]  1286626.0   
913  2018-07-25 12:37:46.585723  0.o1uswa16     1   [1]  1286626.0   
955  2018-07-25 12:43:29.472051  0.7yt9b42g     1   [1]  1286626.0   
960  2018-07-25 12:44:02.492204  0.efijlr9w     1   [1]  1286626.0   
970  2018-07-25 12:45:27.428620  0.qx0s2632     1   [1]  1286626.0   
1028 2018-07-25 13:09:57.557492  0.nw32dgy4     1   [1]  1286626.0   
1032 2018-07-25 13:10:49.083104  0.agatd07z     1   [1]  1286626.0   
1060 2018-07-25 13:15:52.443402  0.u8hwfvhi     1   [1]  1286626.0   

               time                     times  worker_id  
0     1532518056947  [None, 0, 0, 0, 0, 0, 0]   44031296  
1     1532518078966  [None, 0, 0, 0, 0, 0, 0]   43415523  
2     1532518090184  [None, 0, 0, 0, 0, 0, 0]    6335115  
3     1532518080321  [None, 0, 0, 0, 0, 0, 0]   44643986  
4     1532518095276  [None, 0, 0, 0, 0, 0, 0]   42787196  
6     1532518118193  [None, 0, 0, 0, 0, 0, 0]   33256761  
7     1532518123569  [None, 0, 0, 0, 0, 0, 0]   35362876  
9     1532519962030  [None, 0, 0, 0, 0, 0, 0]   38575223  
12    1532518135209  [None, 0, 0, 0, 0, 0, 0]   44192534  
13    1532518143776  [None, 0, 0, 0, 0, 0, 0]   44234159  
15    1532518155242  [None, 0, 0, 0, 0, 0, 0]   43415523  
21    1532518160313  [None, 0, 0, 0, 0, 0, 0]   44123371  
31    1532518225328  [None, 0, 0, 0, 0, 0, 0]   34837948  
32    1532518223536  [None, 0, 0, 0, 0, 0, 0]   32671231  
35    1532514353601  [None, 0, 0, 0, 0, 0, 0]   43723980  
38    1532518244784  [None, 0, 0, 0, 0, 0, 0]   38021992  
44    1532518258070  [None, 0, 0, 0, 0, 0, 0]   44076544  
57    1532518275191  [None, 0, 0, 0, 0, 0, 0]   44568443  
59    1532518279807  [None, 0, 0, 0, 0, 0, 0]   44488763  
60    1532518282674  [None, 0, 0, 0, 0, 0, 0]   34320317  
75    1532518312258  [None, 0, 0, 0, 0, 0, 0]   44548295  
86    1532518332156  [None, 0, 0, 0, 0, 0, 0]   43799932  
95    1532518347952  [None, 0, 0, 0, 0, 0, 0]   43202278  
97    1532518354495  [None, 0, 0, 0, 0, 0, 0]   36163077  
98    1532518359213  [None, 0, 0, 0, 0, 0, 0]   40033214  
105   1532518363562  [None, 0, 0, 0, 0, 0, 0]   43911682  
108   1532518376017  [None, 0, 0, 0, 0, 0, 0]   40031413  
120   1532518398647  [None, 0, 0, 0, 0, 0, 0]   44208825  
123   1532518400660  [None, 0, 0, 0, 0, 0, 0]   44493113  
135   1532518419766  [None, 0, 0, 0, 0, 0, 0]   44654571  
...             ...                       ...        ...  
584   1532520381363  [None, 0, 0, 0, 0, 0, 0]   44277803  
586   1532520391542  [None, 0, 0, 0, 0, 0, 0]   44637880  
590   1532520415313  [None, 0, 0, 0, 0, 0, 0]   43823098  
591   1532520488096  [None, 0, 0, 0, 0, 0, 0]   33838842  
593   1532520439577  [None, 0, 0, 0, 0, 0, 0]   43716648  
595   1532516805162  [None, 0, 0, 0, 0, 0, 0]   44341922  
614   1532442865605  [None, 0, 0, 0, 0, 0, 0]   39475257  
622   1532520550413  [None, 0, 0, 0, 0, 0, 0]   44345631  
628   1532520588447  [None, 0, 0, 0, 0, 0, 0]   36242343  
647   1532520686246  [None, 0, 0, 0, 0, 0, 0]   42364062  
665   1532520730814  [None, 0, 0, 0, 0, 0, 0]   43752416  
681   1532520816578  [None, 0, 0, 0, 0, 0, 0]   43718452  
705   1532520882820  [None, 0, 0, 0, 0, 0, 0]   30128662  
717   1532527976928  [None, 0, 0, 0, 0, 0, 0]   44584177  
756   1532521115801  [None, 0, 0, 0, 0, 0, 0]   44526507  
828   1532521761346  [None, 0, 0, 0, 0, 0, 0]   44277308  
834   1532521788084  [None, 0, 0, 0, 0, 0, 0]   24282354  
841   1532521827278  [None, 0, 0, 0, 0, 0, 0]   43861639  
877   1532522033892  [None, 0, 0, 0, 0, 0, 0]   44293746  
895   1532522251891  [None, 0, 0, 0, 0, 0, 0]   44408094  
899   1532522207012  [None, 0, 0, 0, 0, 0, 0]   40949381  
904   1532522233969  [None, 0, 0, 0, 0, 0, 0]   37858570  
911   1532522261581  [None, 0, 0, 0, 0, 0, 0]   35354906  
913   1532522263406  [None, 0, 0, 0, 0, 0, 0]   37858570  
955   1532522616032  [None, 0, 0, 0, 0, 0, 0]   39740302  
960   1532522634898  [None, 0, 0, 0, 0, 0, 0]   44321127  
970   1532522725343  [None, 0, 0, 0, 0, 0, 0]   44489879  
1028  1532524192745  [None, 0, 0, 0, 0, 0, 0]   16826096  
1032  1532524249684  [None, 0, 0, 0, 0, 0, 0]   43850656  
1060  1532524549775  [None, 0, 0, 0, 0, 0, 0]   43881723  

[106 rows x 11 columns]
                          judgments message  pay                server_time  \
29   [None, -1, -1, -1, -1, -1, -1]   Start   10 2018-07-25 11:30:22.854003   
56   [None, -1, -1, -1, -1, -1, -1]   Start   10 2018-07-25 11:31:18.695495   
179  [None, -1, -1, -1, -1, -1, -1]   Start   10 2018-07-25 11:34:42.146185   
199  [None, -1, -1, -1, -1, -1, -1]   Start   10 2018-07-25 11:35:26.116800   
222  [None, -1, -1, -1, -1, -1, -1]   Start   10 2018-07-25 11:36:20.953012   
280  [None, -1, -1, -1, -1, -1, -1]   Start   10 2018-07-25 11:38:55.349382   
290  [None, -1, -1, -1, -1, -1, -1]   Start   10 2018-07-25 11:39:47.560676   
292  [None, -1, -1, -1, -1, -1, -1]   Start   10 2018-07-25 11:39:56.789048   
296  [None, -1, -1, -1, -1, -1, -1]   Start   10 2018-07-25 11:40:21.912651   
307  [None, -1, -1, -1, -1, -1, -1]   Start   10 2018-07-25 11:41:49.318127   
320  [None, -1, -1, -1, -1, -1, -1]   Start   10 2018-07-25 11:42:38.443237   
346  [None, -1, -1, -1, -1, -1, -1]   Start   10 2018-07-25 11:43:33.309884   
348  [None, -1, -1, -1, -1, -1, -1]   Start   10 2018-07-25 11:43:33.466957   
352  [None, -1, -1, -1, -1, -1, -1]   Start   10 2018-07-25 11:43:51.761975   
469  [None, -1, -1, -1, -1, -1, -1]   Start   10 2018-07-25 11:59:05.369987   
506  [None, -1, -1, -1, -1, -1, -1]   Start   10 2018-07-25 12:01:04.958886   
535  [None, -1, -1, -1, -1, -1, -1]   Start   10 2018-07-25 12:03:24.344568   
583  [None, -1, -1, -1, -1, -1, -1]   Start   10 2018-07-25 12:06:20.103895   

     session_id  step steps    task_id           time  \
29   0.qbpl57i5     1   [1]  1286626.0  1532518217322   
56   0.ixyib73m     1   [1]  1286626.0  1532518273473   
179  0.ka9qiumt     1   [1]  1286626.0  1532518477693   
199  0.7jbw7mqy     1   [1]  1286626.0  1532518525160   
222  0.v57p4k39     1   [1]  1286626.0  1532518577912   
280  0.52gygjph     1   [1]  1286626.0  1532518722699   
290  0.0bg134w8     1   [1]  1286626.0  1532518784213   
292  0.v0ro572o     1   [1]  1286626.0  1532518794752   
296  0.0614oh8s     1   [1]  1286626.0  1532518819906   
307  0.htg552fd     1   [1]  1286626.0  1532518632880   
320  0.w45fs0mk     1   [1]  1286626.0  1532518957637   
346  0.v0vzzxr9     1   [1]  1286626.0  1532519007868   
348  0.39srcqz3     1   [1]  1286626.0  1532518737205   
352  0.d0qes17y     1   [1]  1286626.0  1532519027935   
469  0.f90oqwgo     1   [1]  1286626.0  1532521731596   
506  0.h0qnhojg     1   [1]  1286626.0  1532521851341   
535  0.54lc9jww     1   [1]  1286626.0  1532520202761   
583  0.z1smpva8     1   [1]  1286626.0  1532520367463   

                        times  worker_id  
29   [None, 0, 0, 0, 0, 0, 0]   41705130  
56   [None, 0, 0, 0, 0, 0, 0]   41705130  
179  [None, 0, 0, 0, 0, 0, 0]   42310829  
199  [None, 0, 0, 0, 0, 0, 0]   43438576  
222  [None, 0, 0, 0, 0, 0, 0]   43721092  
280  [None, 0, 0, 0, 0, 0, 0]   44610353  
290  [None, 0, 0, 0, 0, 0, 0]   43721092  
292  [None, 0, 0, 0, 0, 0, 0]   43721092  
296  [None, 0, 0, 0, 0, 0, 0]   43721092  
307  [None, 0, 0, 0, 0, 0, 0]   30976232  
320  [None, 0, 0, 0, 0, 0, 0]   43438576  
346  [None, 0, 0, 0, 0, 0, 0]   43721092  
348  [None, 0, 0, 0, 0, 0, 0]   30976232  
352  [None, 0, 0, 0, 0, 0, 0]   42310829  
469  [None, 0, 0, 0, 0, 0, 0]   42982664  
506  [None, 0, 0, 0, 0, 0, 0]   42982664  
535  [None, 0, 0, 0, 0, 0, 0]   43721092  
583  [None, 0, 0, 0, 0, 0, 0]   44610353  
                           judgments     message  pay  \
0     [None, -1, -1, -1, -1, -1, -1]       Start   30   
1     [None, -1, -1, -1, -1, -1, -1]       Start   30   
2     [None, -1, -1, -1, -1, -1, -1]       Start   30   
4     [None, -1, -1, -1, -1, -1, -1]       Start   30   
5     [None, -1, -1, -1, -1, -1, -1]       Start   30   
6     [None, -1, -1, -1, -1, -1, -1]       Start   30   
7     [None, -1, -1, -1, -1, -1, -1]       Start   30   
8     [None, -1, -1, -1, -1, -1, -1]       Start   30   
9     [None, -1, -1, -1, -1, -1, -1]       Start   30   
13    [None, -1, -1, -1, -1, -1, -1]       Start   30   
16    [None, -1, -1, -1, -1, -1, -1]       Start   30   
19    [None, -1, -1, -1, -1, -1, -1]       Start   30   
23    [None, -1, -1, -1, -1, -1, -1]       Start   30   
26    [None, -1, -1, -1, -1, -1, -1]       Start   30   
28    [None, -1, -1, -1, -1, -1, -1]       Start   30   
29    [None, -1, -1, -1, -1, -1, -1]       Start   30   
31    [None, -1, -1, -1, -1, -1, -1]       Start   30   
34    [None, -1, -1, -1, -1, -1, -1]       Start   30   
37    [None, -1, -1, -1, -1, -1, -1]       Start   30   
40    [None, -1, -1, -1, -1, -1, -1]       Start   30   
49    [None, -1, -1, -1, -1, -1, -1]       Start   30   
68    [None, -1, -1, -1, -1, -1, -1]       Start   30   
99    [None, -1, -1, -1, -1, -1, -1]       Start   30   
103   [None, -1, -1, -1, -1, -1, -1]       Start   30   
105   [None, -1, -1, -1, -1, -1, -1]       Start   30   
122   [None, -1, -1, -1, -1, -1, -1]       Start   30   
127   [None, -1, -1, -1, -1, -1, -1]       Start   30   
149   [None, -1, -1, -1, -1, -1, -1]       Start   30   
161   [None, -1, -1, -1, -1, -1, -1]       Start   30   
169   [None, -1, -1, -1, -1, -1, -1]       Start   30   
...                              ...         ...  ...   
653   [None, -1, -1, -1, -1, -1, -1]       Start   30   
668   [None, -1, -1, -1, -1, -1, -1]       Start   30   
670   [None, -1, -1, -1, -1, -1, -1]       Start   30   
671   [None, -1, -1, -1, -1, -1, -1]       Start   30   
672   [None, -1, -1, -1, -1, -1, -1]       Start   30   
681    [None, 1, -1, -1, -1, -1, -1]  nextButton   30   
694     [None, 1, 0, -1, -1, -1, -1]  nextButton   30   
704   [None, -1, -1, -1, -1, -1, -1]       Start   30   
790   [None, -1, -1, -1, -1, -1, -1]       Start   30   
793   [None, -1, -1, -1, -1, -1, -1]       Start   30   
841   [None, -1, -1, -1, -1, -1, -1]       Start   30   
908      [None, 1, 0, 0, -1, -1, -1]  nextButton   30   
911       [None, 1, 0, 0, 1, -1, -1]  nextButton   30   
913        [None, 1, 0, 0, 1, 1, -1]  nextButton   30   
920         [None, 1, 0, 0, 1, 1, 2]  nextButton   30   
921         [None, 1, 0, 0, 1, 1, 2]    Final_OK   30   
922         [None, 1, 0, 0, 1, 1, 2]      paying   30   
943    [None, 0, -1, -1, -1, -1, -1]  nextButton   30   
954    [None, 0, -1, -1, -1, -1, -1]  backButton   30   
956    [None, 1, -1, -1, -1, -1, -1]  nextButton   30   
957     [None, 1, 0, -1, -1, -1, -1]  nextButton   30   
992   [None, -1, -1, -1, -1, -1, -1]       Start   30   
1016  [None, -1, -1, -1, -1, -1, -1]       Start   30   
1018  [None, -1, -1, -1, -1, -1, -1]       Start   30   
1027   [None, 1, -1, -1, -1, -1, -1]  nextButton   30   
1035  [None, -1, -1, -1, -1, -1, -1]       Start   30   
1038  [None, -1, -1, -1, -1, -1, -1]       Start   30   
1039  [None, -1, -1, -1, -1, -1, -1]       Start   30   
1060  [None, -1, -1, -1, -1, -1, -1]       Start   30   
1119  [None, -1, -1, -1, -1, -1, -1]       Start   30   

                    server_time  session_id  step                     steps  \
0    2018-07-25 11:38:40.945403  0.65r4pvwl     1                       [1]   
1    2018-07-25 11:38:47.104114  0.qb8js7xf     1                       [1]   
2    2018-07-25 11:38:49.715756  0.lg1lc3xo     1                       [1]   
4    2018-07-25 11:39:01.493515  0.j5fkne3k     1                       [1]   
5    2018-07-25 11:39:09.060431  0.imqqdpci     1                       [1]   
6    2018-07-25 11:39:14.169239  0.g3hf7rw5     1                       [1]   
7    2018-07-25 11:39:17.912350  0.raymam81     1                       [1]   
8    2018-07-25 11:39:18.014127  0.qipa5bs7     1                       [1]   
9    2018-07-25 11:39:27.717445  0.uz1yhbuy     1                       [1]   
13   2018-07-25 11:39:43.470387  0.77rbzfm7     1                       [1]   
16   2018-07-25 11:39:48.350790  0.gf59fqsg     1                       [1]   
19   2018-07-25 11:39:52.941367  0.ok5gtegg     1                       [1]   
23   2018-07-25 11:39:56.499427  0.g8behdz4     1                       [1]   
26   2018-07-25 11:39:59.682027  0.u5257z70     1                       [1]   
28   2018-07-25 11:40:08.889952  0.bm3y40m7     1                       [1]   
29   2018-07-25 11:40:09.289060  0.9l8z25mf     1                       [1]   
31   2018-07-25 11:40:16.774842  0.cotcmwhi     1                       [1]   
34   2018-07-25 11:40:21.260538  0.0khe805h     1                       [1]   
37   2018-07-25 11:40:28.707449  0.5oxtz177     1                       [1]   
40   2018-07-25 11:40:32.034748  0.3h1352xi     1                       [1]   
49   2018-07-25 11:40:50.614123  0.4wjrdlvd     1                       [1]   
68   2018-07-25 11:41:08.952410  0.3qosix2w     1                       [1]   
99   2018-07-25 11:41:54.492515  0.f797rtte     1                       [1]   
103  2018-07-25 11:41:58.093233  0.vpmqb8wf     1                       [1]   
105  2018-07-25 11:41:59.281123  0.vu8jsxfx     1                       [1]   
122  2018-07-25 11:42:17.304285  0.9mtz7zdi     1                       [1]   
127  2018-07-25 11:42:22.488727  0.7vt7ot96     1                       [1]   
149  2018-07-25 11:42:48.777193  0.w3wffirr     1                       [1]   
161  2018-07-25 11:43:02.054015  0.1oh6oya1     1                       [1]   
169  2018-07-25 11:43:10.616456  0.ffs34sy9     1                       [1]   
...                         ...         ...   ...                       ...   
653  2018-07-25 12:10:17.059397  0.t8zmdads     1                       [1]   
668  2018-07-25 12:11:39.682317  0.tonrwwbo     1                       [1]   
670  2018-07-25 12:11:47.752762  0.2x62azbr     1                       [1]   
671  2018-07-25 12:11:56.778445  0.aqto2u3w     1                       [1]   
672  2018-07-25 12:12:01.485241  0.lehqv5jn     1                       [1]   
681  2018-07-25 12:12:23.452726  0.tonrwwbo     1                       [1]   
694  2018-07-25 12:13:18.887998  0.tonrwwbo     2                    [1, 2]   
704  2018-07-25 12:13:52.486090  0.3plecxhq     1                       [1]   
790  2018-07-25 12:17:20.077602  0.jp42fx5k     1                       [1]   
793  2018-07-25 12:17:25.374280  0.dnwzvve6     1                       [1]   
841  2018-07-25 12:20:00.473570  0.pm8fk419     1                       [1]   
908  2018-07-25 12:24:33.224348  0.lnq7opds     3                 [1, 2, 3]   
911  2018-07-25 12:24:44.474504  0.lnq7opds     4              [1, 2, 3, 4]   
913  2018-07-25 12:24:48.629923  0.lnq7opds     5           [1, 2, 3, 4, 5]   
920  2018-07-25 12:25:07.244679  0.lnq7opds     6        [1, 2, 3, 4, 5, 6]   
921  2018-07-25 12:25:07.296219  0.lnq7opds     7     [1, 2, 3, 4, 5, 6, 7]   
922  2018-07-25 12:25:07.594441  0.lnq7opds     8  [1, 2, 3, 4, 5, 6, 7, 8]   
943  2018-07-25 12:26:31.597178  0.pm8fk419     1                       [1]   
954  2018-07-25 12:27:12.037464  0.pm8fk419     2                    [1, 2]   
956  2018-07-25 12:27:20.891837  0.pm8fk419     1                 [1, 2, 1]   
957  2018-07-25 12:27:24.939880  0.pm8fk419     2              [1, 2, 1, 2]   
992  2018-07-25 12:40:10.077365  0.9063m0vt     1                       [1]   
1016 2018-07-25 12:42:22.332658  0.uebnaq66     1                       [1]   
1018 2018-07-25 12:42:32.943722  0.i1a2shua     1                       [1]   
1027 2018-07-25 12:42:56.486706  0.9063m0vt     1                       [1]   
1035 2018-07-25 12:44:02.463077  0.jybceboh     1                       [1]   
1038 2018-07-25 12:45:11.272473  0.cpcypwsi     1                       [1]   
1039 2018-07-25 12:45:14.615651  0.kaksb563     1                       [1]   
1060 2018-07-25 12:49:08.365610  0.v2qldh2u     1                       [1]   
1119 2018-07-25 13:21:41.182510  0.dux1x7l0     1                       [1]   

        task_id           time  \
0     1286628.0  1532518716700   
1     1286628.0  1532518738893   
2     1286628.0  1532518810676   
4     1286628.0  1532518741131   
5     1286628.0  1532515106291   
6     1286628.0  1532518750157   
7     1286628.0  1532518758002   
8     1286628.0  1532518869621   
9     1286628.0  1532518766426   
13    1286628.0  1532520323975   
16    1286628.0  1532518788964   
19    1286628.0  1532518791624   
23    1286628.0  1532518794280   
26    1286628.0  1532518797778   
28    1286628.0  1532518820158   
29    1286628.0  1532518802184   
31    1286628.0  1532518814291   
34    1286628.0  1532518820129   
37    1286628.0  1532518827855   
40    1286628.0  1532518821779   
49    1286628.0  1532518846256   
68    1286628.0  1532518868312   
99    1286628.0  1532515025901   
103   1286628.0  1532518921597   
105   1286628.0  1532461257873   
122   1286628.0  1532518932767   
127   1286628.0  1532518924719   
149   1286628.0  1532518969625   
161   1286628.0  1532518979862   
169   1286628.0  1532518985822   
...         ...            ...   
653   1286628.0  1532520613414   
668   1286628.0  1532520692496   
670   1286628.0  1532520698566   
671   1286628.0  1532520738824   
672   1286628.0  1532527778179   
681   1286628.0  1532520736591   
694   1286628.0  1532520792018   
704   1286628.0  1532520833221   
790   1286628.0  1532521034802   
793   1286628.0  1532521046503   
841   1286628.0  1532521201274   
908   1286628.0  1532521466071   
911   1286628.0  1532521477559   
913   1286628.0  1532521481919   
920   1286628.0  1532521500310   
921   1286628.0  1532521500312   
922   1286628.0  1532521500314   
943   1286628.0  1532521592710   
954   1286628.0  1532521633130   
956   1286628.0  1532521641924   
957   1286628.0  1532521646686   
992   1286628.0  1532524157041   
1016  1286628.0  1532522539323   
1018  1286628.0  1532522543998   
1027  1286628.0  1532524323770   
1035  1286628.0  1532522634805   
1038  1286628.0  1532522732943   
1039  1286628.0  1532522956902   
1060  1286628.0  1532522946748   
1119  1286628.0  1532524895322   

                                                  times  worker_id  
0                              [None, 0, 0, 0, 0, 0, 0]   44569193  
1                              [None, 0, 0, 0, 0, 0, 0]   43935138  
2                              [None, 0, 0, 0, 0, 0, 0]   20225558  
4                              [None, 0, 0, 0, 0, 0, 0]   40031413  
5                              [None, 0, 0, 0, 0, 0, 0]   18397774  
6                              [None, 0, 0, 0, 0, 0, 0]   44030505  
7                              [None, 0, 0, 0, 0, 0, 0]   44329703  
8                              [None, 0, 0, 0, 0, 0, 0]   44149492  
9                              [None, 0, 0, 0, 0, 0, 0]   44066498  
13                             [None, 0, 0, 0, 0, 0, 0]   43589128  
16                             [None, 0, 0, 0, 0, 0, 0]   44076544  
19                             [None, 0, 0, 0, 0, 0, 0]   44470720  
23                             [None, 0, 0, 0, 0, 0, 0]   44286055  
26                             [None, 0, 0, 0, 0, 0, 0]   44661757  
28                             [None, 0, 0, 0, 0, 0, 0]   44353501  
29                             [None, 0, 0, 0, 0, 0, 0]   43809947  
31                             [None, 0, 0, 0, 0, 0, 0]   44252654  
34                             [None, 0, 0, 0, 0, 0, 0]   37735052  
37                             [None, 0, 0, 0, 0, 0, 0]   41618465  
40                             [None, 0, 0, 0, 0, 0, 0]   44232681  
49                             [None, 0, 0, 0, 0, 0, 0]   43633682  
68                             [None, 0, 0, 0, 0, 0, 0]   43438576  
99                             [None, 0, 0, 0, 0, 0, 0]   43723980  
103                            [None, 0, 0, 0, 0, 0, 0]   44269349  
105                            [None, 0, 0, 0, 0, 0, 0]   43818439  
122                            [None, 0, 0, 0, 0, 0, 0]   44330227  
127                            [None, 0, 0, 0, 0, 0, 0]   43606451  
149                            [None, 0, 0, 0, 0, 0, 0]   41704307  
161                            [None, 0, 0, 0, 0, 0, 0]   44689036  
169                            [None, 0, 0, 0, 0, 0, 0]   43650019  
...                                                 ...        ...  
653                            [None, 0, 0, 0, 0, 0, 0]   43062056  
668                            [None, 0, 0, 0, 0, 0, 0]   43781014  
670                            [None, 0, 0, 0, 0, 0, 0]   44643986  
671                            [None, 0, 0, 0, 0, 0, 0]   43718452  
672                            [None, 0, 0, 0, 0, 0, 0]   44584177  
681                            [None, 0, 0, 0, 0, 0, 0]   43781014  
694                       [None, 44.213, 0, 0, 0, 0, 0]   43781014  
704                            [None, 0, 0, 0, 0, 0, 0]   44647089  
790                            [None, 0, 0, 0, 0, 0, 0]   44277502  
793                            [None, 0, 0, 0, 0, 0, 0]   43548179  
841                            [None, 0, 0, 0, 0, 0, 0]   43688969  
908                 [None, 922.298, 68.333, 0, 0, 0, 0]   44395248  
911           [None, 922.298, 68.333, 1097.23, 0, 0, 0]   44395248  
913      [None, 922.298, 68.333, 1097.23, 11.486, 0, 0]   44395248  
920   [None, 922.298, 68.333, 1097.23, 11.486, 4.358...   44395248  
921   [None, 922.298, 68.333, 1097.23, 11.486, 4.358...   44395248  
922   [None, 922.298, 68.333, 1097.23, 11.486, 4.358...   44395248  
943                            [None, 0, 0, 0, 0, 0, 0]   43688969  
954                      [None, 393.197, 0, 0, 0, 0, 0]   43688969  
956                 [None, 393.197, 40.419, 0, 0, 0, 0]   43688969  
957                 [None, 401.989, 40.419, 0, 0, 0, 0]   43688969  
992                            [None, 0, 0, 0, 0, 0, 0]   44156891  
1016                           [None, 0, 0, 0, 0, 0, 0]   44493113  
1018                           [None, 0, 0, 0, 0, 0, 0]   39294026  
1027                           [None, 0, 0, 0, 0, 0, 0]   44156891  
1035                           [None, 0, 0, 0, 0, 0, 0]   44321127  
1038                           [None, 0, 0, 0, 0, 0, 0]   37780939  
1039                           [None, 0, 0, 0, 0, 0, 0]   43608691  
1060                           [None, 0, 0, 0, 0, 0, 0]   29812417  
1119                           [None, 0, 0, 0, 0, 0, 0]   44166244  

[94 rows x 11 columns]
                           judgments     message  pay  \
11    [None, -1, -1, -1, -1, -1, -1]       Start   30   
25    [None, -1, -1, -1, -1, -1, -1]       Start   30   
36    [None, -1, -1, -1, -1, -1, -1]       Start   30   
46    [None, -1, -1, -1, -1, -1, -1]       Start   30   
188   [None, -1, -1, -1, -1, -1, -1]       Start   30   
263   [None, -1, -1, -1, -1, -1, -1]       Start   30   
272   [None, -1, -1, -1, -1, -1, -1]       Start   30   
311    [None, 1, -1, -1, -1, -1, -1]  nextButton   30   
459   [None, -1, -1, -1, -1, -1, -1]       Start   30   
732   [None, -1, -1, -1, -1, -1, -1]       Start   30   
823   [None, -1, -1, -1, -1, -1, -1]       Start   30   
826   [None, -1, -1, -1, -1, -1, -1]       Start   30   
834   [None, -1, -1, -1, -1, -1, -1]       Start   30   
873    [None, 1, -1, -1, -1, -1, -1]  nextButton   30   
896     [None, 1, 0, -1, -1, -1, -1]  nextButton   30   
900      [None, 1, 0, 0, -1, -1, -1]  nextButton   30   
903       [None, 1, 0, 0, 0, -1, -1]  nextButton   30   
910        [None, 1, 0, 0, 0, 1, -1]  nextButton   30   
915         [None, 1, 0, 0, 0, 1, 0]  nextButton   30   
916         [None, 1, 0, 0, 0, 1, 0]      paying   30   
1145  [None, -1, -1, -1, -1, -1, -1]       Start   30   

                    server_time  session_id  step                     steps  \
11   2018-07-25 11:39:37.984898  0.x6pugh9j     1                       [1]   
25   2018-07-25 11:39:59.147616  0.cq1cnwjr     1                       [1]   
36   2018-07-25 11:40:24.818201  0.lv9igas8     1                       [1]   
46   2018-07-25 11:40:49.290275  0.baf84l1z     1                       [1]   
188  2018-07-25 11:43:32.578534  0.7j1etxt2     1                       [1]   
263  2018-07-25 11:45:04.789861  0.gzw8nb5h     1                       [1]   
272  2018-07-25 11:45:19.204205  0.yxjd5r8v     1                       [1]   
311  2018-07-25 11:46:27.767519  0.cq1cnwjr     1                       [1]   
459  2018-07-25 11:51:37.560208  0.a7tjwvx2     1                       [1]   
732  2018-07-25 12:14:53.174083  0.7zqji8at     1                       [1]   
823  2018-07-25 12:18:51.655396  0.ppmiy8m4     1                       [1]   
826  2018-07-25 12:18:53.703486  0.piy0sjdj     1                       [1]   
834  2018-07-25 12:19:00.467907  0.o3dogy9j     1                       [1]   
873  2018-07-25 12:22:11.604295  0.o3dogy9j     1                       [1]   
896  2018-07-25 12:23:15.801870  0.o3dogy9j     2                    [1, 2]   
900  2018-07-25 12:23:40.201952  0.o3dogy9j     3                 [1, 2, 3]   
903  2018-07-25 12:24:05.962186  0.o3dogy9j     4              [1, 2, 3, 4]   
910  2018-07-25 12:24:38.125953  0.o3dogy9j     5           [1, 2, 3, 4, 5]   
915  2018-07-25 12:25:00.190997  0.o3dogy9j     6        [1, 2, 3, 4, 5, 6]   
916  2018-07-25 12:25:00.224167  0.o3dogy9j     8  [1, 2, 3, 4, 5, 6, 7, 8]   
1145 2018-07-25 15:25:12.944230  0.xjwzpucd     1                       [1]   

        task_id           time  \
11    1286628.0  1532518775017   
25    1286628.0  1532518796216   
36    1286628.0  1532518820789   
46    1286628.0  1532518847290   
188   1286628.0  1532519006908   
263   1286628.0  1532519103341   
272   1286628.0  1532519118017   
311   1286628.0  1532519185835   
459   1286628.0  1532519494433   
732   1286628.0  1532520891547   
823   1286628.0  1532521126223   
826   1286628.0  1532521129133   
834   1286628.0  1532521135163   
873   1286628.0  1532521326319   
896   1286628.0  1532521390364   
900   1286628.0  1532521414872   
903   1286628.0  1532521440627   
910   1286628.0  1532521472851   
915   1286628.0  1532521494858   
916   1286628.0  1532521494867   
1145  1286628.0  1532532310428   

                                                  times  worker_id  
11                             [None, 0, 0, 0, 0, 0, 0]   37097770  
25                             [None, 0, 0, 0, 0, 0, 0]   37097770  
36                             [None, 0, 0, 0, 0, 0, 0]   43721092  
46                             [None, 0, 0, 0, 0, 0, 0]   43721092  
188                            [None, 0, 0, 0, 0, 0, 0]   43721092  
263                            [None, 0, 0, 0, 0, 0, 0]   30228652  
272                            [None, 0, 0, 0, 0, 0, 0]   30228652  
311                            [None, 0, 0, 0, 0, 0, 0]   37097770  
459                            [None, 0, 0, 0, 0, 0, 0]   43721092  
732                            [None, 0, 0, 0, 0, 0, 0]   44454884  
823                            [None, 0, 0, 0, 0, 0, 0]   43799932  
826                            [None, 0, 0, 0, 0, 0, 0]   43799932  
834                            [None, 0, 0, 0, 0, 0, 0]   43799932  
873                            [None, 0, 0, 0, 0, 0, 0]   43799932  
896                      [None, 191.241, 0, 0, 0, 0, 0]   43799932  
900                 [None, 191.241, 64.043, 0, 0, 0, 0]   43799932  
903            [None, 191.241, 64.043, 24.512, 0, 0, 0]   43799932  
910       [None, 191.241, 64.043, 24.512, 25.749, 0, 0]   43799932  
915   [None, 191.241, 64.043, 24.512, 25.749, 32.218...   43799932  
916   [None, 191.241, 64.043, 24.512, 25.749, 32.218...   43799932  
1145                           [None, 0, 0, 0, 0, 0, 0]   44454884  
sessionC_completA_1A = getSessionCount(completA_1A)
sessionC_completB_1A = getSessionCount(completB_1A)
sessionC_completA_1B = getSessionCount(completA_1B)
sessionC_completB_1B = getSessionCount(completB_1B)
sessionC_completA_2A = getSessionCount(completA_2A)
sessionC_completB_2A = getSessionCount(completB_2A)
sessionC_completA_2B = getSessionCount(completA_2B)
sessionC_completB_2B = getSessionCount(completB_2B)
count    88.0
mean      1.0
std       0.0
min       1.0
25%       1.0
50%       1.0
75%       1.0
max       1.0
dtype: float64
count    12.000000
mean      2.666667
std       1.230915
min       2.000000
25%       2.000000
50%       2.000000
75%       3.000000
max       6.000000
dtype: float64
count    83.0
mean      1.0
std       0.0
min       1.0
25%       1.0
50%       1.0
75%       1.0
max       1.0
dtype: float64
count    17.000000
mean      2.470588
std       0.874475
min       2.000000
25%       2.000000
50%       2.000000
75%       3.000000
max       5.000000
dtype: float64
count    89.0
mean      1.0
std       0.0
min       1.0
25%       1.0
50%       1.0
75%       1.0
max       1.0
dtype: float64
count    11.000000
mean      2.181818
std       0.404520
min       2.000000
25%       2.000000
50%       2.000000
75%       2.000000
max       3.000000
dtype: float64
count    84.0
mean      1.0
std       0.0
min       1.0
25%       1.0
50%       1.0
75%       1.0
max       1.0
dtype: float64
count    16.00000
mean      2.31250
std       0.60208
min       2.00000
25%       2.00000
50%       2.00000
75%       2.25000
max       4.00000
dtype: float64
'''The Shapiro.Wilk test needs n >= 20 and when splitting we get some of the groups with n under 20
norm_sessionC_abanA_1A = testNormality(sessionC_abanA_1A)
print("final: " + str(norm_sessionC_abanA_1A))
norm_sessionC_abanB_1A = testNormality(sessionC_abanB_1A)
print("final: " + str(norm_sessionC_abanB_1A))

norm_sessionC_abanA_1B = testNormality(sessionC_abanA_1B)
print("final: " + str(norm_sessionC_abanA_1B))
norm_sessionC_abanB_1B = testNormality(sessionC_abanB_1B)
print("final: " + str(norm_sessionC_abanB_1B ))

print('-------')

norm_sessionC_completA_1A = testNormality(sessionC_completA_1A)
print("final: " + str(norm_sessionC_completA_1A))
norm_sessionC_completB_1A = testNormality(sessionC_completB_1A)
print("final: " + str(norm_sessionC_completB_1A))
norm_sessionC_completA_1B = testNormality(sessionC_completA_1B)
print("final: " + str(norm_sessionC_completA_1B))
norm_sessionC_completB_1B = testNormality(sessionC_completB_1B)
print("final: " + str(norm_sessionC_completB_1B ))'''
# merge the two types of abandoned
sessionC_aban_1A = sessionC_abanA_1A.append(sessionC_abanB_1A,ignore_index=True)
sessionC_aban_1B = sessionC_abanA_1B.append(sessionC_abanB_1B,ignore_index=True)
sessionC_aban_2A = sessionC_abanA_2A.append(sessionC_abanB_2A,ignore_index=True)
sessionC_aban_2B = sessionC_abanA_2B.append(sessionC_abanB_2B,ignore_index=True)

# merge the two types of completed
sessionC_complet_1A = sessionC_completA_1A.append(sessionC_completB_1A,ignore_index=True)
sessionC_complet_1B = sessionC_completA_1B.append(sessionC_completB_1B,ignore_index=True)
sessionC_complet_2A = sessionC_completA_2A.append(sessionC_completB_2A,ignore_index=True)
sessionC_complet_2B = sessionC_completA_2B.append(sessionC_completB_2B,ignore_index=True)


norm_sessionC_aban_1A = testNormality(sessionC_aban_1A)
print("final: " + str(norm_sessionC_aban_1A))
norm_sessionC_aban_1B = testNormality(sessionC_aban_1B)
print("final: " + str(norm_sessionC_aban_1B))
norm_sessionC_aban_2A = testNormality(sessionC_aban_2A)
print("final: " + str(norm_sessionC_aban_2A))
norm_sessionC_aban_2B = testNormality(sessionC_aban_2B)
print("final: " + str(norm_sessionC_aban_2B))


norm_sessionC_complet_1A = testNormality(sessionC_complet_1A)
print("final: " + str(norm_sessionC_complet_1A))
norm_sessionC_complet_1B = testNormality(sessionC_complet_1B)
print("final: " + str(norm_sessionC_complet_1B))
norm_sessionC_complet_2A = testNormality(sessionC_complet_2A)
print("final: " + str(norm_sessionC_complet_2A))
norm_sessionC_complet_2B = testNormality(sessionC_complet_2B)
print("final: " + str(norm_sessionC_complet_2B))

print('\n')
print('-- TESTS for all abandoned and all completed --')

normal = norm_sessionC_aban_1A and norm_sessionC_aban_1B
print('Abandoned 1A and 1B')
compareTwoSamples(sessionC_aban_1A, sessionC_aban_1B, normal )

normal = norm_sessionC_complet_1A and norm_sessionC_compet_1B
print('Completed 1A and 1B')
compareTwoSamples(sessionC_complet_1A, sessionC_complet_1B, normal)

print('--')

normal = norm_sessionC_aban_2A and norm_sessionC_aban_2B
print('Abandoned 2A and 2B')
compareTwoSamples(sessionC_aban_2A, sessionC_aban_2B, normal)

normal = norm_sessionC_complet_2A and norm_sessionC_compet_2B
print('Completed 2A and 2B')
compareTwoSamples(sessionC_complet_2A, sessionC_complet_2B, normal)

print('--')
# Calling for all cases
 
 
 

Time

Compare per group, compare per experiment controlled.

# Functions to compute the measurements
log1A['times'].head(100)
def computeTime
 
# Calling for all cases
 
 
 

1. Hypothesis: BLABLABLA

H0: avg_a = avg_b H1: otherwise

 
 
 
 
 
 

Quality

Compare per group, compare per experiment controlled.

# Functions to compute the measurements
 
 
 
# Calling for all cases
 
 

Prediction task

TODO: Feature engineering

(non confidence determination)

  • Number of starts
  • Length of start-gone-start-...
  • Time between sessions on non completed work?

There are indeed several possible prediction tasks:

  • % of abandonment in the job (as they wrote in the paper but that is hard with only with one job, right?)
  • Classification of people abandoning person or not - try this