Reasoning
Planning
No coding guidance extracted.
Experiment objective: Load the dataset and generate summary statistics.
Steps for the programmer:
-
Load the dataset(s) at ['/workspace/datasets/user_uuid=d6b8184f-c2cb-45d1-a6e4-0b49820cfd64/thread_id=f675a76b-c116-44be-bb4c-a9cd43d13216/uploads/cox-violent-parsed.csv', '/workspace/datasets/user_uuid=d6b8184f-c2cb-45d1-a6e4-0b49820cfd64/thread_id=f675a76b-c116-44be-bb4c-a9cd43d13216/uploads/cox-parsed.csv', '/workspace/datasets/user_uuid=d6b8184f-c2cb-45d1-a6e4-0b49820cfd64/thread_id=f675a76b-c116-44be-bb4c-a9cd43d13216/uploads/compas-scores.csv', '/workspace/datasets/user_uuid=d6b8184f-c2cb-45d1-a6e4-0b49820cfd64/thread_id=f675a76b-c116-44be-bb4c-a9cd43d13216/uploads/compas-scores-two-years.csv', '/workspace/datasets/user_uuid=d6b8184f-c2cb-45d1-a6e4-0b49820cfd64/thread_id=f675a76b-c116-44be-bb4c-a9cd43d13216/uploads/compas-scores-two-years-violent.csv', '/workspace/datasets/user_uuid=d6b8184f-c2cb-45d1-a6e4-0b49820cfd64/thread_id=f675a76b-c116-44be-bb4c-a9cd43d13216/uploads/compas-scores-raw.csv'].
-
Generate summary statistics for the dataset(s).
Deliverables:
-
Dataset(s) loaded.
-
Summary statistics generated.
Programming
import pandas as pd
# Mapping of dataset names to file paths
file_paths = {
'cox_violent_parsed': '/workspace/datasets/user_uuid=d6b8184f-c2cb-45d1-a6e4-0b49820cfd64/thread_id=f675a76b-c116-44be-bb4c-a9cd43d13216/uploads/cox-violent-parsed.csv',
'cox_parsed': '/workspace/datasets/user_uuid=d6b8184f-c2cb-45d1-a6e4-0b49820cfd64/thread_id=f675a76b-c116-44be-bb4c-a9cd43d13216/uploads/cox-parsed.csv',
'compas_scores': '/workspace/datasets/user_uuid=d6b8184f-c2cb-45d1-a6e4-0b49820cfd64/thread_id=f675a76b-c116-44be-bb4c-a9cd43d13216/uploads/compas-scores.csv',
'compas_scores_two_years': '/workspace/datasets/user_uuid=d6b8184f-c2cb-45d1-a6e4-0b49820cfd64/thread_id=f675a76b-c116-44be-bb4c-a9cd43d13216/uploads/compas-scores-two-years.csv',
'compas_scores_two_years_violent': '/workspace/datasets/user_uuid=d6b8184f-c2cb-45d1-a6e4-0b49820cfd64/thread_id=f675a76b-c116-44be-bb4c-a9cd43d13216/uploads/compas-scores-two-years-violent.csv',
'compas_scores_raw': '/workspace/datasets/user_uuid=d6b8184f-c2cb-45d1-a6e4-0b49820cfd64/thread_id=f675a76b-c116-44be-bb4c-a9cd43d13216/uploads/compas-scores-raw.csv'
}
# Load datasets and generate summary statistics
for name, path in file_paths.items():
print(f"\n=== Dataset: {name} ===")
try:
df = pd.read_csv(path)
except Exception as e:
print(f"Error loading {name}: {e}")
continue
# Basic info
print(f"Shape: {df.shape}")
print("Column dtypes:")
print(df.dtypes)
# Descriptive statistics for numeric and categorical columns
print("\nDescriptive statistics (numeric):")
print(df.describe().T)
print("\nDescriptive statistics (all columns):")
print(df.describe(include='all', datetime_is_numeric=True).T)
exitcode: 1 (execution failed)
Code output:
=== Dataset: cox_violent_parsed ===
Shape: (18316, 52)
Column dtypes:
id int64
name str
first str
last str
compas_screening_date str
sex str
dob str
age int64
age_cat str
race str
juv_fel_count int64
decile_score int64
juv_misd_count int64
juv_other_count int64
priors_count int64
days_b_screening_arrest float64
c_jail_in str
c_jail_out str
c_case_number str
c_offense_date str
c_arrest_date str
c_days_from_compas float64
c_charge_degree str
c_charge_desc str
is_recid int64
r_case_number str
r_charge_degree str
r_days_from_arrest float64
r_offense_date str
r_charge_desc str
r_jail_in str
r_jail_out str
violent_recid float64
is_violent_recid int64
vr_case_number str
vr_charge_degree str
vr_offense_date str
vr_charge_desc str
type_of_assessment str
decile_score.1 int64
score_text str
screening_date str
v_type_of_assessment str
v_decile_score int64
v_score_text str
v_screening_date str
in_custody str
out_custody str
priors_count.1 int64
start int64
end int64
event int64
dtype: object
Descriptive statistics (numeric):
count mean std min 25% \
id 18316.0 5492.908932 3165.256572 1.0 2772.0
age 18316.0 34.019273 11.667811 18.0 25.0
juv_fel_count 18316.0 0.076873 0.464272 0.0 0.0
decile_score 18316.0 4.997052 2.937569 -1.0 2.0
juv_misd_count 18316.0 0.098439 0.517639 0.0 0.0
juv_other_count 18316.0 0.124263 0.524537 0.0 0.0
priors_count 18316.0 3.913191 5.299864 0.0 0.0
days_b_screening_arrest 17019.0 4.302485 81.159881 -597.0 -1.0
c_days_from_compas 17449.0 57.694596 317.994694 0.0 1.0
is_recid 18316.0 0.414774 0.576449 -1.0 0.0
r_days_from_arrest 6359.0 19.961629 74.139695 -1.0 0.0
violent_recid 0.0 NaN NaN NaN NaN
is_violent_recid 18316.0 0.073105 0.260317 0.0 0.0
decile_score.1 18316.0 4.997052 2.937569 -1.0 2.0
v_decile_score 18316.0 4.022822 2.614189 -1.0 2.0
priors_count.1 18316.0 3.913191 5.299864 0.0 0.0
start 18316.0 186.394901 283.373004 0.0 0.0
end 18316.0 625.210636 335.881439 0.0 364.0
event 18316.0 0.044715 0.206683 0.0 0.0
50% 75% max
id 5489.0 8205.25 11001.0
age 31.0 41.00 96.0
juv_fel_count 0.0 0.00 20.0
decile_score 5.0 8.00 10.0
juv_misd_count 0.0 0.00 13.0
juv_other_count 0.0 0.00 17.0
priors_count 2.0 5.00 43.0
days_b_screening_arrest -1.0 0.00 1057.0
c_days_from_compas 1.0 2.00 9485.0
is_recid 0.0 1.00 1.0
r_days_from_arrest 0.0 1.00 993.0
violent_recid NaN NaN NaN
is_violent_recid 0.0 0.00 1.0
decile_score.1 5.0 8.00 10.0
v_decile_score 4.0 6.00 10.0
priors_count.1 2.0 5.00 43.0
start 7.0 309.00 1197.0
end 634.0 897.00 1187.0
event 0.0 0.00 1.0
Descriptive statistics (all columns):
ERROR: NDFrame.describe() got an unexpected keyword argument 'datetime_is_numeric'
Analyzing
The script successfully loaded the first dataset (cox_violent_parsed), displayed its shape, column types, and numeric summary statistics. However, the call to df.describe(include='all', datetime_is_numeric=True) failed with “unexpected keyword argument 'datetime_is_numeric',” causing the run to abort before processing the remaining files. To complete the experiment, remove the unsupported parameter or upgrade your pandas version.
Programming
import pandas as pd
# Mapping of dataset names to file paths
file_paths = {
'cox_violent_parsed': '/workspace/datasets/user_uuid=d6b8184f-c2cb-45d1-a6e4-0b49820cfd64/thread_id=f675a76b-c116-44be-bb4c-a9cd43d13216/uploads/cox-violent-parsed.csv',
'cox_parsed': '/workspace/datasets/user_uuid=d6b8184f-c2cb-45d1-a6e4-0b49820cfd64/thread_id=f675a76b-c116-44be-bb4c-a9cd43d13216/uploads/cox-parsed.csv',
'compas_scores': '/workspace/datasets/user_uuid=d6b8184f-c2cb-45d1-a6e4-0b49820cfd64/thread_id=f675a76b-c116-44be-bb4c-a9cd43d13216/uploads/compas-scores.csv',
'compas_scores_two_years': '/workspace/datasets/user_uuid=d6b8184f-c2cb-45d1-a6e4-0b49820cfd64/thread_id=f675a76b-c116-44be-bb4c-a9cd43d13216/uploads/compas-scores-two-years.csv',
'compas_scores_two_years_violent': '/workspace/datasets/user_uuid=d6b8184f-c2cb-45d1-a6e4-0b49820cfd64/thread_id=f675a76b-c116-44be-bb4c-a9cd43d13216/uploads/compas-scores-two-years-violent.csv',
'compas_scores_raw': '/workspace/datasets/user_uuid=d6b8184f-c2cb-45d1-a6e4-0b49820cfd64/thread_id=f675a76b-c116-44be-bb4c-a9cd43d13216/uploads/compas-scores-raw.csv'
}
for name, path in file_paths.items():
print(f"\n=== Dataset: {name} ===")
try:
df = pd.read_csv(path)
except Exception as e:
print(f"Error loading {name}: {e}")
continue
# Basic info
print(f"Shape: {df.shape}")
print("Column dtypes:")
print(df.dtypes)
# Numeric summary
print("\nNumeric summary statistics:")
print(df.describe().T)
# All columns summary
print("\nAll-columns summary statistics:")
print(df.describe(include='all').T)
exitcode: 0 (execution succeeded)
Code output:
=== Dataset: cox_violent_parsed ===
Shape: (18316, 52)
Column dtypes:
id int64
name str
first str
last str
compas_screening_date str
sex str
dob str
age int64
age_cat str
race str
juv_fel_count int64
decile_score int64
juv_misd_count int64
juv_other_count int64
priors_count int64
days_b_screening_arrest float64
c_jail_in str
c_jail_out str
c_case_number str
c_offense_date str
c_arrest_date str
c_days_from_compas float64
c_charge_degree str
c_charge_desc str
is_recid int64
r_case_number str
r_charge_degree str
r_days_from_arrest float64
r_offense_date str
r_charge_desc str
r_jail_in str
r_jail_out str
violent_recid float64
is_violent_recid int64
vr_case_number str
vr_charge_degree str
vr_offense_date str
vr_charge_desc str
type_of_assessment str
decile_score.1 int64
score_text str
screening_date str
v_type_of_assessment str
v_decile_score int64
v_score_text str
v_screening_date str
in_custody str
out_custody str
priors_count.1 int64
start int64
end int64
event int64
dtype: object
Numeric summary statistics:
count mean std min 25% \
id 18316.0 5492.908932 3165.256572 1.0 2772.0
age 18316.0 34.019273 11.667811 18.0 25.0
juv_fel_count 18316.0 0.076873 0.464272 0.0 0.0
decile_score 18316.0 4.997052 2.937569 -1.0 2.0
juv_misd_count 18316.0 0.098439 0.517639 0.0 0.0
juv_other_count 18316.0 0.124263 0.524537 0.0 0.0
priors_count 18316.0 3.913191 5.299864 0.0 0.0
days_b_screening_arrest 17019.0 4.302485 81.159881 -597.0 -1.0
c_days_from_compas 17449.0 57.694596 317.994694 0.0 1.0
is_recid 18316.0 0.414774 0.576449 -1.0 0.0
r_days_from_arrest 6359.0 19.961629 74.139695 -1.0 0.0
violent_recid 0.0 NaN NaN NaN NaN
is_violent_recid 18316.0 0.073105 0.260317 0.0 0.0
decile_score.1 18316.0 4.997052 2.937569 -1.0 2.0
v_decile_score 18316.0 4.022822 2.614189 -1.0 2.0
priors_count.1 18316.0 3.913191 5.299864 0.0 0.0
start 18316.0 186.394901 283.373004 0.0 0.0
end 18316.0 625.210636 335.881439 0.0 364.0
event 18316.0 0.044715 0.206683 0.0 0.0
50% 75% max
id 5489.0 8205.25 11001.0
age 31.0 41.00 96.0
juv_fel_count 0.0 0.00 20.0
decile_score 5.0 8.00 10.0
juv_misd_count 0.0 0.00 13.0
juv_other_count 0.0 0.00 17.0
priors_count 2.0 5.00 43.0
days_b_screening_arrest -1.0 0.00 1057.0
c_days_from_compas 1.0 2.00 9485.0
is_recid 0.0 1.00 1.0
r_days_from_arrest 0.0 1.00 993.0
violent_recid NaN NaN NaN
is_violent_recid 0.0 0.00 1.0
decile_score.1 5.0 8.00 10.0
v_decile_score 4.0 6.00 10.0
priors_count.1 2.0 5.00 43.0
start 7.0 309.00 1197.0
end 634.0 897.00 1187.0
event 0.0 0.00 1.0
All-columns summary statistics:
count unique top \
id 18316.0 NaN NaN
name 18316 10855 brandon ross
first 18316 3876 michael
last 18316 5635 williams
compas_screening_date 18316 703 2013-03-20
sex 18316 2 Male
dob 18316 7485 1991-11-21
age 18316.0 NaN NaN
age_cat 18316 3 25 - 45
race 18316 6 African-American
juv_fel_count 18316.0 NaN NaN
decile_score 18316.0 NaN NaN
juv_misd_count 18316.0 NaN NaN
juv_other_count 18316.0 NaN NaN
priors_count 18316.0 NaN NaN
days_b_screening_arrest 17019.0 NaN NaN
c_jail_in 17019 9903 2013-02-04 07:24:28
c_jail_out 17019 9854 2013-03-15 08:19:29
c_case_number 17449 10309 13001740CF10A
c_offense_date 14364 1025 2013-02-08
c_arrest_date 3085 782 2013-01-11
c_days_from_compas 17449.0 NaN NaN
c_charge_degree 17449 14 (F3)
c_charge_desc 17435 513 arrest case no charge
is_recid 18316.0 NaN NaN
r_case_number 8417 3473 13007175MM10A
r_charge_degree 8417 10 (M1)
r_days_from_arrest 6359.0 NaN NaN
r_offense_date 8417 1075 2014-10-30
r_charge_desc 8277 341 Possess Cannabis/20 Grams Or Less
r_jail_in 6359 972 2014-04-29
r_jail_out 6359 938 2014-02-18
violent_recid 0.0 NaN NaN
is_violent_recid 18316.0 NaN NaN
vr_case_number 1339 819 15001565MM10A
vr_charge_degree 1339 9 (M1)
vr_offense_date 1339 570 2015-01-06
vr_charge_desc 1339 83 Battery
type_of_assessment 18316 1 Risk of Recidivism
decile_score.1 18316.0 NaN NaN
score_text 18293 3 Low
screening_date 18316 703 2013-03-20
v_type_of_assessment 18316 1 Risk of Violence
v_decile_score 18316.0 NaN NaN
v_score_text 18310 3 Low
v_screening_date 18316 703 2013-03-20
in_custody 17722 1237 2014-06-05
out_custody 17722 1253 2020-01-01
priors_count.1 18316.0 NaN NaN
start 18316.0 NaN NaN
end 18316.0 NaN NaN
event 18316.0 NaN NaN
freq mean std min 25% \
id NaN 5492.908932 3165.256572 1.0 2772.0
name 13 NaN NaN NaN NaN
first 421 NaN NaN NaN NaN
last 239 NaN NaN NaN NaN
compas_screening_date 72 NaN NaN NaN NaN
sex 14933 NaN NaN NaN NaN
dob 21 NaN NaN NaN NaN
age NaN 34.019273 11.667811 18.0 25.0
age_cat 10372 NaN NaN NaN NaN
race 9791 NaN NaN NaN NaN
juv_fel_count NaN 0.076873 0.464272 0.0 0.0
decile_score NaN 4.997052 2.937569 -1.0 2.0
juv_misd_count NaN 0.098439 0.517639 0.0 0.0
juv_other_count NaN 0.124263 0.524537 0.0 0.0
priors_count NaN 3.913191 5.299864 0.0 0.0
days_b_screening_arrest NaN 4.302485 81.159881 -597.0 -1.0
c_jail_in 13 NaN NaN NaN NaN
c_jail_out 13 NaN NaN NaN NaN
c_case_number 13 NaN NaN NaN NaN
c_offense_date 57 NaN NaN NaN NaN
c_arrest_date 27 NaN NaN NaN NaN
c_days_from_compas NaN 57.694596 317.994694 0.0 1.0
c_charge_degree 10294 NaN NaN NaN NaN
c_charge_desc 3085 NaN NaN NaN NaN
is_recid NaN 0.414774 0.576449 -1.0 0.0
r_case_number 13 NaN NaN NaN NaN
r_charge_degree 2922 NaN NaN NaN NaN
r_days_from_arrest NaN 19.961629 74.139695 -1.0 0.0
r_offense_date 33 NaN NaN NaN NaN
r_charge_desc 635 NaN NaN NaN NaN
r_jail_in 29 NaN NaN NaN NaN
r_jail_out 28 NaN NaN NaN NaN
violent_recid NaN NaN NaN NaN NaN
is_violent_recid NaN 0.073105 0.260317 0.0 0.0
vr_case_number 11 NaN NaN NaN NaN
vr_charge_degree 550 NaN NaN NaN NaN
vr_offense_date 14 NaN NaN NaN NaN
vr_charge_desc 529 NaN NaN NaN NaN
type_of_assessment 18316 NaN NaN NaN NaN
decile_score.1 NaN 4.997052 2.937569 -1.0 2.0
score_text 8597 NaN NaN NaN NaN
screening_date 72 NaN NaN NaN NaN
v_type_of_assessment 18316 NaN NaN NaN NaN
v_decile_score NaN 4.022822 2.614189 -1.0 2.0
v_score_text 11147 NaN NaN NaN NaN
v_screening_date 72 NaN NaN NaN NaN
in_custody 45 NaN NaN NaN NaN
out_custody 648 NaN NaN NaN NaN
priors_count.1 NaN 3.913191 5.299864 0.0 0.0
start NaN 186.394901 283.373004 0.0 0.0
end NaN 625.210636 335.881439 0.0 364.0
event NaN 0.044715 0.206683 0.0 0.0
50% 75% max
id 5489.0 8205.25 11001.0
name NaN NaN NaN
first NaN NaN NaN
last NaN NaN NaN
compas_screening_date NaN NaN NaN
sex NaN NaN NaN
dob NaN NaN NaN
age 31.0 41.0 96.0
age_cat NaN NaN NaN
race NaN NaN NaN
juv_fel_count 0.0 0.0 20.0
decile_score 5.0 8.0 10.0
juv_misd_count 0.0 0.0 13.0
juv_other_count 0.0 0.0 17.0
priors_count 2.0 5.0 43.0
days_b_screening_arrest -1.0 0.0 1057.0
c_jail_in NaN NaN NaN
c_jail_out NaN NaN NaN
c_case_number NaN NaN NaN
c_offense_date NaN NaN NaN
c_arrest_date NaN NaN NaN
c_days_from_compas 1.0 2.0 9485.0
c_charge_degree NaN NaN NaN
c_charge_desc NaN NaN NaN
is_recid 0.0 1.0 1.0
r_case_number NaN NaN NaN
r_charge_degree NaN NaN NaN
r_days_from_arrest 0.0 1.0 993.0
r_offense_date NaN NaN NaN
r_charge_desc NaN NaN NaN
r_jail_in NaN NaN NaN
r_jail_out NaN NaN NaN
violent_recid NaN NaN NaN
is_violent_recid 0.0 0.0 1.0
vr_case_number NaN NaN NaN
vr_charge_degree NaN NaN NaN
vr_offense_date NaN NaN NaN
vr_charge_desc NaN NaN NaN
type_of_assessment NaN NaN NaN
decile_score.1 5.0 8.0 10.0
score_text NaN NaN NaN
screening_date NaN NaN NaN
v_type_of_assessment NaN NaN NaN
v_decile_score 4.0 6.0 10.0
v_score_text NaN NaN NaN
v_screening_date NaN NaN NaN
in_custody NaN NaN NaN
out_custody NaN NaN NaN
priors_count.1 2.0 5.0 43.0
start 7.0 309.0 1197.0
end 634.0 897.0 1187.0
event 0.0 0.0 1.0
=== Dataset: cox_parsed ===
Shape: (13419, 52)
Column dtypes:
id int64
name str
first str
last str
compas_screening_date str
sex str
dob str
age int64
age_cat str
race str
juv_fel_count int64
decile_score int64
juv_misd_count int64
juv_other_count int64
priors_count int64
days_b_screening_arrest float64
c_jail_in str
c_jail_out str
c_case_number str
c_offense_date str
c_arrest_date str
c_days_from_compas float64
c_charge_degree str
c_charge_desc str
is_recid int64
r_case_number str
r_charge_degree str
r_days_from_arrest float64
r_offense_date str
r_charge_desc str
r_jail_in str
r_jail_out str
violent_recid float64
is_violent_recid int64
vr_case_number str
vr_charge_degree str
vr_offense_date str
vr_charge_desc str
type_of_assessment str
decile_score.1 int64
score_text str
screening_date str
v_type_of_assessment str
v_decile_score int64
v_score_text str
v_screening_date str
in_custody str
out_custody str
priors_count.1 int64
start int64
end int64
event int64
dtype: object
Numeric summary statistics:
count mean std min 25% \
id 13419.0 5492.285714 3168.021547 1.0 2769.5
age 13419.0 34.618153 11.833707 18.0 25.0
juv_fel_count 13419.0 0.071168 0.456568 0.0 0.0
decile_score 13419.0 4.692451 2.909933 -1.0 2.0
juv_misd_count 13419.0 0.087190 0.475043 0.0 0.0
juv_other_count 13419.0 0.109919 0.509185 0.0 0.0
priors_count 13419.0 3.558536 4.908545 0.0 0.0
days_b_screening_arrest 12958.0 2.720636 80.089131 -597.0 -1.0
c_days_from_compas 13388.0 68.180684 354.077422 0.0 1.0
is_recid 13419.0 0.323422 0.467800 0.0 0.0
r_days_from_arrest 2951.0 18.526940 71.562237 -1.0 0.0
violent_recid 0.0 NaN NaN NaN NaN
is_violent_recid 13419.0 0.074298 0.262264 0.0 0.0
decile_score.1 13419.0 4.692451 2.909933 -1.0 2.0
v_decile_score 13419.0 3.798644 2.567786 -1.0 1.0
priors_count.1 13419.0 3.558536 4.908545 0.0 0.0
start 13419.0 108.344959 227.248751 0.0 0.0
end 13419.0 568.629033 348.820277 0.0 249.0
event 13419.0 0.259110 0.438163 0.0 0.0
50% 75% max
id 5466.0 8230.5 11001.0
age 31.0 42.0 96.0
juv_fel_count 0.0 0.0 20.0
decile_score 4.0 7.0 10.0
juv_misd_count 0.0 0.0 13.0
juv_other_count 0.0 0.0 17.0
priors_count 2.0 5.0 43.0
days_b_screening_arrest -1.0 -1.0 1057.0
c_days_from_compas 1.0 3.0 9485.0
is_recid 0.0 1.0 1.0
r_days_from_arrest 0.0 1.0 993.0
violent_recid NaN NaN NaN
is_violent_recid 0.0 0.0 1.0
decile_score.1 4.0 7.0 10.0
v_decile_score 3.0 6.0 10.0
priors_count.1 2.0 5.0 43.0
start 1.0 76.0 1185.0
end 568.0 849.0 1186.0
event 0.0 1.0 1.0
All-columns summary statistics:
count unique top \
id 13419.0 NaN NaN
name 13419 10217 lauren perry
first 13419 3669 michael
last 13419 5331 williams
compas_screening_date 13419 703 2013-03-20
sex 13419 2 Male
dob 13419 7164 1991-03-23
age 13419.0 NaN NaN
age_cat 13419 3 25 - 45
race 13419 6 African-American
juv_fel_count 13419.0 NaN NaN
decile_score 13419.0 NaN NaN
juv_misd_count 13419.0 NaN NaN
juv_other_count 13419.0 NaN NaN
priors_count 13419.0 NaN NaN
days_b_screening_arrest 12958.0 NaN NaN
c_jail_in 12958 9902 2013-01-25 01:47:43
c_jail_out 12958 9853 2014-02-12 10:41:00
c_case_number 13388 10308 13001264CF10A
c_offense_date 10924 1025 2013-01-09
c_arrest_date 2464 782 2013-01-15
c_days_from_compas 13388.0 NaN NaN
c_charge_degree 13388 14 (F3)
c_charge_desc 13376 513 arrest case no charge
is_recid 13419.0 NaN NaN
r_case_number 4340 3473 15003386CF10A
r_charge_degree 4340 10 (M1)
r_days_from_arrest 2951.0 NaN NaN
r_offense_date 4340 1075 2014-12-08
r_charge_desc 4248 341 Possess Cannabis/20 Grams Or Less
r_jail_in 2951 972 2014-07-10
r_jail_out 2951 938 2015-05-15
violent_recid 0.0 NaN NaN
is_violent_recid 13419.0 NaN NaN
vr_case_number 997 819 16000353CF10A
vr_charge_degree 997 9 (M1)
vr_offense_date 997 570 2015-08-15
vr_charge_desc 997 83 Battery
type_of_assessment 13419 1 Risk of Recidivism
decile_score.1 13419.0 NaN NaN
score_text 13407 3 Low
screening_date 13419 703 2013-03-20
v_type_of_assessment 13419 1 Risk of Violence
v_decile_score 13419.0 NaN NaN
v_score_text 13414 3 Low
v_screening_date 13419 703 2013-03-20
in_custody 13093 1225 2014-09-17
out_custody 13093 1233 2020-01-01
priors_count.1 13419.0 NaN NaN
start 13419.0 NaN NaN
end 13419.0 NaN NaN
event 13419.0 NaN NaN
freq mean std min 25% \
id NaN 5492.285714 3168.021547 1.0 2769.5
name 9 NaN NaN NaN NaN
first 294 NaN NaN NaN NaN
last 155 NaN NaN NaN NaN
compas_screening_date 46 NaN NaN NaN NaN
sex 10781 NaN NaN NaN NaN
dob 13 NaN NaN NaN NaN
age NaN 34.618153 11.833707 18.0 25.0
age_cat 7641 NaN NaN NaN NaN
race 6899 NaN NaN NaN NaN
juv_fel_count NaN 0.071168 0.456568 0.0 0.0
decile_score NaN 4.692451 2.909933 -1.0 2.0
juv_misd_count NaN 0.08719 0.475043 0.0 0.0
juv_other_count NaN 0.109919 0.509185 0.0 0.0
priors_count NaN 3.558536 4.908545 0.0 0.0
days_b_screening_arrest NaN 2.720636 80.089131 -597.0 -1.0
c_jail_in 9 NaN NaN NaN NaN
c_jail_out 10 NaN NaN NaN NaN
c_case_number 9 NaN NaN NaN NaN
c_offense_date 40 NaN NaN NaN NaN
c_arrest_date 19 NaN NaN NaN NaN
c_days_from_compas NaN 68.180684 354.077422 0.0 1.0
c_charge_degree 7582 NaN NaN NaN NaN
c_charge_desc 2464 NaN NaN NaN NaN
is_recid NaN 0.323422 0.4678 0.0 0.0
r_case_number 6 NaN NaN NaN NaN
r_charge_degree 1530 NaN NaN NaN NaN
r_days_from_arrest NaN 18.52694 71.562237 -1.0 0.0
r_offense_date 16 NaN NaN NaN NaN
r_charge_desc 313 NaN NaN NaN NaN
r_jail_in 13 NaN NaN NaN NaN
r_jail_out 12 NaN NaN NaN NaN
violent_recid NaN NaN NaN NaN NaN
is_violent_recid NaN 0.074298 0.262264 0.0 0.0
vr_case_number 4 NaN NaN NaN NaN
vr_charge_degree 408 NaN NaN NaN NaN
vr_offense_date 7 NaN NaN NaN NaN
vr_charge_desc 390 NaN NaN NaN NaN
type_of_assessment 13419 NaN NaN NaN NaN
decile_score.1 NaN 4.692451 2.909933 -1.0 2.0
score_text 6908 NaN NaN NaN NaN
screening_date 46 NaN NaN NaN NaN
v_type_of_assessment 13419 NaN NaN NaN NaN
v_decile_score NaN 3.798644 2.567786 -1.0 1.0
v_score_text 8671 NaN NaN NaN NaN
v_screening_date 46 NaN NaN NaN NaN
in_custody 37 NaN NaN NaN NaN
out_custody 402 NaN NaN NaN NaN
priors_count.1 NaN 3.558536 4.908545 0.0 0.0
start NaN 108.344959 227.248751 0.0 0.0
end NaN 568.629033 348.820277 0.0 249.0
event NaN 0.25911 0.438163 0.0 0.0
50% 75% max
id 5466.0 8230.5 11001.0
name NaN NaN NaN
first NaN NaN NaN
last NaN NaN NaN
compas_screening_date NaN NaN NaN
sex NaN NaN NaN
dob NaN NaN NaN
age 31.0 42.0 96.0
age_cat NaN NaN NaN
race NaN NaN NaN
juv_fel_count 0.0 0.0 20.0
decile_score 4.0 7.0 10.0
juv_misd_count 0.0 0.0 13.0
juv_other_count 0.0 0.0 17.0
priors_count 2.0 5.0 43.0
days_b_screening_arrest -1.0 -1.0 1057.0
c_jail_in NaN NaN NaN
c_jail_out NaN NaN NaN
c_case_number NaN NaN NaN
c_offense_date NaN NaN NaN
c_arrest_date NaN NaN NaN
c_days_from_compas 1.0 3.0 9485.0
c_charge_degree NaN NaN NaN
c_charge_desc NaN NaN NaN
is_recid 0.0 1.0 1.0
r_case_number NaN NaN NaN
r_charge_degree NaN NaN NaN
r_days_from_arrest 0.0 1.0 993.0
r_offense_date NaN NaN NaN
r_charge_desc NaN NaN NaN
r_jail_in NaN NaN NaN
r_jail_out NaN NaN NaN
violent_recid NaN NaN NaN
is_violent_recid 0.0 0.0 1.0
vr_case_number NaN NaN NaN
vr_charge_degree NaN NaN NaN
vr_offense_date NaN NaN NaN
vr_charge_desc NaN NaN NaN
type_of_assessment NaN NaN NaN
decile_score.1 4.0 7.0 10.0
score_text NaN NaN NaN
screening_date NaN NaN NaN
v_type_of_assessment NaN NaN NaN
v_decile_score 3.0 6.0 10.0
v_score_text NaN NaN NaN
v_screening_date NaN NaN NaN
in_custody NaN NaN NaN
out_custody NaN NaN NaN
priors_count.1 2.0 5.0 43.0
start 1.0 76.0 1185.0
end 568.0 849.0 1186.0
event 0.0 1.0 1.0
=== Dataset: compas_scores ===
Shape: (11757, 47)
Column dtypes:
id int64
name str
first str
last str
compas_screening_date str
sex str
dob str
age int64
age_cat str
race str
juv_fel_count int64
decile_score int64
juv_misd_count int64
juv_other_count int64
priors_count int64
days_b_screening_arrest float64
c_jail_in str
c_jail_out str
c_case_number str
c_offense_date str
c_arrest_date str
c_days_from_compas float64
c_charge_degree str
c_charge_desc str
is_recid int64
num_r_cases float64
r_case_number str
r_charge_degree str
r_days_from_arrest float64
r_offense_date str
r_charge_desc str
r_jail_in str
r_jail_out str
is_violent_recid int64
num_vr_cases float64
vr_case_number str
vr_charge_degree str
vr_offense_date str
vr_charge_desc str
v_type_of_assessment str
v_decile_score int64
v_score_text str
v_screening_date str
type_of_assessment str
decile_score.1 int64
score_text str
screening_date str
dtype: object
Numeric summary statistics:
count mean std min 25% \
id 11757.0 5879.000000 3394.097892 1.0 2940.0
age 11757.0 35.143319 12.022894 18.0 25.0
juv_fel_count 11757.0 0.061580 0.445328 0.0 0.0
decile_score 11757.0 4.371268 2.877598 -1.0 2.0
juv_misd_count 11757.0 0.076040 0.449757 0.0 0.0
juv_other_count 11757.0 0.093561 0.472003 0.0 0.0
priors_count 11757.0 3.082164 4.687410 0.0 0.0
days_b_screening_arrest 10577.0 -0.878037 72.889298 -597.0 -1.0
c_days_from_compas 11015.0 63.587653 341.899711 0.0 1.0
is_recid 11757.0 0.253806 0.558324 -1.0 0.0
num_r_cases 0.0 NaN NaN NaN NaN
r_days_from_arrest 2460.0 20.410569 74.354840 -1.0 0.0
is_violent_recid 11757.0 0.075019 0.263433 0.0 0.0
num_vr_cases 0.0 NaN NaN NaN NaN
v_decile_score 11757.0 3.571489 2.500479 -1.0 1.0
decile_score.1 11757.0 4.371268 2.877598 -1.0 2.0
50% 75% max
id 5879.0 8818.0 11757.0
age 32.0 43.0 96.0
juv_fel_count 0.0 0.0 20.0
decile_score 4.0 7.0 10.0
juv_misd_count 0.0 0.0 13.0
juv_other_count 0.0 0.0 17.0
priors_count 1.0 4.0 43.0
days_b_screening_arrest -1.0 -1.0 1057.0
c_days_from_compas 1.0 2.0 9485.0
is_recid 0.0 1.0 1.0
num_r_cases NaN NaN NaN
r_days_from_arrest 0.0 1.0 993.0
is_violent_recid 0.0 0.0 1.0
num_vr_cases NaN NaN NaN
v_decile_score 3.0 5.0 10.0
decile_score.1 4.0 7.0 10.0
All-columns summary statistics:
count unique top freq \
id 11757.0 NaN NaN NaN
name 11757 11584 michael cunningham 4
first 11757 4058 michael 264
last 11757 5921 williams 145
compas_screening_date 11757 704 2013-03-20 39
sex 11757 2 Male 9336
dob 11757 7800 1989-04-27 6
age 11757.0 NaN NaN NaN
age_cat 11757 3 25 - 45 6649
race 11757 6 African-American 5813
juv_fel_count 11757.0 NaN NaN NaN
decile_score 11757.0 NaN NaN NaN
juv_misd_count 11757.0 NaN NaN NaN
juv_other_count 11757.0 NaN NaN NaN
priors_count 11757.0 NaN NaN NaN
days_b_screening_arrest 10577.0 NaN NaN NaN
c_jail_in 10577 10577 2013-08-13 06:03:42 1
c_jail_out 10577 10517 2014-02-12 10:41:00 4
c_case_number 11015 11015 13011352CF10A 1
c_offense_date 9157 1036 2013-03-20 29
c_arrest_date 1858 802 2013-05-15 9
c_days_from_compas 11015.0 NaN NaN NaN
c_charge_degree 11757 3 F 7232
c_charge_desc 11008 531 arrest case no charge 1858
is_recid 11757.0 NaN NaN NaN
num_r_cases 0.0 NaN NaN NaN
r_case_number 3703 3703 13009779CF10A 1
r_charge_degree 11757 3 O 8056
r_days_from_arrest 2460.0 NaN NaN NaN
r_offense_date 3703 1090 2014-12-08 12
r_charge_desc 3643 352 Driving License Suspended 279
r_jail_in 2460 984 2014-05-27 9
r_jail_out 2460 953 2014-02-18 10
is_violent_recid 11757.0 NaN NaN NaN
num_vr_cases 0.0 NaN NaN NaN
vr_case_number 882 882 13009779CF10A 1
vr_charge_degree 882 9 (M1) 372
vr_offense_date 882 599 2015-08-15 6
vr_charge_desc 882 88 Battery 356
v_type_of_assessment 11757 1 Risk of Violence 11757
v_decile_score 11757.0 NaN NaN NaN
v_score_text 11752 3 Low 7968
v_screening_date 11757 704 2013-03-20 39
type_of_assessment 11757 1 Risk of Recidivism 11757
decile_score.1 11757.0 NaN NaN NaN
score_text 11742 3 Low 6607
screening_date 11757 704 2013-03-20 39
mean std min 25% 50% \
id 5879.0 3394.097892 1.0 2940.0 5879.0
name NaN NaN NaN NaN NaN
first NaN NaN NaN NaN NaN
last NaN NaN NaN NaN NaN
compas_screening_date NaN NaN NaN NaN NaN
sex NaN NaN NaN NaN NaN
dob NaN NaN NaN NaN NaN
age 35.143319 12.022894 18.0 25.0 32.0
age_cat NaN NaN NaN NaN NaN
race NaN NaN NaN NaN NaN
juv_fel_count 0.06158 0.445328 0.0 0.0 0.0
decile_score 4.371268 2.877598 -1.0 2.0 4.0
juv_misd_count 0.07604 0.449757 0.0 0.0 0.0
juv_other_count 0.093561 0.472003 0.0 0.0 0.0
priors_count 3.082164 4.68741 0.0 0.0 1.0
days_b_screening_arrest -0.878037 72.889298 -597.0 -1.0 -1.0
c_jail_in NaN NaN NaN NaN NaN
c_jail_out NaN NaN NaN NaN NaN
c_case_number NaN NaN NaN NaN NaN
c_offense_date NaN NaN NaN NaN NaN
c_arrest_date NaN NaN NaN NaN NaN
c_days_from_compas 63.587653 341.899711 0.0 1.0 1.0
c_charge_degree NaN NaN NaN NaN NaN
c_charge_desc NaN NaN NaN NaN NaN
is_recid 0.253806 0.558324 -1.0 0.0 0.0
num_r_cases NaN NaN NaN NaN NaN
r_case_number NaN NaN NaN NaN NaN
r_charge_degree NaN NaN NaN NaN NaN
r_days_from_arrest 20.410569 74.35484 -1.0 0.0 0.0
r_offense_date NaN NaN NaN NaN NaN
r_charge_desc NaN NaN NaN NaN NaN
r_jail_in NaN NaN NaN NaN NaN
r_jail_out NaN NaN NaN NaN NaN
is_violent_recid 0.075019 0.263433 0.0 0.0 0.0
num_vr_cases NaN NaN NaN NaN NaN
vr_case_number NaN NaN NaN NaN NaN
vr_charge_degree NaN NaN NaN NaN NaN
vr_offense_date NaN NaN NaN NaN NaN
vr_charge_desc NaN NaN NaN NaN NaN
v_type_of_assessment NaN NaN NaN NaN NaN
v_decile_score 3.571489 2.500479 -1.0 1.0 3.0
v_score_text NaN NaN NaN NaN NaN
v_screening_date NaN NaN NaN NaN NaN
type_of_assessment NaN NaN NaN NaN NaN
decile_score.1 4.371268 2.877598 -1.0 2.0 4.0
score_text NaN NaN NaN NaN NaN
screening_date NaN NaN NaN NaN NaN
75% max
id 8818.0 11757.0
name NaN NaN
first NaN NaN
last NaN NaN
compas_screening_date NaN NaN
sex NaN NaN
dob NaN NaN
age 43.0 96.0
age_cat NaN NaN
race NaN NaN
juv_fel_count 0.0 20.0
decile_score 7.0 10.0
juv_misd_count 0.0 13.0
juv_other_count 0.0 17.0
priors_count 4.0 43.0
days_b_screening_arrest -1.0 1057.0
c_jail_in NaN NaN
c_jail_out NaN NaN
c_case_number NaN NaN
c_offense_date NaN NaN
c_arrest_date NaN NaN
c_days_from_compas 2.0 9485.0
c_charge_degree NaN NaN
c_charge_desc NaN NaN
is_recid 1.0 1.0
num_r_cases NaN NaN
r_case_number NaN NaN
r_charge_degree NaN NaN
r_days_from_arrest 1.0 993.0
r_offense_date NaN NaN
r_charge_desc NaN NaN
r_jail_in NaN NaN
r_jail_out NaN NaN
is_violent_recid 0.0 1.0
num_vr_cases NaN NaN
vr_case_number NaN NaN
vr_charge_degree NaN NaN
vr_offense_date NaN NaN
vr_charge_desc NaN NaN
v_type_of_assessment NaN NaN
v_decile_score 5.0 10.0
v_score_text NaN NaN
v_screening_date NaN NaN
type_of_assessment NaN NaN
decile_score.1 7.0 10.0
score_text NaN NaN
screening_date NaN NaN
=== Dataset: compas_scores_two_years ===
Shape: (7214, 53)
Column dtypes:
id int64
name str
first str
last str
compas_screening_date str
sex str
dob str
age int64
age_cat str
race str
juv_fel_count int64
decile_score int64
juv_misd_count int64
juv_other_count int64
priors_count int64
days_b_screening_arrest float64
c_jail_in str
c_jail_out str
c_case_number str
c_offense_date str
c_arrest_date str
c_days_from_compas float64
c_charge_degree str
c_charge_desc str
is_recid int64
r_case_number str
r_charge_degree str
r_days_from_arrest float64
r_offense_date str
r_charge_desc str
r_jail_in str
r_jail_out str
violent_recid float64
is_violent_recid int64
vr_case_number str
vr_charge_degree str
vr_offense_date str
vr_charge_desc str
type_of_assessment str
decile_score.1 int64
score_text str
screening_date str
v_type_of_assessment str
v_decile_score int64
v_score_text str
v_screening_date str
in_custody str
out_custody str
priors_count.1 int64
start int64
end int64
event int64
two_year_recid int64
dtype: object
Numeric summary statistics:
count mean std min 25% \
id 7214.0 5501.255753 3175.706870 1.0 2735.25
age 7214.0 34.817993 11.888922 18.0 25.00
juv_fel_count 7214.0 0.067230 0.473972 0.0 0.00
decile_score 7214.0 4.509565 2.856396 1.0 2.00
juv_misd_count 7214.0 0.090934 0.485239 0.0 0.00
juv_other_count 7214.0 0.109371 0.501586 0.0 0.00
priors_count 7214.0 3.472415 4.882538 0.0 0.00
days_b_screening_arrest 6907.0 3.304763 75.809505 -414.0 -1.00
c_days_from_compas 7192.0 57.731368 329.740215 0.0 1.00
is_recid 7214.0 0.481148 0.499679 0.0 0.00
r_days_from_arrest 2316.0 20.269430 74.871668 -1.0 0.00
violent_recid 0.0 NaN NaN NaN NaN
is_violent_recid 7214.0 0.113529 0.317261 0.0 0.00
decile_score.1 7214.0 4.509565 2.856396 1.0 2.00
v_decile_score 7214.0 3.691849 2.510148 1.0 1.00
priors_count.1 7214.0 3.472415 4.882538 0.0 0.00
start 7214.0 11.465068 46.954563 0.0 0.00
end 7214.0 553.436651 399.020583 0.0 148.25
event 7214.0 0.382867 0.486120 0.0 0.00
two_year_recid 7214.0 0.450652 0.497593 0.0 0.00
50% 75% max
id 5509.5 8246.5 11001.0
age 31.0 42.0 96.0
juv_fel_count 0.0 0.0 20.0
decile_score 4.0 7.0 10.0
juv_misd_count 0.0 0.0 13.0
juv_other_count 0.0 0.0 17.0
priors_count 2.0 5.0 38.0
days_b_screening_arrest -1.0 0.0 1057.0
c_days_from_compas 1.0 2.0 9485.0
is_recid 0.0 1.0 1.0
r_days_from_arrest 0.0 1.0 993.0
violent_recid NaN NaN NaN
is_violent_recid 0.0 0.0 1.0
decile_score.1 4.0 7.0 10.0
v_decile_score 3.0 5.0 10.0
priors_count.1 2.0 5.0 38.0
start 0.0 1.0 937.0
end 530.5 914.0 1186.0
event 0.0 1.0 1.0
two_year_recid 0.0 1.0 1.0
All-columns summary statistics:
count unique top freq \
id 7214.0 NaN NaN NaN
name 7214 7158 anthony smith 3
first 7214 2800 michael 149
last 7214 3950 williams 83
compas_screening_date 7214 690 2013-02-20 32
sex 7214 2 Male 5819
dob 7214 5452 1989-04-27 5
age 7214.0 NaN NaN NaN
age_cat 7214 3 25 - 45 4109
race 7214 6 African-American 3696
juv_fel_count 7214.0 NaN NaN NaN
decile_score 7214.0 NaN NaN NaN
juv_misd_count 7214.0 NaN NaN NaN
juv_other_count 7214.0 NaN NaN NaN
priors_count 7214.0 NaN NaN NaN
days_b_screening_arrest 6907.0 NaN NaN NaN
c_jail_in 6907 6907 2013-08-13 06:03:42 1
c_jail_out 6907 6880 2013-09-28 02:10:00 3
c_case_number 7192 7192 13011352CF10A 1
c_offense_date 6055 927 2013-02-22 26
c_arrest_date 1137 580 2013-02-06 9
c_days_from_compas 7192.0 NaN NaN NaN
c_charge_degree 7214 2 F 4666
c_charge_desc 7185 437 Battery 1156
is_recid 7214.0 NaN NaN NaN
r_case_number 3471 3471 13009779CF10A 1
r_charge_degree 3471 10 (M1) 1201
r_days_from_arrest 2316.0 NaN NaN NaN
r_offense_date 3471 1075 2014-12-08 12
r_charge_desc 3413 340 Driving License Suspended 258
r_jail_in 2316 972 2014-05-27 9
r_jail_out 2316 938 2014-12-09 9
violent_recid 0.0 NaN NaN NaN
is_violent_recid 7214.0 NaN NaN NaN
vr_case_number 819 819 13009779CF10A 1
vr_charge_degree 819 9 (M1) 344
vr_offense_date 819 570 2015-08-15 6
vr_charge_desc 819 83 Battery 329
type_of_assessment 7214 1 Risk of Recidivism 7214
decile_score.1 7214.0 NaN NaN NaN
score_text 7214 3 Low 3897
screening_date 7214 690 2013-02-20 32
v_type_of_assessment 7214 1 Risk of Violence 7214
v_decile_score 7214.0 NaN NaN NaN
v_score_text 7214 3 Low 4761
v_screening_date 7214 690 2013-02-20 32
in_custody 6978 1156 2014-01-22 20
out_custody 6978 1169 2020-01-01 61
priors_count.1 7214.0 NaN NaN NaN
start 7214.0 NaN NaN NaN
end 7214.0 NaN NaN NaN
event 7214.0 NaN NaN NaN
two_year_recid 7214.0 NaN NaN NaN
mean std min 25% 50% \
id 5501.255753 3175.70687 1.0 2735.25 5509.5
name NaN NaN NaN NaN NaN
first NaN NaN NaN NaN NaN
last NaN NaN NaN NaN NaN
compas_screening_date NaN NaN NaN NaN NaN
sex NaN NaN NaN NaN NaN
dob NaN NaN NaN NaN NaN
age 34.817993 11.888922 18.0 25.0 31.0
age_cat NaN NaN NaN NaN NaN
race NaN NaN NaN NaN NaN
juv_fel_count 0.06723 0.473972 0.0 0.0 0.0
decile_score 4.509565 2.856396 1.0 2.0 4.0
juv_misd_count 0.090934 0.485239 0.0 0.0 0.0
juv_other_count 0.109371 0.501586 0.0 0.0 0.0
priors_count 3.472415 4.882538 0.0 0.0 2.0
days_b_screening_arrest 3.304763 75.809505 -414.0 -1.0 -1.0
c_jail_in NaN NaN NaN NaN NaN
c_jail_out NaN NaN NaN NaN NaN
c_case_number NaN NaN NaN NaN NaN
c_offense_date NaN NaN NaN NaN NaN
c_arrest_date NaN NaN NaN NaN NaN
c_days_from_compas 57.731368 329.740215 0.0 1.0 1.0
c_charge_degree NaN NaN NaN NaN NaN
c_charge_desc NaN NaN NaN NaN NaN
is_recid 0.481148 0.499679 0.0 0.0 0.0
r_case_number NaN NaN NaN NaN NaN
r_charge_degree NaN NaN NaN NaN NaN
r_days_from_arrest 20.26943 74.871668 -1.0 0.0 0.0
r_offense_date NaN NaN NaN NaN NaN
r_charge_desc NaN NaN NaN NaN NaN
r_jail_in NaN NaN NaN NaN NaN
r_jail_out NaN NaN NaN NaN NaN
violent_recid NaN NaN NaN NaN NaN
is_violent_recid 0.113529 0.317261 0.0 0.0 0.0
vr_case_number NaN NaN NaN NaN NaN
vr_charge_degree NaN NaN NaN NaN NaN
vr_offense_date NaN NaN NaN NaN NaN
vr_charge_desc NaN NaN NaN NaN NaN
type_of_assessment NaN NaN NaN NaN NaN
decile_score.1 4.509565 2.856396 1.0 2.0 4.0
score_text NaN NaN NaN NaN NaN
screening_date NaN NaN NaN NaN NaN
v_type_of_assessment NaN NaN NaN NaN NaN
v_decile_score 3.691849 2.510148 1.0 1.0 3.0
v_score_text NaN NaN NaN NaN NaN
v_screening_date NaN NaN NaN NaN NaN
in_custody NaN NaN NaN NaN NaN
out_custody NaN NaN NaN NaN NaN
priors_count.1 3.472415 4.882538 0.0 0.0 2.0
start 11.465068 46.954563 0.0 0.0 0.0
end 553.436651 399.020583 0.0 148.25 530.5
event 0.382867 0.48612 0.0 0.0 0.0
two_year_recid 0.450652 0.497593 0.0 0.0 0.0
75% max
id 8246.5 11001.0
name NaN NaN
first NaN NaN
last NaN NaN
compas_screening_date NaN NaN
sex NaN NaN
dob NaN NaN
age 42.0 96.0
age_cat NaN NaN
race NaN NaN
juv_fel_count 0.0 20.0
decile_score 7.0 10.0
juv_misd_count 0.0 13.0
juv_other_count 0.0 17.0
priors_count 5.0 38.0
days_b_screening_arrest 0.0 1057.0
c_jail_in NaN NaN
c_jail_out NaN NaN
c_case_number NaN NaN
c_offense_date NaN NaN
c_arrest_date NaN NaN
c_days_from_compas 2.0 9485.0
c_charge_degree NaN NaN
c_charge_desc NaN NaN
is_recid 1.0 1.0
r_case_number NaN NaN
r_charge_degree NaN NaN
r_days_from_arrest 1.0 993.0
r_offense_date NaN NaN
r_charge_desc NaN NaN
r_jail_in NaN NaN
r_jail_out NaN NaN
violent_recid NaN NaN
is_violent_recid 0.0 1.0
vr_case_number NaN NaN
vr_charge_degree NaN NaN
vr_offense_date NaN NaN
vr_charge_desc NaN NaN
type_of_assessment NaN NaN
decile_score.1 7.0 10.0
score_text NaN NaN
screening_date NaN NaN
v_type_of_assessment NaN NaN
v_decile_score 5.0 10.0
v_score_text NaN NaN
v_screening_date NaN NaN
in_custody NaN NaN
out_custody NaN NaN
priors_count.1 5.0 38.0
start 1.0 937.0
end 914.0 1186.0
event 1.0 1.0
two_year_recid 1.0 1.0
=== Dataset: compas_scores_two_years_violent ===
Shape: (4743, 54)
Column dtypes:
id int64
name str
first str
last str
compas_screening_date str
sex str
dob str
age int64
age_cat str
race str
juv_fel_count int64
decile_score int64
juv_misd_count int64
juv_other_count int64
priors_count int64
days_b_screening_arrest float64
c_jail_in str
c_jail_out str
c_case_number str
c_offense_date str
c_arrest_date str
c_days_from_compas float64
c_charge_degree str
c_charge_desc str
is_recid int64
r_case_number str
r_charge_degree str
r_days_from_arrest float64
r_offense_date str
r_charge_desc str
r_jail_in str
r_jail_out str
violent_recid float64
is_violent_recid int64
vr_case_number str
vr_charge_degree str
vr_offense_date str
vr_charge_desc str
type_of_assessment str
decile_score.1 int64
score_text str
screening_date str
v_type_of_assessment str
v_decile_score int64
v_score_text str
v_screening_date str
in_custody str
out_custody str
priors_count.1 int64
start int64
end int64
event int64
two_year_recid int64
two_year_recid.1 int64
dtype: object
Numeric summary statistics:
count mean std min 25% \
id 4743.0 5454.230445 3189.515451 1.0 2631.0
age 4743.0 36.002952 12.200715 18.0 26.0
juv_fel_count 4743.0 0.045330 0.420694 0.0 0.0
decile_score 4743.0 3.978073 2.749706 -1.0 1.0
juv_misd_count 4743.0 0.070841 0.406142 0.0 0.0
juv_other_count 4743.0 0.080751 0.468779 0.0 0.0
priors_count 4743.0 2.677209 4.084277 0.0 0.0
days_b_screening_arrest 4475.0 2.723799 78.074361 -345.0 -1.0
c_days_from_compas 4732.0 72.691251 385.694476 0.0 1.0
is_recid 4743.0 0.209783 0.407196 0.0 0.0
r_days_from_arrest 690.0 8.918841 40.179739 -1.0 0.0
violent_recid 0.0 NaN NaN NaN NaN
is_violent_recid 4743.0 0.172676 0.378006 0.0 0.0
decile_score.1 4743.0 3.978073 2.749706 -1.0 1.0
v_decile_score 4743.0 3.338815 2.415791 1.0 1.0
priors_count.1 4743.0 2.677209 4.084277 0.0 0.0
start 4743.0 9.084124 36.867544 0.0 0.0
end 4743.0 730.143369 365.435158 1.0 422.5
event 4743.0 0.165507 0.371677 0.0 0.0
two_year_recid 4743.0 0.163399 0.369768 0.0 0.0
two_year_recid.1 4743.0 0.163399 0.369768 0.0 0.0
50% 75% max
id 5475.0 8239.5 11000.0
age 33.0 44.0 83.0
juv_fel_count 0.0 0.0 20.0
decile_score 3.0 6.0 10.0
juv_misd_count 0.0 0.0 8.0
juv_other_count 0.0 0.0 17.0
priors_count 1.0 3.0 38.0
days_b_screening_arrest -1.0 -1.0 1057.0
c_days_from_compas 1.0 2.0 9485.0
is_recid 0.0 0.0 1.0
r_days_from_arrest 0.0 0.0 596.0
violent_recid NaN NaN NaN
is_violent_recid 0.0 0.0 1.0
decile_score.1 3.0 6.0 10.0
v_decile_score 3.0 5.0 10.0
priors_count.1 1.0 3.0 38.0
start 0.0 1.0 672.0
end 837.0 1039.0 1186.0
event 0.0 0.0 1.0
two_year_recid 0.0 0.0 1.0
two_year_recid.1 0.0 0.0 1.0
All-columns summary statistics:
count unique top freq \
id 4743.0 NaN NaN NaN
name 4743 4721 clinton johnson 2
first 4743 2114 michael 95
last 4743 2897 williams 49
compas_screening_date 4743 586 2013-09-23 24
sex 4743 2 Male 3746
dob 4743 3978 1983-02-03 4
age 4743.0 NaN NaN NaN
age_cat 4743 3 25 - 45 2699
race 4743 6 African-American 2266
juv_fel_count 4743.0 NaN NaN NaN
decile_score 4743.0 NaN NaN NaN
juv_misd_count 4743.0 NaN NaN NaN
juv_other_count 4743.0 NaN NaN NaN
priors_count 4743.0 NaN NaN NaN
days_b_screening_arrest 4475.0 NaN NaN NaN
c_jail_in 4475 4475 2013-08-13 06:03:42 1
c_jail_out 4475 4466 2013-09-28 02:10:00 3
c_case_number 4732 4732 13011352CF10A 1
c_offense_date 3972 766 2013-01-14 21
c_arrest_date 760 425 2013-05-15 7
c_days_from_compas 4732.0 NaN NaN NaN
c_charge_degree 4743 2 F 2880
c_charge_desc 4726 362 Battery 886
is_recid 4743.0 NaN NaN NaN
r_case_number 995 995 13009779CF10A 1
r_charge_degree 995 10 (M1) 423
r_days_from_arrest 690.0 NaN NaN NaN
r_offense_date 995 675 2014-04-05 5
r_charge_desc 972 196 Battery 192
r_jail_in 690 522 2014-02-18 5
r_jail_out 690 494 2014-01-16 5
violent_recid 0.0 NaN NaN NaN
is_violent_recid 4743.0 NaN NaN NaN
vr_case_number 819 819 13009779CF10A 1
vr_charge_degree 819 9 (M1) 344
vr_offense_date 819 570 2015-08-15 6
vr_charge_desc 819 83 Battery 329
type_of_assessment 4743 1 Risk of Recidivism 4743
decile_score.1 4743.0 NaN NaN NaN
score_text 4738 3 Low 2939
screening_date 4743 586 2013-09-23 24
v_type_of_assessment 4743 1 Risk of Violence 4743
v_decile_score 4743.0 NaN NaN NaN
v_score_text 4743 3 Low 3389
v_screening_date 4743 586 2013-09-23 24
in_custody 4534 1003 2014-01-04 19
out_custody 4534 1012 2020-01-01 27
priors_count.1 4743.0 NaN NaN NaN
start 4743.0 NaN NaN NaN
end 4743.0 NaN NaN NaN
event 4743.0 NaN NaN NaN
two_year_recid 4743.0 NaN NaN NaN
two_year_recid.1 4743.0 NaN NaN NaN
mean std min 25% 50% \
id 5454.230445 3189.515451 1.0 2631.0 5475.0
name NaN NaN NaN NaN NaN
first NaN NaN NaN NaN NaN
last NaN NaN NaN NaN NaN
compas_screening_date NaN NaN NaN NaN NaN
sex NaN NaN NaN NaN NaN
dob NaN NaN NaN NaN NaN
age 36.002952 12.200715 18.0 26.0 33.0
age_cat NaN NaN NaN NaN NaN
race NaN NaN NaN NaN NaN
juv_fel_count 0.04533 0.420694 0.0 0.0 0.0
decile_score 3.978073 2.749706 -1.0 1.0 3.0
juv_misd_count 0.070841 0.406142 0.0 0.0 0.0
juv_other_count 0.080751 0.468779 0.0 0.0 0.0
priors_count 2.677209 4.084277 0.0 0.0 1.0
days_b_screening_arrest 2.723799 78.074361 -345.0 -1.0 -1.0
c_jail_in NaN NaN NaN NaN NaN
c_jail_out NaN NaN NaN NaN NaN
c_case_number NaN NaN NaN NaN NaN
c_offense_date NaN NaN NaN NaN NaN
c_arrest_date NaN NaN NaN NaN NaN
c_days_from_compas 72.691251 385.694476 0.0 1.0 1.0
c_charge_degree NaN NaN NaN NaN NaN
c_charge_desc NaN NaN NaN NaN NaN
is_recid 0.209783 0.407196 0.0 0.0 0.0
r_case_number NaN NaN NaN NaN NaN
r_charge_degree NaN NaN NaN NaN NaN
r_days_from_arrest 8.918841 40.179739 -1.0 0.0 0.0
r_offense_date NaN NaN NaN NaN NaN
r_charge_desc NaN NaN NaN NaN NaN
r_jail_in NaN NaN NaN NaN NaN
r_jail_out NaN NaN NaN NaN NaN
violent_recid NaN NaN NaN NaN NaN
is_violent_recid 0.172676 0.378006 0.0 0.0 0.0
vr_case_number NaN NaN NaN NaN NaN
vr_charge_degree NaN NaN NaN NaN NaN
vr_offense_date NaN NaN NaN NaN NaN
vr_charge_desc NaN NaN NaN NaN NaN
type_of_assessment NaN NaN NaN NaN NaN
decile_score.1 3.978073 2.749706 -1.0 1.0 3.0
score_text NaN NaN NaN NaN NaN
screening_date NaN NaN NaN NaN NaN
v_type_of_assessment NaN NaN NaN NaN NaN
v_decile_score 3.338815 2.415791 1.0 1.0 3.0
v_score_text NaN NaN NaN NaN NaN
v_screening_date NaN NaN NaN NaN NaN
in_custody NaN NaN NaN NaN NaN
out_custody NaN NaN NaN NaN NaN
priors_count.1 2.677209 4.084277 0.0 0.0 1.0
start 9.084124 36.867544 0.0 0.0 0.0
end 730.143369 365.435158 1.0 422.5 837.0
event 0.165507 0.371677 0.0 0.0 0.0
two_year_recid 0.163399 0.369768 0.0 0.0 0.0
two_year_recid.1 0.163399 0.369768 0.0 0.0 0.0
75% max
id 8239.5 11000.0
name NaN NaN
first NaN NaN
last NaN NaN
compas_screening_date NaN NaN
sex NaN NaN
dob NaN NaN
age 44.0 83.0
age_cat NaN NaN
race NaN NaN
juv_fel_count 0.0 20.0
decile_score 6.0 10.0
juv_misd_count 0.0 8.0
juv_other_count 0.0 17.0
priors_count 3.0 38.0
days_b_screening_arrest -1.0 1057.0
c_jail_in NaN NaN
c_jail_out NaN NaN
c_case_number NaN NaN
c_offense_date NaN NaN
c_arrest_date NaN NaN
c_days_from_compas 2.0 9485.0
c_charge_degree NaN NaN
c_charge_desc NaN NaN
is_recid 0.0 1.0
r_case_number NaN NaN
r_charge_degree NaN NaN
r_days_from_arrest 0.0 596.0
r_offense_date NaN NaN
r_charge_desc NaN NaN
r_jail_in NaN NaN
r_jail_out NaN NaN
violent_recid NaN NaN
is_violent_recid 0.0 1.0
vr_case_number NaN NaN
vr_charge_degree NaN NaN
vr_offense_date NaN NaN
vr_charge_desc NaN NaN
type_of_assessment NaN NaN
decile_score.1 6.0 10.0
score_text NaN NaN
screening_date NaN NaN
v_type_of_assessment NaN NaN
v_decile_score 5.0 10.0
v_score_text NaN NaN
v_screening_date NaN NaN
in_custody NaN NaN
out_custody NaN NaN
priors_count.1 3.0 38.0
start 1.0 672.0
end 1039.0 1186.0
event 0.0 1.0
two_year_recid 0.0 1.0
two_year_recid.1 0.0 1.0
=== Dataset: compas_scores_raw ===
Shape: (60843, 28)
Column dtypes:
Person_ID int64
AssessmentID int64
Case_ID int64
Agency_Text str
LastName str
FirstName str
MiddleName str
Sex_Code_Text str
Ethnic_Code_Text str
DateOfBirth str
ScaleSet_ID int64
ScaleSet str
AssessmentReason str
Language str
LegalStatus str
CustodyStatus str
MaritalStatus str
Screening_Date str
RecSupervisionLevel int64
RecSupervisionLevelText str
Scale_ID int64
DisplayText str
RawScore float64
DecileScore int64
ScoreText str
AssessmentType str
IsCompleted int64
IsDeleted int64
dtype: object
Numeric summary statistics:
count mean std min 25% \
Person_ID 60843.0 53683.206154 14363.648515 656.00 52039.00
AssessmentID 60843.0 68061.029190 7320.208226 649.00 62582.00
Case_ID 60843.0 60209.128149 9638.501654 350.00 56021.00
ScaleSet_ID 60843.0 21.819536 0.932614 17.00 22.00
RecSupervisionLevel 60843.0 1.630048 0.944220 1.00 1.00
Scale_ID 60843.0 11.000000 4.966596 7.00 7.00
RawScore 60843.0 5.081457 10.080518 -4.79 -2.09
DecileScore 60843.0 3.571701 2.617854 -1.00 1.00
IsCompleted 60843.0 1.000000 0.000000 1.00 1.00
IsDeleted 60843.0 0.000000 0.000000 0.00 0.00
50% 75% max
Person_ID 57321.00 62748.0 68608.0
AssessmentID 68229.00 73870.0 79678.0
Case_ID 61261.00 66554.0 72045.0
ScaleSet_ID 22.00 22.0 22.0
RecSupervisionLevel 1.00 2.0 4.0
Scale_ID 8.00 18.0 18.0
RawScore -0.71 14.0 51.0
DecileScore 3.00 5.0 10.0
IsCompleted 1.00 1.0 1.0
IsDeleted 0.00 0.0 0.0
All-columns summary statistics:
count unique top freq \
Person_ID 60843.0 NaN NaN NaN
AssessmentID 60843.0 NaN NaN NaN
Case_ID 60843.0 NaN NaN NaN
Agency_Text 60843 4 PRETRIAL 41100
LastName 60843 10896 Williams 417
FirstName 60843 7225 Michael 777
MiddleName 15624 1870 A 606
Sex_Code_Text 60843 2 Male 47514
Ethnic_Code_Text 60843 9 African-American 27018
DateOfBirth 60843 10382 11/10/88 30
ScaleSet_ID 60843.0 NaN NaN NaN
ScaleSet 60843 2 Risk and Prescreen 58647
AssessmentReason 60843 1 Intake 60843
Language 60843 2 English 60591
LegalStatus 60843 7 Pretrial 37578
CustodyStatus 60843 6 Jail Inmate 30129
MaritalStatus 60843 7 Single 45126
Screening_Date 60843 705 3/21/13 0:00 183
RecSupervisionLevel 60843.0 NaN NaN NaN
RecSupervisionLevelText 60843 4 Low 38472
Scale_ID 60843.0 NaN NaN NaN
DisplayText 60843 3 Risk of Violence 20281
RawScore 60843.0 NaN NaN NaN
DecileScore 60843.0 NaN NaN NaN
ScoreText 60798 3 Low 41487
AssessmentType 60843 2 New 56139
IsCompleted 60843.0 NaN NaN NaN
IsDeleted 60843.0 NaN NaN NaN
mean std min 25% 50% \
Person_ID 53683.206154 14363.648515 656.0 52039.0 57321.0
AssessmentID 68061.02919 7320.208226 649.0 62582.0 68229.0
Case_ID 60209.128149 9638.501654 350.0 56021.0 61261.0
Agency_Text NaN NaN NaN NaN NaN
LastName NaN NaN NaN NaN NaN
FirstName NaN NaN NaN NaN NaN
MiddleName NaN NaN NaN NaN NaN
Sex_Code_Text NaN NaN NaN NaN NaN
Ethnic_Code_Text NaN NaN NaN NaN NaN
DateOfBirth NaN NaN NaN NaN NaN
ScaleSet_ID 21.819536 0.932614 17.0 22.0 22.0
ScaleSet NaN NaN NaN NaN NaN
AssessmentReason NaN NaN NaN NaN NaN
Language NaN NaN NaN NaN NaN
LegalStatus NaN NaN NaN NaN NaN
CustodyStatus NaN NaN NaN NaN NaN
MaritalStatus NaN NaN NaN NaN NaN
Screening_Date NaN NaN NaN NaN NaN
RecSupervisionLevel 1.630048 0.94422 1.0 1.0 1.0
RecSupervisionLevelText NaN NaN NaN NaN NaN
Scale_ID 11.0 4.966596 7.0 7.0 8.0
DisplayText NaN NaN NaN NaN NaN
RawScore 5.081457 10.080518 -4.79 -2.09 -0.71
DecileScore 3.571701 2.617854 -1.0 1.0 3.0
ScoreText NaN NaN NaN NaN NaN
AssessmentType NaN NaN NaN NaN NaN
IsCompleted 1.0 0.0 1.0 1.0 1.0
IsDeleted 0.0 0.0 0.0 0.0 0.0
75% max
Person_ID 62748.0 68608.0
AssessmentID 73870.0 79678.0
Case_ID 66554.0 72045.0
Agency_Text NaN NaN
LastName NaN NaN
FirstName NaN NaN
MiddleName NaN NaN
Sex_Code_Text NaN NaN
Ethnic_Code_Text NaN NaN
DateOfBirth NaN NaN
ScaleSet_ID 22.0 22.0
ScaleSet NaN NaN
AssessmentReason NaN NaN
Language NaN NaN
LegalStatus NaN NaN
CustodyStatus NaN NaN
MaritalStatus NaN NaN
Screening_Date NaN NaN
RecSupervisionLevel 2.0 4.0
RecSupervisionLevelText NaN NaN
Scale_ID 18.0 18.0
DisplayText NaN NaN
RawScore 14.0 51.0
DecileScore 5.0 10.0
ScoreText NaN NaN
AssessmentType NaN NaN
IsCompleted 1.0 1.0
IsDeleted 0.0 0.0
Analyzing
All six datasets were successfully loaded and for each the script printed: 1) DataFrame shape and column data types, 2) numeric descriptive statistics (count, mean, std, min, 25%, 50%, 75%, max), and 3) full ‘include=all’ summary (counts, unique values, top/freq for categoricals plus statistics for numerics). No errors occurred, and summary statistics are available for all files.
DataVoyager
The experiment was implemented as planned. All six datasets were loaded successfully and summary statistics were generated, including both numeric and all-column descriptive statistics.
Planning
Question: Race and Decile Score Analysis - Explore the distribution of decile scores across different racial groups. This could help identify any disparities in risk assessment scores based on race.
The dataset files have already been synchronized to the shared filesystem. Use the PROGRAMMER agent to load them directly from the file paths listed below and do not attempt to access AWS services from the remote runtime.
-
data_0: /workspace/datasets/user_uuid=d6b8184f-c2cb-45d1-a6e4-0b49820cfd64/thread_id=f675a76b-c116-44be-bb4c-a9cd43d13216/uploads/cox-violent-parsed.csv
-
data_1: /workspace/datasets/user_uuid=d6b8184f-c2cb-45d1-a6e4-0b49820cfd64/thread_id=f675a76b-c116-44be-bb4c-a9cd43d13216/uploads/cox-parsed.csv
-
data_2: /workspace/datasets/user_uuid=d6b8184f-c2cb-45d1-a6e4-0b49820cfd64/thread_id=f675a76b-c116-44be-bb4c-a9cd43d13216/uploads/compas-scores.csv
-
data_3: /workspace/datasets/user_uuid=d6b8184f-c2cb-45d1-a6e4-0b49820cfd64/thread_id=f675a76b-c116-44be-bb4c-a9cd43d13216/uploads/compas-scores-two-years.csv
-
data_4: /workspace/datasets/user_uuid=d6b8184f-c2cb-45d1-a6e4-0b49820cfd64/thread_id=f675a76b-c116-44be-bb4c-a9cd43d13216/uploads/compas-scores-two-years-violent.csv
-
data_5: /workspace/datasets/user_uuid=d6b8184f-c2cb-45d1-a6e4-0b49820cfd64/thread_id=f675a76b-c116-44be-bb4c-a9cd43d13216/uploads/compas-scores-raw.csv
Experiment objective: Explore the distribution of COMPAS decile scores across racial groups to identify any disparities in risk assessment.
Steps for the programmer:
-
Select the primary dataset compas-scores.csv (data_2) and load it into a DataFrame.
-
Inspect the DataFrame to confirm the presence and types of the columns “race” and “decile_score”; convert them to categorical and numeric as needed.
-
Remove or impute any records with missing or invalid values in “race” or “decile_score.”
-
Compute descriptive statistics of decile_score for each race: count, mean, median, standard deviation, and interquartile range.
-
Visualize the distribution of decile_score by race using boxplots or violin plots, and overlay histograms or kernel density estimates (KDE) for finer-grained comparison.
-
Perform a global non-parametric test (e.g., Kruskal–Wallis H-test) to assess whether distributions differ significantly across all racial groups.
-
If the global test is significant, conduct pairwise post-hoc comparisons (e.g., Dunn’s test with Bonferroni correction) to identify which racial groups differ from each other.
-
(Optional) Repeat steps 1–7 on compas-scores-two-years.csv (data_3) and compas-scores-two-years-violent.csv (data_4) to check whether observed disparities persist when focusing on two-year general and violent recidivism subsets.
Deliverables: • A cleaned DataFrame with valid “race” and “decile_score” entries. • A summary table of decile_score descriptive statistics by race. • Distribution visualizations (boxplots/violins, histograms/KDE) comparing decile_score across races. • Results of the Kruskal–Wallis test (H statistic, p-value) and post-hoc pairwise tests with adjusted p-values. • A brief report interpreting the statistical findings and highlighting any racial disparities in decile scores.
Programming
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
from scipy.stats import kruskal
import subprocess, sys
# Helper to install if needed
def install(pkg):
subprocess.check_call([sys.executable, "-m", "pip", "install", "--quiet", pkg])
# Try importing Dunn's test library
try:
import scikit_posthocs as sp
except ImportError:
install('scikit-posthocs')
import scikit_posthocs as sp
# 1. Load primary dataset
path = (
'/workspace/datasets/user_uuid=d6b8184f-c2cb-45d1-a6e4-0b49820cfd64/' \
'thread_id=f675a76b-c116-44be-bb4c-a9cd43d13216/uploads/compas-scores.csv'
)
df = pd.read_csv(path)
# 2. Ensure correct types and drop invalid
df = df[['race', 'decile_score']].dropna()
df['decile_score'] = pd.to_numeric(df['decile_score'], errors='coerce')
df = df.dropna(subset=['race', 'decile_score'])
df['race'] = df['race'].astype('category')
# 3. Descriptive statistics by race
def iqr(x): return x.quantile(0.75) - x.quantile(0.25)
stats = df.groupby('race')['decile_score'].agg(
count='count', mean='mean', median='median', std='std', iqr=iqr
)
print("Descriptive statistics of Decile Score by Race:\n", stats)
# 4. Boxplot
plt.figure(figsize=(10, 6))
sns.boxplot(x='race', y='decile_score', data=df)
plt.xticks(rotation=45)
plt.title('Boxplot of COMPAS Decile Score by Race')
plt.xlabel('Race')
plt.ylabel('Decile Score')
plt.tight_layout()
plt.show()
# 5. KDE overlay
plt.figure(figsize=(10, 6))
for r in df['race'].cat.categories:
sns.kdeplot(df.loc[df['race'] == r, 'decile_score'], label=r, bw_adjust=1)
plt.title('KDE of COMPAS Decile Score by Race')
plt.xlabel('Decile Score')
plt.legend()
plt.tight_layout()
plt.show()
# 6. Kruskal-Wallis test
groups = [grp['decile_score'].values for _, grp in df.groupby('race')]
H, p = kruskal(*groups)
print(f"Kruskal-Wallis H-test result: H = {H:.3f}, p = {p:.3e}")
# 7. Pairwise Dunn's test with Bonferroni correction
dunn_res = sp.posthoc_dunn(df, val_col='decile_score', group_col='race', p_adjust='bonferroni')
print("\nPairwise Dunn's test p-values (Bonferroni):")
print(dunn_res)
exitcode: 0 (execution succeeded)
Code output: Descriptive statistics of Decile Score by Race:
count mean median std iqr
race
African-American 5813 5.301910 5.0 2.888116 5.0
Asian 58 2.741379 2.0 2.328770 2.0
Caucasian 4085 3.618605 3.0 2.579765 4.0
Hispanic 1100 3.279091 2.0 2.554469 4.0
Native American 40 4.750000 4.0 2.862221 5.0
Other 661 2.776097 2.0 2.264637 3.0
Kruskal-Wallis H-test result: H = 1305.799, p = 3.540e-280
Pairwise Dunn's test p-values (Bonferroni):
African-American Asian Caucasian \
African-American 1.000000e+00 2.343148e-11 8.129544e-179
Asian 2.343148e-11 1.000000e+00 1.253481e-01
Caucasian 8.129544e-179 1.253481e-01 1.000000e+00
Hispanic 9.577445e-105 1.000000e+00 1.103014e-03
Native American 1.000000e+00 3.064300e-03 1.362688e-01
Other 2.820249e-107 1.000000e+00 1.621201e-13
Hispanic Native American Other
African-American 9.577445e-105 1.000000 2.820249e-107
Asian 1.000000e+00 0.003064 1.000000e+00
Caucasian 1.103014e-03 0.136269 1.621201e-13
Hispanic 1.000000e+00 0.009678 1.785555e-03
Native American 9.678387e-03 1.000000 8.606260e-05
Other 1.785555e-03 0.000086 1.000000e+00
STDERR:
[33mWARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv[0m[33m
[0m
[1m[[0m[34;49mnotice[0m[1;39;49m][0m[39;49m A new release of pip is available: [0m[31;49m23.3.2[0m[39;49m -> [0m[32;49m26.2.2[0m
[1m[[0m[34;49mnotice[0m[1;39;49m][0m[39;49m To update, run: [0m[32;49mpip install --upgrade pip[0m
=== Plot Analysis (figure 1) ===
1. Plot Type: The plot is a box plot used to represent the distribution of COMPAS decile scores across different racial groups.
2. Axes:
* X-Axis: Labeled 'Race' with categories - African-American, Asian, Caucasian, Hispanic, Native American, and Other.
* Y-Axis: Labeled 'Decile Score' with values ranging from 0 to 10.
3. Data Trends:
* African-Americans and Native Americans have higher median decile scores, whereas Asians have the lowest.
* There are outliers present among Asian, Native American, and Other groups.
* Distributions show that score variability is larger in the African-American group compared to the Asian group.
4. Annotations and Legends: There are no additional annotations or legends present.
5. Statistical Insights: The box plot highlights racial disparities in COMPAS decile scores, suggesting implicit biases in the score distribution, with African-Americans and Native Americans having notably higher scores.
==================================================
=== Plot Analysis (figure 2) ===
1. Plot Type: The plot is a Kernel Density Estimate (KDE) plot used to visualize distribution by smoothing the distribution curve over different decile scores for various racial groups.
2. Axes: The x-axis is titled "Decile Score" with a range from approximately -2.5 to 12.5. The y-axis is titled "Density" with values ranging from 0.00 to 0.35.
3. Data Trends: The KDE plot shows distinct distribution patterns for different racial groups. The "Other" category has the highest density peak. African-American and Native American distributions are more spread out with lower density peaks compared to others.
4. Annotations and Legends: The legend identifies the racial groups by color: African-American (blue), Asian (orange), Caucasian (green), Hispanic (red), Native American (purple), Other (brown).
5. Statistical Insights: The KDE plot shows that different racial groups have different concentration and spread of decile scores, suggesting varying central tendencies and dispersion across groups.
==================================================
Interpretation for figure 1:
- Plot Type: The plot is a box plot used to represent the distribution of COMPAS decile scores across different racial groups.
-
Axes:
- X-Axis: Labeled 'Race' with categories - African-American, Asian, Caucasian, Hispanic, Native American, and Other.
- Y-Axis: Labeled 'Decile Score' with values ranging from 0 to 10.
-
Data Trends:
- African-Americans and Native Americans have higher median decile scores, whereas Asians have the lowest.
- There are outliers present among Asian, Native American, and Other groups.
- Distributions show that score variability is larger in the African-American group compared to the Asian group.
-
Annotations and Legends: There are no additional annotations or legends present.
-
Statistical Insights: The box plot highlights racial disparities in COMPAS decile scores, suggesting implicit biases in the score distribution, with African-Americans and Native Americans having notably higher scores.
Interpretation for figure 2:
-
Plot Type: The plot is a Kernel Density Estimate (KDE) plot used to visualize distribution by smoothing the distribution curve over different decile scores for various racial groups.
-
Axes: The x-axis is titled "Decile Score" with a range from approximately -2.5 to 12.5. The y-axis is titled "Density" with values ranging from 0.00 to 0.35.
-
Data Trends: The KDE plot shows distinct distribution patterns for different racial groups. The "Other" category has the highest density peak. African-American and Native American distributions are more spread out with lower density peaks compared to others.
-
Annotations and Legends: The legend identifies the racial groups by color: African-American (blue), Asian (orange), Caucasian (green), Hispanic (red), Native American (purple), Other (brown).
-
Statistical Insights: The KDE plot shows that different racial groups have different concentration and spread of decile scores, suggesting varying central tendencies and dispersion across groups.
Analyzing
The analysis of COMPAS decile scores by race (n=11,762) reveals clear disparities:
-
Descriptive Statistics: • African-American (n=5,813): mean=5.30, median=5, SD=2.89, IQR=5 • Asian (n=58): mean=2.74, median=2, SD=2.33, IQR=2 • Caucasian (n=4,085): mean=3.62, median=3, SD=2.58, IQR=4 • Hispanic (n=1,100): mean=3.28, median=2, SD=2.55, IQR=4 • Native American (n=40): mean=4.75, median=4, SD=2.86, IQR=5 • Other (n=661): mean=2.78, median=2, SD=2.26, IQR=3
-
Visualizations: • Boxplots show higher median and broader spread for African-American and Native American groups, and the lowest medians for Asian and Other groups. • KDE curves confirm these differences in central tendency and dispersion.
-
Statistical Testing: • Kruskal–Wallis H=1305.80, p<1e-200, indicates significant overall differences among races. • Post-hoc Dunn’s tests (Bonferroni-adjusted) highlight significant pairwise differences, notably: – African-American vs. Caucasian, Hispanic, Other (p<1e-3) – Asian vs. African-American, Native American (p<0.01) – Native American vs. Other (p<1e-4)
Conclusion: African-American defendants receive substantially higher COMPAS decile scores than most other racial groups, while Asian and Other categories have the lowest scores. These disparities are statistically significant and warrant further investigation into potential biases in the risk-assessment algorithm.
DataVoyager
The experiment to analyze COMPAS decile score distributions by race was implemented correctly per the plan. Using the compas-scores.csv dataset, the analysis:
• Cleaned the data (dropped missing values in race and decile_score).
• Computed descriptive statistics (count, mean, median, std, IQR) by race.
• Visualized disparities with boxplots and KDE plots.
• Conducted a Kruskal–Wallis test (H=1305.80, p<3.5e-280), confirming significant overall differences.
• Performed Dunn’s post-hoc tests with Bonferroni correction, identifying which racial pairs differ significantly.
Findings indicate African-Americans have the highest median and mean decile scores, while Asians and the “Other” category have the lowest. The disparities are statistically significant, suggesting potential bias in COMPAS risk assessments.