missing data

Steen Flammild Harsted & Søren O’Neill

The Workflow

The Workflow

Missing data

What?

Where?

Why?

How much?

pattern or missing at random?

How much?

summary(msleep)
     name              genus               vore              order          
 Length:83          Length:83          Length:83          Length:83         
 Class :character   Class :character   Class :character   Class :character  
 Mode  :character   Mode  :character   Mode  :character   Mode  :character  
                                                                            
                                                                            
                                                                            
                                                                            
 conservation        sleep_total      sleep_rem      sleep_cycle    
 Length:83          Min.   : 1.90   Min.   :0.100   Min.   :0.1167  
 Class :character   1st Qu.: 7.85   1st Qu.:0.900   1st Qu.:0.1833  
 Mode  :character   Median :10.10   Median :1.500   Median :0.3333  
                    Mean   :10.43   Mean   :1.875   Mean   :0.4396  
                    3rd Qu.:13.75   3rd Qu.:2.400   3rd Qu.:0.5792  
                    Max.   :19.90   Max.   :6.600   Max.   :1.5000  
                                    NA's   :22      NA's   :51      
     awake          brainwt            bodywt        
 Min.   : 4.10   Min.   :0.00014   Min.   :   0.005  
 1st Qu.:10.25   1st Qu.:0.00290   1st Qu.:   0.174  
 Median :13.90   Median :0.01240   Median :   1.670  
 Mean   :13.57   Mean   :0.28158   Mean   : 166.136  
 3rd Qu.:16.15   3rd Qu.:0.12550   3rd Qu.:  41.750  
 Max.   :22.10   Max.   :5.71200   Max.   :6654.000  
                 NA's   :27                          

How much?

is.na(starwars)
       name height  mass hair_color skin_color eye_color birth_year   sex
 [1,] FALSE  FALSE FALSE      FALSE      FALSE     FALSE      FALSE FALSE
 [2,] FALSE  FALSE FALSE       TRUE      FALSE     FALSE      FALSE FALSE
 [3,] FALSE  FALSE FALSE       TRUE      FALSE     FALSE      FALSE FALSE
 [4,] FALSE  FALSE FALSE      FALSE      FALSE     FALSE      FALSE FALSE
 [5,] FALSE  FALSE FALSE      FALSE      FALSE     FALSE      FALSE FALSE
 [6,] FALSE  FALSE FALSE      FALSE      FALSE     FALSE      FALSE FALSE
 [7,] FALSE  FALSE FALSE      FALSE      FALSE     FALSE      FALSE FALSE
 [8,] FALSE  FALSE FALSE       TRUE      FALSE     FALSE       TRUE FALSE
 [9,] FALSE  FALSE FALSE      FALSE      FALSE     FALSE      FALSE FALSE
[10,] FALSE  FALSE FALSE      FALSE      FALSE     FALSE      FALSE FALSE
[11,] FALSE  FALSE FALSE      FALSE      FALSE     FALSE      FALSE FALSE
[12,] FALSE  FALSE  TRUE      FALSE      FALSE     FALSE      FALSE FALSE
[13,] FALSE  FALSE FALSE      FALSE      FALSE     FALSE      FALSE FALSE
[14,] FALSE  FALSE FALSE      FALSE      FALSE     FALSE      FALSE FALSE
[15,] FALSE  FALSE FALSE       TRUE      FALSE     FALSE      FALSE FALSE
[16,] FALSE  FALSE FALSE       TRUE      FALSE     FALSE      FALSE FALSE
[17,] FALSE  FALSE FALSE      FALSE      FALSE     FALSE      FALSE FALSE
[18,] FALSE  FALSE FALSE      FALSE      FALSE     FALSE       TRUE  TRUE
[19,] FALSE  FALSE FALSE      FALSE      FALSE     FALSE      FALSE FALSE
[20,] FALSE  FALSE FALSE      FALSE      FALSE     FALSE      FALSE FALSE
[21,] FALSE  FALSE FALSE      FALSE      FALSE     FALSE      FALSE FALSE
[22,] FALSE  FALSE FALSE      FALSE      FALSE     FALSE      FALSE FALSE
[23,] FALSE  FALSE FALSE      FALSE      FALSE     FALSE      FALSE FALSE
[24,] FALSE  FALSE FALSE      FALSE      FALSE     FALSE      FALSE FALSE
[25,] FALSE  FALSE FALSE      FALSE      FALSE     FALSE      FALSE FALSE
[26,] FALSE  FALSE FALSE      FALSE      FALSE     FALSE      FALSE FALSE
[27,] FALSE  FALSE  TRUE      FALSE      FALSE     FALSE      FALSE FALSE
[28,] FALSE   TRUE  TRUE      FALSE      FALSE     FALSE       TRUE FALSE
[29,] FALSE  FALSE FALSE      FALSE      FALSE     FALSE      FALSE FALSE
[30,] FALSE  FALSE FALSE      FALSE      FALSE     FALSE       TRUE FALSE
[31,] FALSE  FALSE FALSE      FALSE      FALSE     FALSE      FALSE FALSE
[32,] FALSE  FALSE FALSE      FALSE      FALSE     FALSE       TRUE FALSE
[33,] FALSE  FALSE  TRUE      FALSE      FALSE     FALSE      FALSE FALSE
[34,] FALSE  FALSE FALSE      FALSE      FALSE     FALSE      FALSE FALSE
[35,] FALSE  FALSE FALSE      FALSE      FALSE     FALSE      FALSE FALSE
[36,] FALSE  FALSE FALSE      FALSE      FALSE     FALSE       TRUE FALSE
[37,] FALSE  FALSE  TRUE      FALSE      FALSE     FALSE       TRUE FALSE
[38,] FALSE  FALSE  TRUE      FALSE      FALSE     FALSE       TRUE FALSE
[39,] FALSE  FALSE  TRUE      FALSE      FALSE     FALSE       TRUE FALSE
[40,] FALSE  FALSE FALSE      FALSE      FALSE     FALSE       TRUE FALSE
[41,] FALSE  FALSE  TRUE      FALSE      FALSE     FALSE      FALSE FALSE
[42,] FALSE  FALSE  TRUE      FALSE      FALSE     FALSE      FALSE FALSE
[43,] FALSE  FALSE FALSE      FALSE      FALSE     FALSE      FALSE FALSE
[44,] FALSE  FALSE  TRUE      FALSE      FALSE     FALSE       TRUE FALSE
[45,] FALSE  FALSE FALSE      FALSE      FALSE     FALSE      FALSE FALSE
[46,] FALSE  FALSE FALSE      FALSE      FALSE     FALSE       TRUE FALSE
[47,] FALSE  FALSE FALSE      FALSE      FALSE     FALSE       TRUE FALSE
[48,] FALSE  FALSE  TRUE      FALSE      FALSE     FALSE       TRUE FALSE
[49,] FALSE  FALSE FALSE      FALSE      FALSE     FALSE       TRUE FALSE
[50,] FALSE  FALSE FALSE      FALSE      FALSE     FALSE      FALSE FALSE
[51,] FALSE  FALSE FALSE      FALSE      FALSE     FALSE      FALSE FALSE
[52,] FALSE  FALSE FALSE      FALSE      FALSE     FALSE       TRUE FALSE
[53,] FALSE  FALSE  TRUE      FALSE      FALSE     FALSE       TRUE FALSE
[54,] FALSE  FALSE FALSE      FALSE      FALSE     FALSE       TRUE FALSE
[55,] FALSE  FALSE  TRUE      FALSE      FALSE     FALSE       TRUE FALSE
[56,] FALSE  FALSE  TRUE      FALSE      FALSE     FALSE       TRUE FALSE
[57,] FALSE  FALSE FALSE      FALSE      FALSE     FALSE      FALSE FALSE
[58,] FALSE  FALSE  TRUE      FALSE      FALSE     FALSE       TRUE FALSE
[59,] FALSE  FALSE FALSE      FALSE      FALSE     FALSE       TRUE  TRUE
[60,] FALSE  FALSE  TRUE      FALSE      FALSE     FALSE       TRUE  TRUE
[61,] FALSE  FALSE  TRUE      FALSE      FALSE     FALSE      FALSE FALSE
[62,] FALSE  FALSE FALSE      FALSE      FALSE     FALSE       TRUE FALSE
[63,] FALSE  FALSE FALSE      FALSE      FALSE     FALSE      FALSE FALSE
[64,] FALSE  FALSE FALSE      FALSE      FALSE     FALSE      FALSE FALSE
[65,] FALSE  FALSE  TRUE      FALSE      FALSE     FALSE       TRUE FALSE
[66,] FALSE  FALSE FALSE      FALSE      FALSE     FALSE      FALSE FALSE
[67,] FALSE  FALSE  TRUE      FALSE      FALSE     FALSE      FALSE FALSE
[68,] FALSE  FALSE FALSE      FALSE      FALSE     FALSE      FALSE FALSE
[69,] FALSE  FALSE FALSE      FALSE      FALSE     FALSE       TRUE FALSE
[70,] FALSE  FALSE FALSE      FALSE      FALSE     FALSE       TRUE FALSE
[71,] FALSE  FALSE FALSE      FALSE      FALSE     FALSE       TRUE FALSE
[72,] FALSE  FALSE  TRUE      FALSE      FALSE     FALSE       TRUE FALSE
[73,] FALSE  FALSE  TRUE      FALSE      FALSE     FALSE       TRUE FALSE
[74,] FALSE  FALSE  TRUE      FALSE      FALSE     FALSE       TRUE FALSE
[75,] FALSE  FALSE FALSE      FALSE      FALSE     FALSE       TRUE FALSE
[76,] FALSE  FALSE  TRUE      FALSE      FALSE     FALSE       TRUE FALSE
[77,] FALSE  FALSE FALSE      FALSE      FALSE     FALSE       TRUE FALSE
[78,] FALSE  FALSE FALSE      FALSE      FALSE     FALSE       TRUE FALSE
[79,] FALSE  FALSE FALSE      FALSE      FALSE     FALSE       TRUE FALSE
[80,] FALSE  FALSE FALSE      FALSE      FALSE     FALSE       TRUE FALSE
[81,] FALSE  FALSE FALSE      FALSE      FALSE     FALSE       TRUE  TRUE
[82,] FALSE  FALSE FALSE      FALSE      FALSE     FALSE       TRUE FALSE
[83,] FALSE   TRUE  TRUE      FALSE      FALSE     FALSE       TRUE FALSE
[84,] FALSE   TRUE  TRUE      FALSE      FALSE     FALSE       TRUE FALSE
[85,] FALSE   TRUE  TRUE      FALSE      FALSE     FALSE       TRUE FALSE
[86,] FALSE   TRUE  TRUE      FALSE      FALSE     FALSE       TRUE FALSE
[87,] FALSE   TRUE  TRUE      FALSE      FALSE     FALSE       TRUE FALSE
      gender homeworld species films vehicles starships
 [1,]  FALSE     FALSE   FALSE FALSE    FALSE     FALSE
 [2,]  FALSE     FALSE   FALSE FALSE    FALSE     FALSE
 [3,]  FALSE     FALSE   FALSE FALSE    FALSE     FALSE
 [4,]  FALSE     FALSE   FALSE FALSE    FALSE     FALSE
 [5,]  FALSE     FALSE   FALSE FALSE    FALSE     FALSE
 [6,]  FALSE     FALSE   FALSE FALSE    FALSE     FALSE
 [7,]  FALSE     FALSE   FALSE FALSE    FALSE     FALSE
 [8,]  FALSE     FALSE   FALSE FALSE    FALSE     FALSE
 [9,]  FALSE     FALSE   FALSE FALSE    FALSE     FALSE
[10,]  FALSE     FALSE   FALSE FALSE    FALSE     FALSE
[11,]  FALSE     FALSE   FALSE FALSE    FALSE     FALSE
[12,]  FALSE     FALSE   FALSE FALSE    FALSE     FALSE
[13,]  FALSE     FALSE   FALSE FALSE    FALSE     FALSE
[14,]  FALSE     FALSE   FALSE FALSE    FALSE     FALSE
[15,]  FALSE     FALSE   FALSE FALSE    FALSE     FALSE
[16,]  FALSE     FALSE   FALSE FALSE    FALSE     FALSE
[17,]  FALSE     FALSE   FALSE FALSE    FALSE     FALSE
[18,]   TRUE     FALSE    TRUE FALSE    FALSE     FALSE
[19,]  FALSE      TRUE   FALSE FALSE    FALSE     FALSE
[20,]  FALSE     FALSE   FALSE FALSE    FALSE     FALSE
[21,]  FALSE     FALSE   FALSE FALSE    FALSE     FALSE
[22,]  FALSE      TRUE   FALSE FALSE    FALSE     FALSE
[23,]  FALSE     FALSE   FALSE FALSE    FALSE     FALSE
[24,]  FALSE     FALSE   FALSE FALSE    FALSE     FALSE
[25,]  FALSE     FALSE   FALSE FALSE    FALSE     FALSE
[26,]  FALSE     FALSE   FALSE FALSE    FALSE     FALSE
[27,]  FALSE     FALSE   FALSE FALSE    FALSE     FALSE
[28,]  FALSE      TRUE   FALSE FALSE    FALSE     FALSE
[29,]  FALSE     FALSE   FALSE FALSE    FALSE     FALSE
[30,]  FALSE     FALSE   FALSE FALSE    FALSE     FALSE
[31,]  FALSE      TRUE   FALSE FALSE    FALSE     FALSE
[32,]  FALSE     FALSE   FALSE FALSE    FALSE     FALSE
[33,]  FALSE     FALSE   FALSE FALSE    FALSE     FALSE
[34,]  FALSE     FALSE   FALSE FALSE    FALSE     FALSE
[35,]  FALSE     FALSE   FALSE FALSE    FALSE     FALSE
[36,]  FALSE     FALSE   FALSE FALSE    FALSE     FALSE
[37,]  FALSE     FALSE   FALSE FALSE    FALSE     FALSE
[38,]  FALSE     FALSE   FALSE FALSE    FALSE     FALSE
[39,]  FALSE     FALSE   FALSE FALSE    FALSE     FALSE
[40,]  FALSE     FALSE   FALSE FALSE    FALSE     FALSE
[41,]  FALSE     FALSE   FALSE FALSE    FALSE     FALSE
[42,]  FALSE     FALSE   FALSE FALSE    FALSE     FALSE
[43,]  FALSE     FALSE   FALSE FALSE    FALSE     FALSE
[44,]  FALSE     FALSE   FALSE FALSE    FALSE     FALSE
[45,]  FALSE     FALSE   FALSE FALSE    FALSE     FALSE
[46,]  FALSE     FALSE   FALSE FALSE    FALSE     FALSE
[47,]  FALSE     FALSE   FALSE FALSE    FALSE     FALSE
[48,]  FALSE     FALSE   FALSE FALSE    FALSE     FALSE
[49,]  FALSE     FALSE   FALSE FALSE    FALSE     FALSE
[50,]  FALSE     FALSE   FALSE FALSE    FALSE     FALSE
[51,]  FALSE     FALSE   FALSE FALSE    FALSE     FALSE
[52,]  FALSE     FALSE   FALSE FALSE    FALSE     FALSE
[53,]  FALSE     FALSE   FALSE FALSE    FALSE     FALSE
[54,]  FALSE     FALSE   FALSE FALSE    FALSE     FALSE
[55,]  FALSE     FALSE   FALSE FALSE    FALSE     FALSE
[56,]  FALSE     FALSE   FALSE FALSE    FALSE     FALSE
[57,]  FALSE     FALSE   FALSE FALSE    FALSE     FALSE
[58,]  FALSE     FALSE   FALSE FALSE    FALSE     FALSE
[59,]   TRUE     FALSE    TRUE FALSE    FALSE     FALSE
[60,]   TRUE     FALSE    TRUE FALSE    FALSE     FALSE
[61,]  FALSE     FALSE   FALSE FALSE    FALSE     FALSE
[62,]  FALSE     FALSE   FALSE FALSE    FALSE     FALSE
[63,]  FALSE     FALSE   FALSE FALSE    FALSE     FALSE
[64,]  FALSE     FALSE   FALSE FALSE    FALSE     FALSE
[65,]  FALSE     FALSE   FALSE FALSE    FALSE     FALSE
[66,]  FALSE     FALSE   FALSE FALSE    FALSE     FALSE
[67,]  FALSE     FALSE   FALSE FALSE    FALSE     FALSE
[68,]  FALSE     FALSE   FALSE FALSE    FALSE     FALSE
[69,]  FALSE     FALSE   FALSE FALSE    FALSE     FALSE
[70,]  FALSE     FALSE   FALSE FALSE    FALSE     FALSE
[71,]  FALSE     FALSE   FALSE FALSE    FALSE     FALSE
[72,]  FALSE     FALSE   FALSE FALSE    FALSE     FALSE
[73,]  FALSE     FALSE   FALSE FALSE    FALSE     FALSE
[74,]  FALSE      TRUE   FALSE FALSE    FALSE     FALSE
[75,]  FALSE     FALSE   FALSE FALSE    FALSE     FALSE
[76,]  FALSE     FALSE   FALSE FALSE    FALSE     FALSE
[77,]  FALSE     FALSE   FALSE FALSE    FALSE     FALSE
[78,]  FALSE     FALSE   FALSE FALSE    FALSE     FALSE
[79,]  FALSE     FALSE   FALSE FALSE    FALSE     FALSE
[80,]  FALSE     FALSE   FALSE FALSE    FALSE     FALSE
[81,]   TRUE     FALSE    TRUE FALSE    FALSE     FALSE
[82,]  FALSE     FALSE   FALSE FALSE    FALSE     FALSE
[83,]  FALSE      TRUE   FALSE FALSE    FALSE     FALSE
[84,]  FALSE      TRUE   FALSE FALSE    FALSE     FALSE
[85,]  FALSE      TRUE   FALSE FALSE    FALSE     FALSE
[86,]  FALSE      TRUE   FALSE FALSE    FALSE     FALSE
[87,]  FALSE      TRUE   FALSE FALSE    FALSE     FALSE

How much?

is.na(starwars) |> 
  as_tibble() |> 
  summarise(across(everything(), sum))
# A tibble: 1 × 14
   name height  mass hair_color skin_color eye_color birth_year   sex gender
  <int>  <int> <int>      <int>      <int>     <int>      <int> <int>  <int>
1     0      6    28          5          0         0         44     4      4
# ℹ 5 more variables: homeworld <int>, species <int>, films <int>,
#   vehicles <int>, starships <int>

The naniar package

The naniar package vis_miss()

msleep |> vis_miss()

The naniar package vis_miss()

Use arrange() to get more information ::: {.cell output-location=‘fragment’}

msleep |> 
  arrange(sleep_cycle) |> 
  vis_miss()

:::

The naniar package gg_miss_upset()

gg_miss_upset(msleep)

Thanks!