Agregats sous R et Python dans Rstudio

class: center, middle, inverse, title-slide

# Agregats sous R et Python dans Rstudio
### Sebastien Foulle
### 07/09/2018

---

pre {
  white-space: pre-wrap;
  box-shadow: 10px 5px 5px lightblue;
}
</style>

## Contenu du document

On presente dans les quelques slides suivantes les operations d'agregation usuelles en R avec *dplyr* et en Python avec *pandas*.

Prerequis pour produire cette presentation **xaringan** avec Rstudio :

- une distribution Python (Anaconda convient tres bien)
- le package R *reticulate* qui permet d'utiliser des objets R dans des chunks Python (syntaxe *r.mon_objet_r*) et inversement (syntaxe *py$mon_objet_python*)

.pull-left[

```r
library("reticulate")
# chemin vers l'executable python
use_python("C:/Users/Sebastien/Anaconda3/python.exe")

library("dplyr")
tips = reshape2::tips[75:78,-1]
```

```python
import numpy as np
import pandas as pd
```
]

---

## Les donnees

.pull-left[
<p style="text-align:center"><svg aria-hidden="true" role="img" viewBox="0 0 581 512" style="height:60px;width:68.09px;vertical-align:-0.125em;margin-left:auto;margin-right:auto;font-size:inherit;fill:blue;overflow:visible;position:relative;"><path d="M581 226.6C581 119.1 450.9 32 290.5 32S0 119.1 0 226.6C0 322.4 103.3 402 239.4 418.1V480h99.1v-61.5c24.3-2.7 47.6-7.4 69.4-13.9L448 480h112l-67.4-113.7c54.5-35.4 88.4-84.9 88.4-139.7zm-466.8 14.5c0-73.5 98.9-133 220.8-133s211.9 40.7 211.9 133c0 50.1-26.5 85-70.3 106.4-2.4-1.6-4.7-2.9-6.4-3.7-10.2-5.2-27.8-10.5-27.8-10.5s86.6-6.4 86.6-92.7-90.6-87.9-90.6-87.9h-199V361c-74.1-21.5-125.2-67.1-125.2-119.9zm225.1 38.3v-55.6c57.8 0 87.8-6.8 87.8 27.3 0 36.5-38.2 28.3-87.8 28.3zm-.9 72.5H365c10.8 0 18.9 11.7 24 19.2-16.1 1.9-33 2.8-50.6 2.9v-22.1z"/></svg></p>

```r
tips
```

```
    tip    sex smoker  day   time size
75 2.20 Female     No  Sat Dinner    2
76 1.25   Male     No  Sat Dinner    2
77 3.08   Male    Yes  Sat Dinner    2
78 4.00   Male     No Thur  Lunch    4
```
]

.pull-right[
<p style="text-align:center"><svg aria-hidden="true" role="img" viewBox="0 0 448 512" style="height:60px;width:52.5px;vertical-align:-0.125em;margin-left:auto;margin-right:auto;font-size:inherit;fill:gold;overflow:visible;position:relative;"><path d="M439.8 200.5c-7.7-30.9-22.3-54.2-53.4-54.2h-40.1v47.4c0 36.8-31.2 67.8-66.8 67.8H172.7c-29.2 0-53.4 25-53.4 54.3v101.8c0 29 25.2 46 53.4 54.3 33.8 9.9 66.3 11.7 106.8 0 26.9-7.8 53.4-23.5 53.4-54.3v-40.7H226.2v-13.6h160.2c31.1 0 42.6-21.7 53.4-54.2 11.2-33.5 10.7-65.7 0-108.6zM286.2 404c11.1 0 20.1 9.1 20.1 20.3 0 11.3-9 20.4-20.1 20.4-11 0-20.1-9.2-20.1-20.4.1-11.3 9.1-20.3 20.1-20.3zM167.8 248.1h106.8c29.7 0 53.4-24.5 53.4-54.3V91.9c0-29-24.4-50.7-53.4-55.6-35.8-5.9-74.7-5.6-106.8.1-45.2 8-53.4 24.7-53.4 55.6v40.7h106.9v13.6h-147c-31.1 0-58.3 18.7-66.8 54.2-9.8 40.7-10.2 66.1 0 108.6 7.6 31.6 25.7 54.2 56.8 54.2H101v-48.8c0-35.3 30.5-66.4 66.8-66.4zm-6.7-142.6c-11.1 0-20.1-9.1-20.1-20.3.1-11.3 9-20.4 20.1-20.4 11 0 20.1 9.2 20.1 20.4s-9 20.3-20.1 20.3z"/></svg></p>

```python
tips = r.tips; tips
```

```
     tip     sex smoker   day    time  size
75  2.20  Female     No   Sat  Dinner     2
76  1.25    Male     No   Sat  Dinner     2
77  3.08    Male    Yes   Sat  Dinner     2
78  4.00    Male     No  Thur   Lunch     4
```
]

---

## Le type des donnees

```r
str(tips)
```

```
'data.frame':	4 obs. of  6 variables:
 $ tip   : num  2.2 1.25 3.08 4
 $ sex   : Factor w/ 2 levels "Female","Male": 1 2 2 2
 $ smoker: Factor w/ 2 levels "No","Yes": 1 1 2 1
 $ day   : Factor w/ 4 levels "Fri","Sat","Sun",..: 2 2 2 4
 $ time  : Factor w/ 2 levels "Dinner","Lunch": 1 1 1 2
 $ size  : int  2 2 2 4
```
]

```python
tips.info()
```

```
<class 'pandas.core.frame.DataFrame'>
Index: 4 entries, 75 to 78
Data columns (total 6 columns):
 #   Column  Non-Null Count  Dtype   
---  ------  --------------  -----   
 0   tip     4 non-null      float64 
 1   sex     4 non-null      category
 2   smoker  4 non-null      category
 3   day     4 non-null      category
 4   time    4 non-null      category
 5   size    4 non-null      int32   
dtypes: category(4), float64(1), int32(1)
memory usage: 672.0+ bytes
```
]

---

## Agregats simples

```r
tips %>% summarise(moy = mean(tip))
```

```
     moy
1 2.6325
```
]

```python
tips[["tip"]].mean()
```

```
tip    2.6325
dtype: float64
```
]

---

## Agregats simples pour toutes les colonnes numeriques

```r
tips %>% summarise_if(is.numeric, mean)
```

```
     tip size
1 2.6325  2.5
```
]

```python
tips.mean()
```

```
tip     2.6325
size    2.5000
dtype: float64
```
]

---

## Application : % de valeurs manquantes par colonne

```r
tips2 = tips; tips2[1,1] = NA;
# en R base : colMeans(is.na(tips2))
is.na(tips2) %>% as.data.frame %>% summarise_all(mean)
```

```
   tip sex smoker day time size
1 0.25   0      0   0    0    0
```
]

```python
tips2 = r.tips2;
tips2.isna().mean().reset_index(name = "pct_na")
```

```
    index  pct_na
0     tip    0.25
1     sex    0.00
2  smoker    0.00
3     day    0.00
4    time    0.00
5    size    0.00
```
]

---

## Agregats par groupe

```r
tips %>% group_by(sex,smoker) %>% summarise(moy=mean(tip))
```

```
# A tibble: 3 x 3
# Groups:   sex [2]
  sex    smoker   moy
  <fct>  <fct>  <dbl>
1 Female No      2.2 
2 Male   No      2.62
3 Male   Yes     3.08
```
]

```python
tips.groupby(["sex", "smoker"], observed = True)["tip"].mean().reset_index(name = "moy")
```

```
      sex smoker    moy
0  Female     No  2.200
1    Male     No  2.625
2    Male    Yes  3.080
```
]

---

## Application : table de contingence

```r
# en bref : tips %>% count(sex, smoker, name = "freq")
tips %>% group_by(sex,smoker) %>% summarise(freq =n())
```

```
# A tibble: 3 x 3
# Groups:   sex [2]
  sex    smoker  freq
  <fct>  <fct>  <int>
1 Female No         1
2 Male   No         2
3 Male   Yes        1
```
]

```python
tips.groupby(["sex", "smoker"], observed = True).size().reset_index(name = "freq")
```

```
      sex smoker  freq
0  Female     No     1
1    Male     No     2
2    Male    Yes     1
```
]

---

## Agregats multiples par groupe

```r
tips %>% group_by(sex,smoker) %>% summarise(moy=mean(tip), somme=sum(size))
```

```
# A tibble: 3 x 4
# Groups:   sex [2]
  sex    smoker   moy somme
  <fct>  <fct>  <dbl> <int>
1 Female No      2.2      2
2 Male   No      2.62     6
3 Male   Yes     3.08     2
```
]

```python
tips.groupby(["sex", "smoker"], observed = True).agg(moy=('tip', 'mean'), somme=('size', 'sum')).reset_index()
```

```
      sex smoker    moy  somme
0  Female     No  2.200      2
1    Male     No  2.625      6
2    Male    Yes  3.080      2
```
]

---

## Agregats par groupe ajoutes dans la table initiale

```r
options(pillar.sigfig = 7)
tips %>% group_by(sex,smoker) %>% mutate(moy=mean(tip))
```

```
# A tibble: 4 x 7
# Groups:   sex, smoker [3]
    tip sex    smoker day   time    size   moy
  <dbl> <fct>  <fct>  <fct> <fct>  <int> <dbl>
1  2.2  Female No     Sat   Dinner     2 2.2  
2  1.25 Male   No     Sat   Dinner     2 2.625
3  3.08 Male   Yes    Sat   Dinner     2 3.08 
4  4    Male   No     Thur  Lunch      4 2.625
```
]

```python
tips["moy"] = tips.groupby(["sex", "smoker"], observed = True)["tip"].transform("mean");tips
```

```
     tip     sex smoker   day    time  size    moy
75  2.20  Female     No   Sat  Dinner     2  2.200
76  1.25    Male     No   Sat  Dinner     2  2.625
77  3.08    Male    Yes   Sat  Dinner     2  3.080
78  4.00    Male     No  Thur   Lunch     4  2.625
```
]

---

## Application : on filtre sur les pourboires au-dessus de la moyenne

```r
tips %>% mutate(moy = mean(tip)) %>% filter(tip > moy)
```

```
    tip  sex smoker  day   time size    moy
77 3.08 Male    Yes  Sat Dinner    2 2.6325
78 4.00 Male     No Thur  Lunch    4 2.6325
```
]

```python
tips["zero"] = 0;
tips["moy"]=tips.groupby("zero")["tip"].transform("mean")
tips.query('tip > moy')
```

```
     tip   sex smoker   day    time  size  zero     moy
77  3.08  Male    Yes   Sat  Dinner     2     0  2.6325
78  4.00  Male     No  Thur   Lunch     4     0  2.6325
```
]

---

## Variante : agregats par groupe avec dictionnaire + mapping

```r
vecteur = tips %>% group_by(day) %>% summarise(moy = mean(tip)) %>% tibble::deframe();vecteur
tips["moy"] = vecteur[as.character(tips$day)];tips
```

```
     Sat     Thur 
2.176667 4.000000 
    tip    sex smoker  day   time size      moy
75 2.20 Female     No  Sat Dinner    2 2.176667
76 1.25   Male     No  Sat Dinner    2 2.176667
77 3.08   Male    Yes  Sat Dinner    2 2.176667
78 4.00   Male     No Thur  Lunch    4 4.000000
```
]

```python
dico = tips.groupby("day", observed = True).tip.mean().to_dict();dico
```

```
{'Sat': 2.1766666666666667, 'Thur': 4.0}
```

```python
tips["moy"] = tips.day.map(dico); tips
```

```
     tip     sex smoker   day    time  size       moy
75  2.20  Female     No   Sat  Dinner     2  2.176667
76  1.25    Male     No   Sat  Dinner     2  2.176667
77  3.08    Male    Yes   Sat  Dinner     2  2.176667
78  4.00    Male     No  Thur   Lunch     4  4.000000
```
]