We will use regression analysis to describe the relationship between one set of variables(the dependent variables), and another set of variables(the independent or explanatory variables)^1 and accomplish the final project with the Boston house dataset.

When there is only one explanatory variable, it is called simple regression.

We will use regression analysis instead of the t-test, ANOVA, and correlation. Details about linear model.

Import libraries

import numpy as np
import pandas as pd
import statsmodels.api as sm

Fetch data

ratings_url = 'https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBMDeveloperSkillsNetwork-ST0151EN-SkillsNetwork/labs/teachingratings.csv'
ratings_df = pd.read_csv(ratings_url)

Regression with T-test

Question:

Using the teachers rating data set, does gender affect teaching evaluation rates?

State the hypothesis:

• H0: Gender has no effect on teaching evaluation scores
• H1: Gender has an effect on teaching evaluation scores

The alpha level is 0.05.

## X is the input variables (or independent variables)
X = ratings_df['female']
## y is the target/dependent variable
y = ratings_df['eval']
## add an intercept (beta_0) to our model
X = sm.add_constant(X) 

model = sm.OLS(y, X).fit()
predictions = model.predict(X)

# Print out the statistics
model.summary()

Results: R-squared: 0.022, F-statistic: 10.56, Prob (F-statistic): 0.00124

Conclusion: Like the t-test, the p-value is less than the alpha (α) level, so we reject the null hypothesis.

There is significant evidence to prove that there is a difference in mean evaluation scores based on gender.

"The coefficient -0.1680 means that females get 0.168 scores less than men."

OLS is the ordinary least squares(OLS).

Regression with ANOVA

Question: Using the teachers' rating data set, does beauty score for instructors differ by age?

We group the data into 3 age groups, and state the hypothesis.

• H0: u1=u2=u3, the three population means are equal
• H1: At least one of the means differ

# Seperate
ratings_df.loc[(ratings_df['age'] <= 40), 'age_group'] = '40 years and younger'
ratings_df.loc[(ratings_df['age'] > 40)&(ratings_df['age'] < 57), 'age_group'] = 'between 40 and 57 years'
ratings_df.loc[(ratings_df['age'] >= 57), 'age_group'] = '57 years and older'

# Regression
from statsmodels.formula.api import ols
lm = ols('beauty ~ age_group', data = ratings_df).fit()
table= sm.stats.anova_lm(lm)
print(table)

The p-value = 4.322549e-08, we will reject the null hypothesis. There is significant evidence that at least one of the means differs.

Correlation

Question: Using the teachers' rating data set, what is the correlation between the number of students who participated in the evaluation survey and evaluation scores?

X = ratings_df['students']
y = ratings_df['eval']

X = sm.add_constant(X)
model = sm.OLS(y, X).fit()
predictions = model.predict(X)

model.summary()

R-squared: 0.001, R is math.sqrt(0.001) which is about 0.03. The correlation coefficient is 0.03. So there is a very weak correlation between the number of students who participated in the evaluation survey and evaluation scores

Here is the source code.

Final project

Project Scenario: You are a Data Scientist with a housing agency in Boston MA, you have been given access to a previous dataset on housing prices derived from the U.S. Census Service to present insights to higher management. Based on your experience in Statistics, what information can you provide them to help with making an informed decision? Upper management will like to get some insight into the following.

• Is there a significant difference in the median value of houses bounded by the Charles river or not?
• Is there a difference in median values of houses of each proportion of owner-occupied units built before 1940?
• Can we conclude that there is no relationship between Nitric oxide concentrations and the proportion of non-retail business acres per town?
• What is the impact of an additional weighted distance to the five Boston employment centers on the median value of owner-occupied homes?

Here are the solutions.

Next, I will go through Data Science For Beginnersof MS and the book named Introduction to Probability for Data Science

Change Log

2022-04-28 Jeremy initialized.

This week, we will learn how "to conduct a hypothesis test on a population mean, how to formulate a decision rule for testing a hypothesis, how to conduct a hypothesis test on a difference in two or more population means, and how to distinguish between correlation tests for two continuous variables and two categorical variables. ^1"

When you are evaluating a hypothesis, you need to account for both the variability in your sample and how large your sample is ^2.

Hypothesis testing is used to compare two or more groups or examine associations between variables^2. For example, we want to know whether gender affects teaching evaluation rates in the teachers' ratings data set. To evaluate this, we could compare the evaluation scores of the male teachers and that of female teachers.

There are five steps in Hypothesis Testing^2.

• Specify the Null Hypothesis
• Specify the Alternative Hypothesis
• Set the significance level(alpha usually 0.05)
• Calculate the test statistics and corresponding P-value
• Drawing a conclusion

We can use z-test or t-test to conduct the hypothesis test.

"A z-test is a statistical test used to determine whether two population means are different when the variances are known and the sample size is large." ^3

A t-test, like the z-test, is used to determine if there is a significant difference between the means of two groups with a small sample size. "For a large sample size, statisticians use a z-test. Other testing options include the chi-square test and the f-test."^4

There are some assumptions^1 that must be met when we use the t-test for independent samples.

• One independent, categorical variable with two levels or group
• One dependent continuous variable
• Independence of the observations. Each subject should belong to only one group. There is no relationship between the observations in each group.
• The dependent variable must follow a normal distribution, bell-shaped distribution curve.
• Assumption of homogeneity of variance

There are 3 types of t-tests, correlated/paired t-test, equal variance/pooled t-test, and unequal variance t-test, categorized as dependent and independent t-tests.^4

How to choose, we can see the screenshot from Investopedia.

Image by Julie Bang © Investopedia 2019

Then how to deal with rejections and tails? See the note about alpha and p-value in week 2

There is an example to use a t-test. "Using the teachers' rating data set, genders affect teaching evaluation rates?"

import numpy as np
import pandas as pd
import scipy.stats as stats
import matplotlib.pyplot as plt
import seaborn as sns

ratings_url = 'https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBMDeveloperSkillsNetwork-ST0151EN-SkillsNetwork/labs/teachingratings.csv'
ratings_df = pd.read_csv(ratings_url)

Let's state the hypothesis:

• H0: µ1 = µ2 ("there is no difference in evaluation scores between males and females")

• H1: µ1 ≠ µ2 ("there is a difference in evaluation scores between males and females")

# Whether normal distribution
ax = sns.distplot(ratings_df['eval'],
                  bins=20,
                  kde=True,
                  color='green',
                  hist_kws={'linewidth': 15, 'alpha': 1})
ax.set(xlabel='Normal Distribution', ylabel='Frequency')

# Assume it is normal
# Equal variance？
mask_female = ratings_df['gender'] == 'female'
mask_male = ratings_df['gender'] == 'male'
stats.levene(ratings_df[mask_female]['eval'],
                   ratings_df[mask_male]['eval'], center='mean')

LeveneResult(statistic=0.1903292243529225, pvalue=0.6628469836244741)

We have a t-test called Levene's test to determine the equality of variances. The null hypothesis of the Levene's test is that population variances are equal if the p-value of the test is less 0.05, reject the null hypothesis of equal variances and assume that the variances are unequal.

The p-value is greater than 0.05, we assume equality of variance.

# T-test
stats.ttest_ind(ratings_df[mask_female]['eval'],
                ratings_df[mask_male]['eval'], equal_var=True)

Ttest_indResult(statistic=-3.249937943510772, pvalue=0.0012387609449522217)

The p-value is less than 0.05/2, we reject the null hypothesis H0. So the difference in evaluation scores between males and females is statistically significant.

When we conduct a test with three or more means, one must use an analysis of variance(ANOVA).

"Analysis of variance (ANOVA) is a collection of statistical models and their associated estimation procedures (such as the "variation" among and between groups) used to analyze the differences among means".^5

Example:

# ANOVA: Using the teachers' rating data set, 
#      does beauty score for instructors differ by age?
#  we group the data into categories as the one-way
#  ANOVA can't work with continuous variable
mask_younger = ratings_df['age'] <= 40
mask_between = (ratings_df['age'] > 40) & (ratings_df['age'] < 57)
mask_older = ratings_df['age'] >= 57

ratings_df.loc[mask_younger, 'age_group'] = '40 years and younger'
ratings_df.loc[mask_between, 'age_group'] = 'between 40 and 57 years'
ratings_df.loc[mask_older, 'age_group'] = '57 years and older'

State the hypothesis:

• H0: µ1 = µ2 = µ3 (the three population means are equal)
• H1: At least one of the means differ

Test for equality of variance:

mask1 = ratings_df['age_group'] == '40 years and younger'
mask2 = ratings_df['age_group'] == 'between 40 and 57 years'
mask3 = ratings_df['age_group'] == '57 years and older'
stats.levene(ratings_df[mask1]['beauty'],
             ratings_df[mask2]['beauty'],
             ratings_df[mask3]['beauty'],
             center='mean')

LeveneResult(statistic=8.60005668392585, pvalue=0.0002153661809934714)

Since the p-value is less than 0.05, we reject the null hypothesis which means the variances are not equal.

ANOVA test:

younger = ratings_df[mask1]['beauty']
between = ratings_df[mask2]['beauty']
older = ratings_df[mask3]['beauty']

# run a one-way ANOVA
f_statistic, p_value = stats.f_oneway(younger, between, older)
print("F_Statistic: {0}, P-value: {1}".format(f_statistic, p_value))

F_Statistic: 17.597558611010122, P-value: 4.3225489816137975e-08

Since the p-value is less than 0.05/2, we will reject the null hypothesis as there is significant evidence that at least one of the means differs.

We can use Chi-square to test whether there is an association between tenure and gender(categorical data sets).

State the hypothesis:

• H0: The proportion of teachers who are tenured is independent of gender
• H1: The proportion of teachers who are tenured is associated with gender

# Cross-tab table
cont_table = pd.crosstab(ratings_df['tenure'], ratings_df['gender'])
stats.chi2_contingency(cont_table, correction = True)

Output:

(2.20678166999886, 0.1374050603563787, 1, array([[ 42.95896328, 59.04103672], [152.04103672, 208.95896328]]))

2.2 is the 𝜒2 value, and 0.137 is the p-value.

Conclusion: Since the p-value is greater than 0.05/2, we fail to reject the null hypothesis. As there is no sufficient evidence that teachers are tenured because of gender.

Next are the correlation tests.

Q: Correlation, Using the teachers' rating dataset, Is teaching evaluation score correlated with beauty score?

State the hypothesis:

• H0: Teaching evaluation score is not correlated with beauty score
• H1: Teaching evaluation score is correlated with beauty score

stats.pearsonr(ratings_df['beauty'], ratings_df['eval'])

Output: (0.18903909084045212, 4.247115419812614e-05)

Conclusion: Since the p-value < 0.05/2, we reject the null hypothesis and conclude that there is a relationship between beauty and teaching evaluation score.

Here is the source code.

Change Log

2022-04-23 Jeremy initialized.

Content: the basic concepts and application of probability and probability distributions.

Objectives: Calculate probabilities given a normal density, State the null and alternative hypothesis when performing tests, and understand the different bell-shaped distributions.

Random Numbers and Probability Distributions

The random variable is a quantity whose possible values depend in some clearly defined way on a set of some random events ^1.

The probability is a measure between zero and one for the likelihood that something or some event might occur ^1.

The course introduces a game ^2 that uses dice to understand random variables and probability.

Let's play the dice game ^3. Let's roll two dice 10000 times. If we look at the distribution of the sum of two dice, we can see it's a normal distribution.

# Two dice game
import matplotlib.pyplot as plt
import seaborn as sns
import numpy as np

np.random.seed(1738)

d1 = np.arange(1, 7)
d2 = np.arange(1, 7)

dice_1 = []
dice_2 = []
sums = []

for _ in range(10000):
    roll_1 = np.random.choice(d1)
    roll_2 = np.random.choice(d2)

    dice_1.append(roll_1)
    dice_2.append(roll_2)

    sums.append(roll_1 + roll_2)

sns.countplot(sums)

The mean is 7.0258, and the standard deviation is about 2.3923

np.mean(sums)
np.std(sums)

Hypothesis

This part introduced the statistical hypothesis testing and how to state our hypothesis, which I learned at the university. We should set the null hypothesis and the alternative hypothesis.

Alpha and P-value

Alpha and p-value are commonly used terms in statistical analysis.

Alpha (α) is known as the significance level (显著性水平 in Chinese). "It is the probability of rejecting the null hypothesis when the null hypothesis is true. The value often used is 5% ^4". This means that there is a 5% chance that we will accept the alternative hypothesis when the null hypothesis is true.

The rejection regions for different kinds of tests are shown in this image below(screenshot from the course.) ^4.

P-value is a calculated value and an output you get as part of conducting your hypothesis test. The p-value can be interpreted as the probability of getting a result that is as extreme or more extreme when the null hypothesis is true i.e. the likelihood of observing that particular sample value if the null hypothesis were true. ^4

For example, in a one-tailed test(right tail), we conduct a test and get a p-value of 0.03 which means that there is a 3% chance of obtaining a value of $$\mu_0$$ or more than $$ \mu_0$$

If the alpha(the significance value) is 5%, we will reject the null hypothesis, because 3% is less than 5%. if the alpha is 2%, we will fail to reject the null hypothesis because 3% is greater than 2%.

Normal Distribution

The function of the normal distribution:

$$ f(x,\mu,\sigma) = \frac{1}{\sigma\sqrt{2\pi}}e^{-\frac{(x-\mu)^2}{2{\sigma}^2}} $$

A random variable: $$x$$
The mean of the variables: $$\mu$$
Standard deviation: $$\sigma$$

The function of the standard normal distribution:

$$ \sigma = 1,
\mu = 0 $$ $$ f(x,0,1) = \frac{1}{\sqrt{2\pi}}e^{-(\frac{x^2}{2})} $$

# Plot between -4 and 4 with 0.1 steps
x_axis = np.arange(-4, 4, 0.1)
# Mean = 0, SD= 1.
plt.plot(x_axis, sta.norm.pdf(x_axis, 0, 1))
plt.show()

CDF: Cumulative distribution function (累积分布函数) PDF: Probability density function (概率密度函数)

We can standardize the normal distribution to the standard normal distribution.

The method:

$$ z = \frac{x - \mu}{\sigma} $$ $$ f(z) = \frac{1}{\sqrt{2\pi}}e^{-\frac{z^2}{2}} $$

T-distribution

The t-distribution describes the standardized distances of sample means to the population mean when the population standard deviation is not known, and the observations come from a normally distributed population. ^5

The normal distribution describes the mean for the population, whereas the T-distribution describes the mean of samples drawn from a population. ^6

When the sample size increases, the t-distribution becomes more similar to the normal distribution^5.

Probability getting from CDF

eval_mean = round(ratings_df['eval'].mean(), 3)
eval_std = round(ratings_df['eval'].std(), 3)
prob0 = scipy.stats.norm.cdf((4.5 - eval_mean)/eval_std)
print(1 - prob0)

The variance prob0 is the probability of receiving an evaluation score of less than 4.5.

x1 = 3.5
x2 = 4.2
prob1 = scipy.stats.norm.cdf((x1 - eval_mean)/eval_std)
prob2 = scipy.stats.norm.cdf((x2 - eval_mean)/eval_std)
print(prob2 - prob1)

The variance prob1 is the probability of receiving an evaluation score of less than 3.5 (x1). The variance prob2 is the probability of receiving an evaluation score of less than 4.2 (x2). So the difference between prob2 and prob1 is the probability of receiving an evaluation score greater than 3.5 and less than 4.2.

Standart Normal Table

You can use the z-table to find a set of “less-than” probabilities for a wide range of z-values. ^7

I used this table. It is so familiar. Haha. And I know how to use it.

Experiments: Probability Distributions

In this part, we can go over some hands-on exercises using Python to do some descriptive statistics. Here is the source code. We can also use Google Colab to execute.

Change Log

2022-04-20 Jeremy Initialized.

The learning objectives1 of week 2 are

• Interpret graphical summaries of data
• Create data visualization in Python
• Effectively choose the right chart type for the audience and data type.

Charts and graphs in statistics

Visualization Fundamentals

• Use bar charts or column chats to compare items with a few categories

sns.set(style="whitegrid")
ax = sns.barplot(x="division", y="eval", data=division_eval)

• Use line charts or line plots to compare behaviors over time(not many time periods, use columns or other approaches)

# Seaborn X-axis as index
fig = sns.lineplot(data=df_clean.reset_index(), x='date', y='value', palette=['red'])
fig.set_xlabel("Date")
fig.set_ylabel("Page Views")
fig.set_title("Daily freeCodeCamp Forum Page Views 5/2016-12/2019")

• Use a scatter plot to depict relationships between two continuous variables

ax = sns.scatterplot(x="age", y="eval", data=ratings_df,
                    hue="gender")

• Use a bubble chart to depict two variables on the x and y-axis and the third variable(size of the circle)

Example: 2

# scatter plot with scatter() function
# transparency with "alpha"
# bubble size with "s"
plt.scatter('X', 'Y', 
             s='bubble_size',
             alpha=0.5, 
             data=df)
plt.xlabel("X", size=16)
plt.ylabel("y", size=16)
plt.title("Bubble Plot with Matplotlib", size=18

• Use a histogram(chart or line) to depict the distribution of the data set

sns.histplot(x="gender", hue="tenure", data=ratings_df, multiple="dodge")

sns.distplot(ratings_df[mask_female]['eval'], color='green', kde=False)
sns.distplot(ratings_df[mask_male]['eval'], color='red', kde=False)
plt.show()

• Use a pie chart to show the composition3

import matplotlib.pyplot as plt

# Pie chart, where the slices will be ordered and plotted counter-clockwise:
labels = 'Frogs', 'Hogs', 'Dogs', 'Logs'
sizes = [15, 30, 45, 10]
explode = (0.1, 0.1, 0, 0)  # only "explode" the 2nd slice (i.e. 'Hogs')

fig1, ax1 = plt.subplots()
ax1.pie(sizes, explode=explode, labels=labels, autopct='%1.1f%%',
        shadow=True, startangle=90)
ax1.axis('equal')  # Equal aspect ratio ensures that pie is drawn as a circle.

plt.show()

• Use stacked columns to show the composition that changes over time(few periods)
• Use stacked area charts (Many periods)

Statistics by Groups

Identify and eliminate "duplicates" data

nonduplicates_ratings_df = ratings_df.drop_duplicates(subset=['prof'])

Statistics Charts

Box plot: Displaying mean, median quartile, and outliers.

ax = sns.boxplot(x="credits", y="beauty", data=ratings_df)

From bottom to top: minimum, first quartile, median, third quartile, maximum. The range between the first quartile and the third quartile is interquartile range(IQR).

For distribution and variance, see the distribution and histogram above.

Experiments: Data Visualization

In this part, we can go over some hands-on exercises using Python to do some descriptive statistics. Here is the source code. You can use Google Colab to execute.

Change Log

2022.04.18 Jeremy Initialized.

The contents of this week:

• Course Introduction and Python Basics
  • Instructors and Course Overview
  • Python Packages for Data Science
  • Basics of Jupyter Notebooks and Python Review
• Understanding the basics of Descriptive Statistics
  • Statistics introduction
  • Types of Data
  • Measure of Central Tendency
  • Measure of Dispersion
• Experiments: Descriptive Statistics

Course Introduction and Python Basics

The instructors are Murtaza Haider(Associate professor @ Ryerson University in Toronto) and Aije Egwaikhide(Senior Data Scientist and Statisticians @ IBM).

This course consists of five modules: Introduction and Descriptive Statistics, Data Visualization, Introduction to Probability Distribution, Hypothesis Testing, and Regression Analysis.

The main packages relevant to analysis in python are divided into 3 groups:

• Scientific Computing Libraries
  • Pandas, offers data structure and tools
  • NumPy, provides a multidimensional array object, various derived objects and so on.
  • SciPy, provides fundamental algorithms for scientific computing in Python.
• Visualization Libraries
  • Matplotlib, makes graphs and plots
  • Seaborn, high level visualization library based on Matplotlib.
• Algorithmic Libraries
  • scikit-learn
  • statsmodels

I have experience in Python, Pandas, NumPy, Matplolib, and scikit-learn. So these are not unfamiliar to me. Here is my repository gopython.

The basics of Descriptive Statistics

Types of Data

In our daily lives, we are surrounded by data and statistics. The most common data would be a cross-sectional data, "which is basically looking at a measurement taken at one point in time"1.

Compared to the cross-sectional data, there are panel or cross-sectional panel data, "which is essentially asking the same group of individuals the same questions repeatedly over time.1"

Another data type is the time series data, as the name indicates, it is "a series of data points indexed (or listed or graphed) in time order.2:Wikipedia"

Measure of Central Tendency

We can use mean, median, and mode to measure the central tendency of the data. In Pandas, dataframe.mean() and dataframe.median() can help us to get the mean and median of the data.

"The mean of a set of observations is the arithmetic average of the values.3" It is also called the arithmetic mean. There are other types of means.

"The median is the middle number in a sorted, ascending, or descending, list of numbers and can be more descriptive of that data set than the average.4"

For example, the mean(or average) of this list of numbers [10, 15, 20, 25, 30] is (10 + 15 + 20 + 25 + 30) / 5 = 20. The median is 20. If the list of numbers is [10, 10, 11, 12, 13, 14], the median is (11 + 12) / 2 = 11.5.

Measure of Dispersion

The common measures of dispersion are standard deviation and variance.

There are sample variance and population variance. We should notice the difference between sample variance(variance of the sampling distribution of the sample mean) and Population Variance.

The standard deviation is the square root of the variance. So the standard deviation of the population is different from the standard deviation of the sample.

Experiments: Descriptive Statistics

In this part, we can go over some hands-on exercises using Python to do some descriptive statistics. Here is the source code. You can use Google Colab to execute.

Someone said: "If you want to make the maximum amount of money possible, if you want to get rich over your life in a deterministically predictable way, stay on the bleeding edge of trends and study technology, design, and art—become really good at something." I agree with him totally.

My interests are very diverse. Now I am learning Data Science, Machine Learning, and Artificial Intelligence. My background is in Mechanical Engineering(BS) and Vehicle Engineering(MS).

I have successfully completed the freeCodeCamp.org Data Analysis with Python. Here is the certification.

Next, I will learn the IBM Statistics for Data Science with Python Course in Coursera. 😁🤩

May peace be with you.

Jeremy
2022.04.11 init
2022.04.13 Certification & Coursera course