Run in Google Colab

View on GitHub

Homework 9 - Parameter Search and Feature Selection¶

In this assignment, you will be working with the cars dataset to perform parameter search and feature selection using various techniques. The goal is to explore the performance of a machine learning model by optimizing its parameters and selecting the most relevant features. We will try to predict the Make of the car based on its attributes.

Complete the missing parts in this guide.

Step 1: Load Data¶

You can load the data from the provided CSV file using pandas.

Step 2: Preprocess and split data¶

Use attributes ['Make', 'Engine HP', 'Engine Cylinders', 'Number of Doors', 'highway MPG', 'city mpg', 'MSRP'] (x) to predict the Make (y, target variable). Note that Make is a categorical variable, so you will need to use a classification model.

• You should handle missing values appropriately (suggestion: drop rows with missing values).
• You should remove Makes that have less than 20 samples.
• You should split the data into training and testing sets.

Step 3: Predict Make using a Decision Tree Classifier¶

Use a Decision Tree Classifier to predict the Make of the car. You will need to:

• Train the model on the training set.
• Evaluate the model on the test set using different metrics (use the report function from sklearn.metrics).
• Showcase the confusion matrix to visualize the performance of the model. Do you see any interesting patterns in the confusion matrix?

Step 4: Perform Parameter Search¶

Use GridSearchCV to perform a parameter search on the Decision Tree Classifier. You can use criterion, max_depth, and splitter as parameters to search over. Check the documentation for Decistion Tree Classifier to see the available parameters.

• Discuss if the model performance improved after the parameter search.

Step 5: Feature Selection¶

Use the best model from the parameter search to perform feature selection. Use feature_importances_ to get the importance of each feature in the model. You can use SelectFromModel from sklearn.feature_selection to select the most important features based on the model's feature importances.

• You can visualize the feature importances using a bar chart for better understanding.
• Discuss which features are most important and how they impact the model's performance.

Step 6: Plot ROC Curve and calculate AUC¶

Use the best model from the parameter search to plot the ROC curve and calculate the AUC (Area Under the Curve) score for one of the classes (e.g., Toyota). You can use roc_curve and auc from sklearn.metrics to do this. You can use the method predict_proba to get the probabilities for each class.

Step 7: Repeat 3-6 for Logistic Regression, KNeighborsClassifier, RandomForestClassifier, and SVM(SVC).¶

• Compare the performance of the classifiers using the same metrics as before.
• Discuss which classifier performed best and why.
• Check the documentation or ask for help if you are unsure about the parameters to use for each classifier. For instance, for KNeighborsClassifier, you can search over n_neighbors and weights.

Dataset Overview¶

The cars dataset contains several attributes of various car models, including their specifications and performance metrics.

Submission Guidelines¶

• Submit your completed notebook as a HTML export, or a PDF file.

To export to HTML, if you are on Jupyter, select File > Export Notebook As > HTML.

If you are on VSCode, you can use the Jupyter: Export to HTML command.

• Open the command palette (Ctrl+Shift+P or Cmd+Shift+P on Mac).
  • Search for Jupyter: Export to HTML.
  • Save the HTML file to your computer and submit it via Canvas.

Make sure the plots appear in the exported file. If you are using plotly or more complicated interactive plots, make sure to a bitmap backend like 'png'.

Let's start by loading several libraries that we will need for this assignment.

In [1]:

import pandas as pd
import sqlite3
from sklearn.metrics import confusion_matrix, classification_report, accuracy_score
from sklearn.model_selection import cross_val_score, GridSearchCV, train_test_split
from sklearn.feature_selection import SelectFromModel
from sklearn.linear_model import LogisticRegression
from sklearn.tree import DecisionTreeClassifier
from sklearn.ensemble import RandomForestClassifier
import matplotlib.pyplot as plt
import seaborn as sns

import pandas as pd import sqlite3 from sklearn.metrics import confusion_matrix, classification_report, accuracy_score from sklearn.model_selection import cross_val_score, GridSearchCV, train_test_split from sklearn.feature_selection import SelectFromModel from sklearn.linear_model import LogisticRegression from sklearn.tree import DecisionTreeClassifier from sklearn.ensemble import RandomForestClassifier import matplotlib.pyplot as plt import seaborn as sns

Now we need to import the data. This time we will import the carfeatures.csv from the Datasets folder. Make any adjustments to the path as necessary.

In [2]:

df = pd.read_csv("../../Datasets/carfeatures.csv")
df.head()

df = pd.read_csv("../../Datasets/carfeatures.csv") df.head()

Out[2]:

	Make	Model	Year	Engine Fuel Type	Engine HP	Engine Cylinders	Transmission Type	Driven_Wheels	Number of Doors	Market Category	Vehicle Size	Vehicle Style	highway MPG	city mpg	Popularity	MSRP
0	BMW	1 Series M	2011	premium unleaded (required)	335.0	6.0	MANUAL	rear wheel drive	2.0	Factory Tuner,Luxury,High-Performance	Compact	Coupe	26	19	3916	46135
1	BMW	1 Series	2011	premium unleaded (required)	300.0	6.0	MANUAL	rear wheel drive	2.0	Luxury,Performance	Compact	Convertible	28	19	3916	40650
2	BMW	1 Series	2011	premium unleaded (required)	300.0	6.0	MANUAL	rear wheel drive	2.0	Luxury,High-Performance	Compact	Coupe	28	20	3916	36350
3	BMW	1 Series	2011	premium unleaded (required)	230.0	6.0	MANUAL	rear wheel drive	2.0	Luxury,Performance	Compact	Coupe	28	18	3916	29450
4	BMW	1 Series	2011	premium unleaded (required)	230.0	6.0	MANUAL	rear wheel drive	2.0	Luxury	Compact	Convertible	28	18	3916	34500

We will use only a subset of the columns for this assignment. Let's select them: We want to predict Make.

In [3]:

# Our variables
x_features = [
'Engine HP',
'Engine Cylinders',
'Number of Doors',
'highway MPG',
'city mpg',
'MSRP'
]

# Target variable
y_feature = 'Make'

# Our variables x_features = [ 'Engine HP', 'Engine Cylinders', 'Number of Doors', 'highway MPG', 'city mpg', 'MSRP' ] # Target variable y_feature = 'Make'

Next, we need to drop rows with missing values among the selected columns. (You can use dropna() method from pandas with a subset parameter that restricts the operation to the selected columns.)

In [ ]:

# remove rows with missing values in x_features
df = ... # COMPLETE

# remove rows with missing values in x_features df = ... # COMPLETE

We need to remove rows where the Make has not many samples. We can use value_counts() to find the counts of each Make. I recommend to plot value counts to visualize the distribution of Makes. Let's see how many samples each Make has:

In [ ]:

# Let's see the distribution of the y_feature. We want to have classes with at least 20 samples.
categories_count = df[y_feature]._____ # COMPLETE
plt.figure(figsize=(10, 6))
categories_count.plot(kind='bar', title=f"{y_feature} Distribution")
#show horizontal line for 20 
plt.axhline(y=20, color='r', linestyle='--')
plt.xlabel(y_feature)
plt.ylabel('Count')
plt.show()

# Let's see the distribution of the y_feature. We want to have classes with at least 20 samples. categories_count = df[y_feature]._____ # COMPLETE plt.figure(figsize=(10, 6)) categories_count.plot(kind='bar', title=f"{y_feature} Distribution") #show horizontal line for 20 plt.axhline(y=20, color='r', linestyle='--') plt.xlabel(y_feature) plt.ylabel('Count') plt.show()

We need to remove any entries from Make that have less than 20 samples. One solution is getting the indices of the Makes that have less than 20 samples and then dropping those rows from the DataFrame. You can use isin() method to filter the DataFrame.

In [ ]:

# Remove classes with less than 20 samples
categories_to_remove = __ # COMPLETE
df = __ # COMPLETE

# Remove classes with less than 20 samples categories_to_remove = __ # COMPLETE df = __ # COMPLETE

Define X variables (input) and the y variable (target) in terms of the features.

In [ ]:

X = df[__] # COMPLETE
y = df[__] # COMPLETE

X = df[__] # COMPLETE y = df[__] # COMPLETE

Let's now split the dataset into training and testing sets. Let's use a test_size of 0.3.

In [ ]:

X_train, X_test, y_train, y_test = __ # COMPLETE

X_train, X_test, y_train, y_test = __ # COMPLETE

Let's train a simple DecisionTreeClassifier

In [ ]:

# Initialize classifiers
# Decision Tree first
dt_classifier = __ # COMPLETE
# Fit the model
dt_classifier.fit(__, __) # COMPLETE

# Initialize classifiers # Decision Tree first dt_classifier = __ # COMPLETE # Fit the model dt_classifier.fit(__, __) # COMPLETE

Out[ ]:

DecisionTreeClassifier()

In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.

Now let's predict and see the report for the decision tree classifier.

In [ ]:

# Predict on the test set
y_pred_dt = dt_classifier.predict(__) # COMPLETE
# Evaluate the Decision Tree model
print("Decision Tree Classifier:")
print(classification_report(__, __)) # COMPLETE

# Predict on the test set y_pred_dt = dt_classifier.predict(__) # COMPLETE # Evaluate the Decision Tree model print("Decision Tree Classifier:") print(classification_report(__, __)) # COMPLETE

Decision Tree Classifier:
               precision    recall  f1-score   support

        Acura       0.76      0.84      0.79        67
 Aston Martin       0.95      0.97      0.96        36
         Audi       0.88      0.85      0.86        91
          BMW       0.83      0.91      0.87       102
      Bentley       1.00      0.83      0.91        24
        Buick       0.64      0.69      0.67        52
     Cadillac       0.93      0.95      0.94       123
    Chevrolet       0.65      0.73      0.68       350
     Chrysler       0.78      0.72      0.75        58
        Dodge       0.83      0.84      0.83       188
         FIAT       0.78      1.00      0.88        14
      Ferrari       0.85      0.94      0.89        18
         Ford       0.89      0.87      0.88       267
          GMC       0.51      0.40      0.45       173
        Honda       0.92      0.88      0.90       147
      Hyundai       0.72      0.83      0.77        88
     Infiniti       0.88      0.87      0.87        89
          Kia       0.76      0.67      0.71        67
  Lamborghini       0.88      1.00      0.94        15
   Land Rover       0.85      0.92      0.88        36
        Lexus       0.78      0.78      0.78        67
      Lincoln       0.74      0.71      0.72        41
        Lotus       0.62      0.83      0.71         6
     Maserati       0.94      0.94      0.94        16
        Mazda       0.86      0.90      0.88       116
Mercedes-Benz       0.83      0.82      0.83        95
   Mitsubishi       0.77      0.77      0.77        60
       Nissan       0.93      0.91      0.92       179
   Oldsmobile       0.74      0.76      0.75        45
     Plymouth       0.67      0.40      0.50        25
      Pontiac       0.57      0.62      0.59        47
      Porsche       0.80      0.75      0.77        32
  Rolls-Royce       1.00      1.00      1.00        10
         Saab       0.91      0.69      0.78        45
        Scion       0.75      0.56      0.64        16
       Subaru       0.87      0.86      0.86        84
       Suzuki       0.87      0.92      0.90        98
       Toyota       0.95      0.90      0.92       220
   Volkswagen       0.92      0.93      0.93       238
        Volvo       0.90      0.90      0.90        84

     accuracy                           0.82      3529
    macro avg       0.82      0.82      0.81      3529
 weighted avg       0.82      0.82      0.82      3529

Plot the confusion matrix to visualize the performance of the model. You can use confusion_matrix from sklearn.metrics and heatmap from seaborn to visualize it.

In [ ]:

conf_matrix_dt = __ # COMPLETE

plt.figure(figsize=(10, 10))
sns.heatmap(conf_matrix_dt, annot=True, fmt='d', cmap='Blues', xticklabels=dt_classifier.classes_, yticklabels=dt_classifier.classes_)
plt.title('Confusion Matrix - Decision Tree Classifier')
plt.xlabel('Predicted')
plt.ylabel('Actual')
plt.show()

conf_matrix_dt = __ # COMPLETE plt.figure(figsize=(10, 10)) sns.heatmap(conf_matrix_dt, annot=True, fmt='d', cmap='Blues', xticklabels=dt_classifier.classes_, yticklabels=dt_classifier.classes_) plt.title('Confusion Matrix - Decision Tree Classifier') plt.xlabel('Predicted') plt.ylabel('Actual') plt.show()

Do you see any interesting patterns in the confusion matrix? Discuss briefly any observations you have.

YOUR DISCUSSION HERE

Now let's explore the space of parameters for the Decision Tree Classifier. We will use GridSearchCV to perform a parameter search on the Decision Tree Classifier. We will search over criterion, max_depth, and splitter.

In [ ]:

# Parameter space for Decision Tree Classifier
# Grid search will evaluate all combinations of these parameters
desicion_tree_params_grid = {
    'criterion': ['gini', 'entropy'],
    'max_depth': [4, 5, 6, 7, 8, 9, 10, 11, 12, 15, 20, 30, 40, 50],
    'splitter': ["best", "random"]
}

# Perform grid search with cross-validation for Decision Tree Classifier
grid_search_decision_tree_classifier = GridSearchCV(
    DecisionTreeClassifier(),
    desicion_tree_params_grid,
    cv=10, # 10-fold cross-validation
    scoring='f1_macro', # Use F1 score for multiclass classification
    verbose=1
    )

grid_search_decision_tree_classifier.fit(X_train, y_train)

print("Decision Tree best grid score in cv: " + str(grid_search_decision_tree_classifier.best_score_))
print("Decision Tree grid test score: " + str(grid_search_decision_tree_classifier.score(X_test, y_test)))

# Parameter space for Decision Tree Classifier # Grid search will evaluate all combinations of these parameters desicion_tree_params_grid = { 'criterion': ['gini', 'entropy'], 'max_depth': [4, 5, 6, 7, 8, 9, 10, 11, 12, 15, 20, 30, 40, 50], 'splitter': ["best", "random"] } # Perform grid search with cross-validation for Decision Tree Classifier grid_search_decision_tree_classifier = GridSearchCV( DecisionTreeClassifier(), desicion_tree_params_grid, cv=10, # 10-fold cross-validation scoring='f1_macro', # Use F1 score for multiclass classification verbose=1 ) grid_search_decision_tree_classifier.fit(X_train, y_train) print("Decision Tree best grid score in cv: " + str(grid_search_decision_tree_classifier.best_score_)) print("Decision Tree grid test score: " + str(grid_search_decision_tree_classifier.score(X_test, y_test)))

Fitting 10 folds for each of 56 candidates, totalling 560 fits
Decision Tree best grid score in cv: 0.808064670198043
Decision Tree grid test score: 0.807472846799344

We can check the best parameters found by the grid search and the best score achieved.

In [13]:

decision_tree_best_params = grid_search_decision_tree_classifier.best_params_
print("Decision Tree best params: " + str(decision_tree_best_params))

decision_tree_best_params = grid_search_decision_tree_classifier.best_params_ print("Decision Tree best params: " + str(decision_tree_best_params))

Decision Tree best params: {'criterion': 'entropy', 'max_depth': 20, 'splitter': 'random'}

The grid_search_decision_tree_classifier object works as a model. You can fit and transform data using it. Let's fit the model to the training data and then evaluate it on the test set. So let's see how the model performs after the parameter search. We will use the report function to evaluate the model on the test set.

In [14]:

decision_tree_best_params = grid_search_decision_tree_classifier.best_params_
print("Decision Tree best params: " + str(decision_tree_best_params))

decision_tree_best_params = grid_search_decision_tree_classifier.best_params_ print("Decision Tree best params: " + str(decision_tree_best_params))

Decision Tree best params: {'criterion': 'entropy', 'max_depth': 20, 'splitter': 'random'}

Now we can run predict() on our grid_search_decision_tree_classifier.

In [15]:

y_pred = grid_search_decision_tree_classifier.predict(X_test)

y_pred = grid_search_decision_tree_classifier.predict(X_test)

Let's look at the resulting report. Call classification_report() below.

In [16]:

grid_search_decision_tree_classification_report = classification_report(y_test, y_pred)
print("Decision Tree Classification report with whole data")
print(grid_search_decision_tree_classification_report)

grid_search_decision_tree_classification_report = classification_report(y_test, y_pred) print("Decision Tree Classification report with whole data") print(grid_search_decision_tree_classification_report)

Decision Tree Classification report with whole data
               precision    recall  f1-score   support

        Acura       0.84      0.84      0.84        67
 Aston Martin       0.94      0.94      0.94        36
         Audi       0.80      0.84      0.82        91
          BMW       0.82      0.87      0.84       102
      Bentley       0.95      0.79      0.86        24
        Buick       0.58      0.63      0.61        52
     Cadillac       0.87      0.85      0.86       123
    Chevrolet       0.65      0.72      0.68       350
     Chrysler       0.65      0.69      0.67        58
        Dodge       0.81      0.84      0.82       188
         FIAT       0.87      0.93      0.90        14
      Ferrari       0.95      1.00      0.97        18
         Ford       0.88      0.90      0.89       267
          GMC       0.49      0.43      0.46       173
        Honda       0.96      0.93      0.94       147
      Hyundai       0.78      0.83      0.80        88
     Infiniti       0.73      0.87      0.79        89
          Kia       0.75      0.69      0.72        67
  Lamborghini       0.83      1.00      0.91        15
   Land Rover       0.85      0.97      0.91        36
        Lexus       0.78      0.73      0.75        67
      Lincoln       0.91      0.73      0.81        41
        Lotus       1.00      1.00      1.00         6
     Maserati       1.00      1.00      1.00        16
        Mazda       0.87      0.85      0.86       116
Mercedes-Benz       0.86      0.76      0.80        95
   Mitsubishi       0.58      0.72      0.64        60
       Nissan       0.94      0.86      0.90       179
   Oldsmobile       0.78      0.80      0.79        45
     Plymouth       0.62      0.40      0.49        25
      Pontiac       0.68      0.60      0.64        47
      Porsche       0.87      0.81      0.84        32
  Rolls-Royce       0.88      0.70      0.78        10
         Saab       1.00      0.62      0.77        45
        Scion       0.67      0.50      0.57        16
       Subaru       0.84      0.82      0.83        84
       Suzuki       0.83      0.88      0.86        98
       Toyota       0.94      0.88      0.91       220
   Volkswagen       0.88      0.91      0.89       238
        Volvo       0.94      0.94      0.94        84

     accuracy                           0.81      3529
    macro avg       0.82      0.80      0.81      3529
 weighted avg       0.81      0.81      0.81      3529

Now, let's see the feature importances of the Decision Tree Classifier. You can access the feature_importances_ attribute of the trained model.

In [ ]:

# Feature importance
feature_importances_dt = dt_classifier.__ # COMPLETE
# Plot feature importances (you can use matplotlib or seaborn)

# YOUR CODE HERE

# Feature importance feature_importances_dt = dt_classifier.__ # COMPLETE # Plot feature importances (you can use matplotlib or seaborn) # YOUR CODE HERE

We'll need to grab the features to use now, using the SelectFromModel() function. Then, let's run fit() on select.

In [18]:

select = SelectFromModel(DecisionTreeClassifier(), threshold='median')
# which features were selected?
print("Features before selection:")
print(X_train.columns.tolist())
select.fit(X_train, y_train) # Your Code Here
X_train_selected = select.transform(X_train)
X_test_selected = select.transform(X_test)
# which features were selected?
selected_features = X_train.columns[select.get_support()]
print("Selected features after selection:")
print(selected_features.tolist())

select = SelectFromModel(DecisionTreeClassifier(), threshold='median') # which features were selected? print("Features before selection:") print(X_train.columns.tolist()) select.fit(X_train, y_train) # Your Code Here X_train_selected = select.transform(X_train) X_test_selected = select.transform(X_test) # which features were selected? selected_features = X_train.columns[select.get_support()] print("Selected features after selection:") print(selected_features.tolist())

Features before selection:
['Engine HP', 'Engine Cylinders', 'Number of Doors', 'highway MPG', 'city mpg', 'MSRP']

Selected features after selection:
['Engine HP', 'highway MPG', 'MSRP']

Now let's apply those best params froem the grid search. Assign the respective fields from decision_tree_best_params for your classifier.

In [19]:

# Applying DecisionTreeClassifier using the best params from the grid search and with selected data
decision_tree_classifier = DecisionTreeClassifier(**decision_tree_best_params)

# Applying DecisionTreeClassifier using the best params from the grid search and with selected data decision_tree_classifier = DecisionTreeClassifier(**decision_tree_best_params)

We need to run the fit() function using X_train_selected and y_train as parameters. Then, run predict() using X_test_selected.

In [ ]:

decision_tree_classifier.fit(__, __) # COMPLETE
y_pred_selected = decision_tree_classifier.predict(__) # COMPLETE

decision_tree_classifier.fit(__, __) # COMPLETE y_pred_selected = decision_tree_classifier.predict(__) # COMPLETE

Lastly, rerun the classification_report() and print out what your results are.

In [ ]:

# Lastly, rerun the `classification_report()` and print out what your results are.
decision_tree_classification_report_selected = classification_report(__, __) # COMPLETE
print("Decision Tree Classification report with selected features")
print(decision_tree_classification_report_selected)

# Lastly, rerun the `classification_report()` and print out what your results are. decision_tree_classification_report_selected = classification_report(__, __) # COMPLETE print("Decision Tree Classification report with selected features") print(decision_tree_classification_report_selected)

Decision Tree Classification report with selected features
               precision    recall  f1-score   support

        Acura       0.73      0.78      0.75        67
 Aston Martin       0.86      0.89      0.88        36
         Audi       0.77      0.80      0.78        91
          BMW       0.85      0.89      0.87       102
      Bentley       0.76      0.92      0.83        24
        Buick       0.60      0.50      0.55        52
     Cadillac       0.84      0.85      0.85       123
    Chevrolet       0.63      0.70      0.66       350
     Chrysler       0.62      0.66      0.64        58
        Dodge       0.80      0.76      0.78       188
         FIAT       0.62      0.71      0.67        14
      Ferrari       0.65      0.72      0.68        18
         Ford       0.85      0.84      0.84       267
          GMC       0.50      0.40      0.44       173
        Honda       0.91      0.82      0.86       147
      Hyundai       0.71      0.80      0.75        88
     Infiniti       0.76      0.85      0.80        89
          Kia       0.61      0.63      0.62        67
  Lamborghini       0.77      0.67      0.71        15
   Land Rover       0.68      0.72      0.70        36
        Lexus       0.80      0.70      0.75        67
      Lincoln       0.73      0.66      0.69        41
        Lotus       1.00      0.83      0.91         6
     Maserati       0.93      0.88      0.90        16
        Mazda       0.84      0.84      0.84       116
Mercedes-Benz       0.86      0.76      0.80        95
   Mitsubishi       0.78      0.75      0.76        60
       Nissan       0.85      0.82      0.83       179
   Oldsmobile       0.72      0.84      0.78        45
     Plymouth       0.45      0.36      0.40        25
      Pontiac       0.47      0.53      0.50        47
      Porsche       0.81      0.78      0.79        32
  Rolls-Royce       1.00      0.80      0.89        10
         Saab       0.80      0.36      0.49        45
        Scion       0.36      0.31      0.33        16
       Subaru       0.79      0.74      0.77        84
       Suzuki       0.71      0.86      0.77        98
       Toyota       0.90      0.86      0.88       220
   Volkswagen       0.86      0.92      0.89       238
        Volvo       0.77      0.89      0.83        84

     accuracy                           0.76      3529
    macro avg       0.75      0.73      0.74      3529
 weighted avg       0.76      0.76      0.76      3529

Do you see any improvements?

Finally, let's plot the ROC (Receiver Operating Characteristic) curve to evaluate the performance of our model. Note that we will need to use the predict_proba() method to get the probabilities for each class. Feel free to choose one of the classes, for example, Toyota. Note that Decision Tree Classifier is not the best for ROC curves as they are not probabilistic models, but we can still plot it for the sake of this exercise. The ROC curves for the other classifiers will be more meaningful.

Plot the ROC with a line for the random classifier (diagonal line) and the ROC curve for the Decision Tree Classifier. Calculate the area under the curve (AUC). You can use roc_curve and auc from sklearn.metrics.

In [ ]:

# ROC
from sklearn.metrics import roc_curve, auc
y_score = grid_search_decision_tree_classifier.predict_proba(X_test)
chosenClass = "Toyota"
chosenClassIndex = grid_search_decision_tree_classifier.classes_.tolist().index(chosenClass)
fpr, tpr, _ = roc_curve(y_test == chosenClass, y_score[:, chosenClassIndex])
roc_auc = auc(fpr, tpr)


# Plot ROC curves: tpr vs fpr (x-axis is fpr, y-axis is tpr)
# Don't forget to draw the random guess line (y=x) use dash line 

# YOUR CODE HERE

# ROC from sklearn.metrics import roc_curve, auc y_score = grid_search_decision_tree_classifier.predict_proba(X_test) chosenClass = "Toyota" chosenClassIndex = grid_search_decision_tree_classifier.classes_.tolist().index(chosenClass) fpr, tpr, _ = roc_curve(y_test == chosenClass, y_score[:, chosenClassIndex]) roc_auc = auc(fpr, tpr) # Plot ROC curves: tpr vs fpr (x-axis is fpr, y-axis is tpr) # Don't forget to draw the random guess line (y=x) use dash line # YOUR CODE HERE

Now let's repeat the steps 3-6 for Logistic Regression, KNeighborsClassifier, RandomForestClassifier, and SVM (SVC).¶

For Logistic Regression and SVM we recommend using standardized X features. You can use StandardScaler from sklearn.preprocessing to standardize the features.

For instance, let's prepare the data for Logistic Regression and SVC:

In [23]:

# Logistic regression works best if the data is scaled, so we will scale the data using StandardScaler.
from sklearn import preprocessing

scaler = preprocessing.StandardScaler().fit(X_train)
X_train_scaled = scaler.transform(X_train)
X_test_scaled = scaler.transform(X_test)

# Logistic regression works best if the data is scaled, so we will scale the data using StandardScaler. from sklearn import preprocessing scaler = preprocessing.StandardScaler().fit(X_train) X_train_scaled = scaler.transform(X_train) X_test_scaled = scaler.transform(X_test)

Now similarly perform the above operations for Logistic Regression

In [24]:

# Your Code Here

# Your Code Here

Now similarly perform the above operations for KNeighborsClassifier.

In [25]:

# Your Code Here

# Your Code Here

Now similarly perform the above operations for Random Forest.

In [26]:

# Your Code Here

# Your Code Here

Now similarly perform the above operations for SVM (SVC).

In [27]:

# Your Code Here

# Your Code Here

Based on all the classification reports, which classifier performed best? Discuss the performance of each classifier and the impact of feature selection and parameter tuning on the results.

YOUR DISCUSSION HERE