[{"content":"","date":"18 January 2023","externalUrl":null,"permalink":"/tags/machine-learning/","section":"Tags","summary":"","title":"Machine Learning","type":"tags"},{"content":" About Me # When I first enrolled at UChicago, my goal was to study physics, attend graduate school, and eventually become a professor. Finding patterns in data and understanding a larger picture through math was exciting, and I was excited to pursue it in such a rigorous environment. I graduated in July 2021 with a BA in Physics with honors and a BS in Mathematics. My favorite classes included Graduate Quantum Mechanics and Graduate Relativity. I was President of the Society of Physics Students, where I ran several fundraisers to promote STEM education, and a Teacher\u0026rsquo;s Assistant for Calculus I–IV.\nI interned at Fermilab, a national physics laboratory, throughout college. As a third-year I worked in the theoretical physics department alongside theoretical, nuclear, and neutrino physicists to develop software that simulates particle interactions — specifically, interactions between known and theoretical neutrinos using Monte Carlo simulations.\nAt Fermilab I found that my real passion was coding and running simulations. During that time I taught myself to implement neural networks for forecasting, NLP, and image recognition, and took classes such as Introduction to Random Matrices to pursue the interest. After several side projects, I decided to pursue data science and engineering full time after college.\nExperience # Paloma Partners Apr 2024 – Present Data Scientist Data scientist at [Paloma Partners](https://www.paloma.com/home). Weiss Multi-Strategy Advisers Sep 2023 – Mar 2024 Senior Data Engineer (also in a Data Scientist capacity) Senior data engineer on the risk team at Weiss, reporting directly to the Chief Risk Officer and building out the data stack for advanced risk analyses.\nRisk dashboard with advanced statistical modelling\nDeveloped and deployed Plotly dashboards providing live insight into intraday investment risk Factor analysis: helped portfolio managers understand how an n-sigma factor move affects PnL via an interactive table built on minimum-torsion analyses of internal and external factor models Single-name analysis: calculated intraday beta-adjusted excess returns and detected outlier movers across strategies with a Mahalanobis distance analysis Internal API migration\nLed the migration of an internal API from R to Python that pulls data from an internal SQL server for use across the firm Deployed the API on a Linux server and helped analysts add their own functionality Wrote unit tests to keep the API\u0026rsquo;s outputs consistent and correct Bloomberg data migration\nDesigned a new SQL schema for Bloomberg Data License data that improved data quality and consistency and made bugs easier to trace Built a program to update the database and run data-validation checks daily Mentorship\nTaught a recent graduate Python techniques, coding best practices, and full-stack development Helped deploy an internal dashboard they built to track the firm\u0026rsquo;s spending IBM Jul 2021 – Aug 2023 AI and Analytics Consultant Full-stack AI and analytics consultant at IBM. I also delivered knowledge-transfer sessions on data science topics to my team.\nDeveloper — increased sign-off delivery speed for new features at a Fortune 500 company by developing custom software.\nCloud architecture designer\nDesigned a data management and analytics architecture for a Fortune 500 automotive client Analyzed employee sentiment from unstructured workshop data to shape the solution Internal developer\nBuilt a Django web application to forecast and map companies\u0026rsquo; financial data Built its unit-testing framework so other developers could build and test new tools faster RTR testing coordinator\nUsed SAP, ALM, and Scrum to help teams implement SAP S/4HANA for a client Facilitated end-to-end testing of Record to Report (RTR) processes, tracked progress in Excel and ALM, and reported status daily Fermilab Jul 2018 – Nov 2020 Research Assistant At Fermilab I grew my coding and analytics skills: writing simulation classes, applying statistics, and working on a research team.\nDeveloped software to simulate theoretical particle interactions with a custom Python class, Monte Carlo and other integration techniques, and unit tests Compared calculations with teammates to validate our predictions Implemented Monte Carlo simulations of particle interactions and compared them with predictions from Quantum Field Theory UChicago STEM Education Jan 2020 – Jun 2021 Research Assistant Worked with the University of Chicago\u0026rsquo;s STEM Education office, drawing insights from a curriculum implemented in Chicago public schools.\nDesigned, executed, and analyzed surveys with Python, Excel, and SQL to evaluate a district-wide school program Identified and reclaimed $600 in unused funds Wrote reports on teachers\u0026rsquo; experiences in the World Chicago Professional Fellows Program based on participant interviews Tutoring 2013 – Present Self-employed TA for Calculus throughout my four years at UChicago, holding office hours and grading assignments. I've taught trigonometry, precalculus, calculus I–IV, linear algebra, differential equations, and mechanics — assigning homework, writing problems that target each student's weaknesses, and reinforcing the topics they're comfortable with. Education # The University of Chicago 2017 – 2021 BA in Physics with Honors, BS in Mathematics · GPA 3.76 Activities: President of the Society of Physics Students · Assistant Chair of Model United Nations · Brother of Alpha Phi Omega, a national service fraternity\nFavorite classes: Graduate Quantum Mechanics I \u0026amp; II · Graduate General Relativity · Abstract Algebra\nScarsdale High School Achievements # Minimum-torsion analysis — developed and deployed a minimum-torsion analysis model based on Meucci et al., 2015. Spark chamber — built a spark chamber with other UChicago students; it\u0026rsquo;s now on display at the Kersten Physics Teaching Center. Integral Bee — planned and ran a competition where students solved integrals as fast as possible, raising money for girls pursuing STEM. Half-marathon — ran the Chicago half-marathon in 2019. Skills # Python Pandas NumPy SQL Plotly / Dash Flask Matplotlib TensorFlow scikit-learn NLP Forecasting Factor decomposition Swagger UI Cron Contact # My inbox is always open. Whether you have a question or just want to say hi, I\u0026rsquo;ll get back to you: myleskornfeld@gmail.com.\n","date":"18 January 2023","externalUrl":null,"permalink":"/","section":"Myles Kornfeld","summary":"About Me # When I first enrolled at UChicago, my goal was to study physics, attend graduate school, and eventually become a professor. Finding patterns in data and understanding a larger picture through math was exciting, and I was excited to pursue it in such a rigorous environment. I graduated in July 2021 with a BA in Physics with honors and a BS in Mathematics. My favorite classes included Graduate Quantum Mechanics and Graduate Relativity. I was President of the Society of Physics Students, where I ran several fundraisers to promote STEM education, and a Teacher’s Assistant for Calculus I–IV.\n","title":"Myles Kornfeld","type":"page"},{"content":" Introduction # Throughout various industries, Natural Language Processing (NLP) models can be used to solve many problems: organizations can use NLP analysis to understand employee satisfaction; companies can use NLP to understand consumer sentiment for a project; investments firms can use NLP to understand the market\u0026rsquo;s sentiment around a specific stock or company. By performing NLP, organizations can gain quantitative insights from qualitative data.\nOne problem that currently faces the field is analyzing sentiment when negations are present. For example, the sentence \u0026ldquo;Don\u0026rsquo;t do that again!\u0026rdquo; can be interpreted in a positive context (if, for example, a person pranked another person) or in a negative context (if a person caused harm to another person). Denials, imperatives, questions, and rejections are all forms have negations that can decrease accuracy. Ultimately, creating a model that can determine sentiment with negations present is important as organizations can use this model alongside other models to gain more accurate insights.\nWith this in mind, my project aims at developing a model that analyzes sentiment analysis with negations present. By analyzing tweets that specifically contain any or all of the words \u0026ldquo;no\u0026rdquo;, \u0026ldquo;not\u0026rdquo;, \u0026ldquo;never\u0026rdquo;, and \u0026ldquo;didn\u0026rsquo;t\u0026rdquo;, I attempt to build a machine learning model that can accurately classify these sentiments as either positive or negative.\nTools # I will be creating these models using Python. In particular, I will be using the pandas and numpy libraries to clean the data, the sklearn library to prepare the data, and keras to build and train the models. I also use the nltk library to provide stopwords.\nimport tensorflow as tf from tensorflow.keras.preprocessing.sequence import pad_sequences from tensorflow.keras.preprocessing.text import text_to_word_sequence from tensorflow.keras.preprocessing.text import Tokenizer import pandas as pd import numpy as np import matplotlib.pyplot as plt from sklearn.model_selection import train_test_split import nltk from nltk.corpus import stopwords nltk.download(\u0026#39;punkt\u0026#39;) nltk.download(\u0026#39;brown\u0026#39;) nltk.download(\u0026#39;stopwords\u0026#39;) stop = stopwords.words(\u0026#39;english\u0026#39;) stop = set(stopwords.words(\u0026#39;english\u0026#39;)) - {\u0026#39;no\u0026#39;, \u0026#39;not\u0026#39;,\u0026#39;never\u0026#39;,\u0026#34;don\u0026#39;t\u0026#34;,\u0026#39;never\u0026#39;,\u0026#39;nor\u0026#39;} When preparing data for NLP, commonly used words, such as \u0026ldquo;our\u0026rdquo;, \u0026ldquo;such\u0026rdquo;, and \u0026ldquo;out\u0026rdquo;, are typically removed from datasets as these are words that do not provide the true sentiment regarding a statement. For this project, I am making sure to include the negations because I want my model to be able to recognize those words, as shown in redefining the set above.\nData Collection and Preparation # I collected data from Sentiment140, a group run by former Computer Science graduate students at Stanford. They offer machine learning sentiment analysis services on tweets, and offer their training and testing datasets free to use. The datasets contain 6 fields:\nThe sentiment of the tweet, where 0 is negative, 2 is neutral, and 4 is positive The id of the tweet The date of the tweet The query of the tweet The twitter user who created the tweet The text contained in the tweet First I load the data, and then run value counts to see how much of each sentiment each dataset contains:\ndf_train = pd.read_csv(\u0026#39;training.1600000.processed.noemoticon.csv\u0026#39;,encoding=\u0026#39;latin-1\u0026#39;) df_test = pd.read_csv(\u0026#39;testdata.manual.2009.06.14.csv\u0026#39;,encoding=\u0026#39;latin-1\u0026#39;) df_train[\u0026#39;sentiment\u0026#39;].value_counts().reset_index() index sentiment 0 0 800000 1 4 800000 df_test[\u0026#39;sentiment\u0026#39;].value_counts().reset_index() index sentiment 0 4 182 1 0 177 2 2 139 Looking at the datasets, we see that the training datasets have a large number of only negative and positive sentiments and no neutral statements, while the testing datasets have a small number of negative and positive sentiments, and a small number of neutral statements. Because the training dataset does not contain any neutral sentiment tweets nor does the testing dataset contain large numbers of validation data, I decide to drop the neutral tweets from the dataset and combine the two datasets. When building my models, I run a 80/20 split on the training/testing data so that the size of the validation dataset increases.\ndf_test = df_test[df_test[\u0026#39;sentiment\u0026#39;]!=2] df_concat = pd.concat([df_train,df_test],axis=0) df_concat.head() sentiment user_id date query user text 0 0 1467810369 Mon Apr 06 22:19:45 PDT 2009 NO_QUERY TheSpecialOne @switchfoot http://twitpic.com/2y1zl - Awww, that\u0026rsquo;s a bummer. You shoulda got David Carr of Third Day to do it. ;D 1 0 1467810672 Mon Apr 06 22:19:49 PDT 2009 NO_QUERY scotthamilton is upset that he can\u0026rsquo;t update his Facebook by texting it\u0026hellip; and might cry as a result School today also. Blah! 2 0 1467810917 Mon Apr 06 22:19:53 PDT 2009 NO_QUERY mattycus @Kenichan I dived many times for the ball. Managed to save 50% The rest go out of bounds 3 0 1467811184 Mon Apr 06 22:19:57 PDT 2009 NO_QUERY ElleCTF my whole body feels itchy and like its on fire 4 0 1467811193 Mon Apr 06 22:19:57 PDT 2009 NO_QUERY Karoli @nationwideclass no, it\u0026rsquo;s not behaving at all. i\u0026rsquo;m mad. why am i here? because I can\u0026rsquo;t see you all over there. Once I combine the datasets, I want to clean them up. First, I make all the letters lowercase so all the words are homogenous. Next, I get rid of twitter usernames and links by filtering out words that contain an \u0026ldquo;@\u0026rdquo; and \u0026ldquo;http\u0026rdquo;, respectively, as those words do not contain sentiment. I also replace 4 to 1 in the sentiment column as it\u0026rsquo;s generally good practice.\ndf_concat[\u0026#39;text\u0026#39;] = df_concat[\u0026#39;text\u0026#39;].str.lower() df_concat[\u0026#39;text\u0026#39;] = df_concat[\u0026#39;text\u0026#39;].apply(lambda x: \u0026#39; \u0026#39;.join([word for word in x.split() if \u0026#39;@\u0026#39; not in word])) df_concat[\u0026#39;text\u0026#39;] = df_concat[\u0026#39;text\u0026#39;].apply(lambda x: \u0026#39; \u0026#39;.join([word for word in x.split() if word not in (stop)])) df_concat[\u0026#39;text\u0026#39;] = df_concat[\u0026#39;text\u0026#39;].apply(lambda x: \u0026#39; \u0026#39;.join([word for word in x.split() if \u0026#39;http\u0026#39; not in word])) df_concat[\u0026#39;text\u0026#39;] = df_concat[\u0026#39;text\u0026#39;].str.replace(r\u0026#39;[^\\w\\s]\u0026#39;,\u0026#39;\u0026#39;,regex=True) df_concat[\u0026#39;sentiment\u0026#39;] = df_concat[\u0026#39;sentiment\u0026#39;].replace(4,1) df_concat.head() sentiment user_id date query user text 0 0 1467810369 Mon Apr 06 22:19:45 PDT 2009 NO_QUERY TheSpecialOne awww thats bummer shoulda got david carr third day 1 0 1467810672 Mon Apr 06 22:19:49 PDT 2009 NO_QUERY scotthamilton upset cant update facebook texting might cry result school today also blah 2 0 1467810917 Mon Apr 06 22:19:53 PDT 2009 NO_QUERY mattycus dived many times ball managed save 50 rest go bounds 3 0 1467811184 Mon Apr 06 22:19:57 PDT 2009 NO_QUERY ElleCTF whole body feels itchy like fire 4 0 1467811193 Mon Apr 06 22:19:57 PDT 2009 NO_QUERY Karoli no not behaving im mad cant see Now, because I removed the apostrophes in each word, I filter to only contain tweets that contain the words \u0026rsquo;no\u0026rsquo;, \u0026rsquo;not\u0026rsquo;, \u0026rsquo;never\u0026rsquo;, \u0026rsquo;never\u0026rsquo;, or \u0026rsquo;nor'.\ndf_neg = df_concat[df_concat[\u0026#39;text\u0026#39;].str.contains(r\u0026#34;no|not|never|didnt|dont|nor\u0026#34;,regex=True)] df_neg = df_neg.reset_index() df_neg.head() index sentiment user_id date query user text 0 4 0 1467811193 Mon Apr 06 22:19:57 PDT 2009 NO_QUERY Karoli no not behaving im mad cant see 1 5 0 1467811372 Mon Apr 06 22:20:00 PDT 2009 NO_QUERY joy_wolf not whole crew 2 7 0 1467811594 Mon Apr 06 22:20:03 PDT 2009 NO_QUERY coZZ hey long time no see yes rains bit bit lol im fine thanks hows 3 8 0 1467811795 Mon Apr 06 22:20:05 PDT 2009 NO_QUERY 2Hood4Hollywood nope 4 10 0 1467812416 Mon Apr 06 22:20:16 PDT 2009 NO_QUERY erinx3leannexo spring break plain city snowing Class Imbalance # When performing a classification analysis, it is important to make sure that the classes you are differentiating are of similar magnitude, i.e. balanced. I will visually check this by plotting the number of negative and positive tweets below.\nx_count = list(df_neg[\u0026#39;sentiment\u0026#39;].unique()) y_count = list(df_neg[\u0026#39;sentiment\u0026#39;].value_counts()) fig = plt.figure(figsize = (12, 8)) # creating the bar plot plt.bar(x_count, y_count, color =\u0026#39;maroon\u0026#39;, width = .8, tick_label=[0,1]) plt.xlabel(\u0026#34;Sentiments\u0026#34;) plt.ylabel(\u0026#34;Value Counts\u0026#34;) plt.title(\u0026#34;Number of Each Sentiment\u0026#34;) plt.show() Clearly, the number of negative and positive tweets with negation are on the same order of magnitude, and so the classes are balanced.\nIdentifying Outliers # Another step in cleaning the data is to eliminate outliers. If a tweet is too small, then it will not contain enough context to tell us how the negations affect the sentiment. On the other hand, if a tweet is too long, the model can pick up trends that fall outside of how the negations impact the sentiment. Below, I plot the length of each tweet against how frequently that length tweet appears.\ndf_neg[\u0026#39;text\u0026#39;].apply(lambda n: len(n.split())).describe().reset_index() index text 0 count 391092 1 mean 9.50027 2 std 4.04891 3 min 1 4 25% 6 5 50% 9 6 75% 13 7 max 29 We see that the max length is 29 words, the minimum is 1 word, the average is 9.5 words, the standard deviation is 4.05 words, and the data has a skew of 0.318. To account for the skew and remove outliers, we will make the minimum number of words required in a tweet to be 3 and the maximum number of words required to be 26, which results in a skew of right skewed with a skew of -0.0312.\ndf_gauss = df_neg[(df_neg[\u0026#39;text\u0026#39;].apply(lambda n: len(n.split()))\u0026gt;=3) \u0026amp; (df_neg[\u0026#39;text\u0026#39;].apply(lambda n: len(n.split()))\u0026lt;=26)] df_gauss[\u0026#39;text\u0026#39;].apply(lambda n: len(n.split())).value_counts().reset_index().describe() index text count 24 24 mean 14.5 15929.7 std 7.07107 13442 min 3 12 25% 8.75 993 50% 14.5 16573.5 75% 20.25 29967 max 26 32067 x_len_new = list(df_gauss[\u0026#39;text\u0026#39;].apply(lambda n: len(n.split())).value_counts().reset_index()[\u0026#39;index\u0026#39;]) y_len_new = list(df_gauss[\u0026#39;text\u0026#39;].apply(lambda n: len(n.split())).value_counts().reset_index()[\u0026#39;text\u0026#39;]) fig = plt.figure(figsize = (12, 8)) plt.bar(x_len_new, y_len_new, color =\u0026#39;maroon\u0026#39;, width = 1) plt.xlabel(\u0026#34;Number of Words\u0026#34;) plt.ylabel(\u0026#34;Value Counts\u0026#34;) plt.title(\u0026#34;Length of Each Tweet\u0026#34;) plt.show() Tokenizing the Data # Now that the data has been cleaned, we need to tokenize it. I tokenize the data with a maximum vocabulary size of 5000, have the max length be the length of the longest tweet, which I restricted to 26, use post padding, and finally join the padded sequences to df_gauss and drop the rest of the columns.\nvocab_size = 4000 max_length = df_gauss[\u0026#39;text\u0026#39;].apply(lambda n: len(n.split())).max() myTokenizer = Tokenizer(num_words=vocab_size) myTokenizer.fit_on_texts(df_concat[\u0026#39;text\u0026#39;]) sequences = myTokenizer.texts_to_sequences(df_gauss[\u0026#39;text\u0026#39;]) padded = pad_sequences(sequences, maxlen=max_length, padding=\u0026#34;post\u0026#34;) df_data = df_gauss.join(pd.DataFrame(padded)) df_data = df_data.drop([\u0026#39;index\u0026#39;,\u0026#39;user_id\u0026#39;,\u0026#39;date\u0026#39;,\u0026#39;query\u0026#39;,\u0026#39;user\u0026#39;,\u0026#39;text\u0026#39;],axis=1) df_data.head() sentiment 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 0 0 7 2 303 1 505 182 14 23 213 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 2 330 2143 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 2 0 83 93 16 7 23 86 2481 165 2454 165 18 1 434 32 715 0 0 0 0 0 0 0 0 0 0 0 4 0 4625 764 375 88 241 311 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 5 0 2178 94 3919 2 22 67 4230 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Preparing the Models # Splitting the Data # When testing the different models and debugging the methods to call the models, I used a subsample of the full dataset for faster debugging. Hence, I created df_sample below. However, for training, I used the full dataset and set frac=1. I then split the data up with an 80/20 split.\ndf_sample = df_data.sample(frac=0.1,random_state=42) y = df_sample[\u0026#39;sentiment\u0026#39;] X = df_sample.drop(\u0026#39;sentiment\u0026#39;,axis=1) X_train, X_test, y_train, y_test = train_test_split(X,y,test_size=0.2) X_train = np.array(X_train) X_test = np.array(X_test) y_train = np.array(y_train) y_test = np.array(y_test) Standardizing the Training # There are several steps that can be taken to train the models efficiently. First, instead of running the models locally on my laptop, I can run them in Google Colab, which provides GPUs for faster training. Next, by implementing callbacks, I save the version of each model that I train at every epoch so that I can access a model at any particular moment during its training. These models are saved to a particular folder path in my google drive for later access. I also implement early stopping, which automatically ends training if the models see no improvement in the validation accuracy after 20 epochs. Finally, plots of the loss and accuracy for the testing and validation data are produced.\nimport datetime datetime.datetime.today().strftime(\u0026#39;%Y-%m-%d_%H:%M:%S\u0026#39;) def fit_model_and_show_results(model, model_name, num_epochs): date = datetime.datetime.today().strftime(\u0026#39;%Y-%m-%d_%H:%M:%S\u0026#39;) checkpoint_filepath = \u0026#39;/content/gdrive/MyDrive/Colab_Notebooks/{folder_path}/\u0026#39;+date+\u0026#39;_weights-{epoch:02d}-{val_accuracy:.3f}.hdf5\u0026#39; checkpoint_dir = os.path.dirname(checkpoint_filepath) es = EarlyStopping(monitor=\u0026#39;val_accuracy\u0026#39;, mode=\u0026#39;max\u0026#39;, verbose=1, patience=20) model_checkpoint_callback = tf.keras.callbacks.ModelCheckpoint( filepath=checkpoint_filepath, save_weights_only=False, monitor=\u0026#39;val_accuracy\u0026#39;, save_freq=\u0026#39;epoch\u0026#39;) callbacks_list = [model_checkpoint_callback] model.summary() history = model.fit(X_train, y_train, epochs=num_epochs, validation_data=(X_test, y_test),callbacks=[callbacks_list,es]) plot_graphs(history, model_name, num_epochs) def plot_graphs(history, model_name, num_epochs): fig, (ax1, ax2) = plt.subplots(1, 2) fig.set_figheight(6) fig.set_figwidth(16) fig.suptitle(model_name + \u0026#39; Accuracy and Loss over \u0026#39;+str(num_epochs) + \u0026#39; Epochs\u0026#39;) ax1.plot(history.history[\u0026#39;accuracy\u0026#39;]) ax1.plot(history.history[\u0026#39;val_accuracy\u0026#39;]) ax1.set(xlabel=\u0026#39;Epochs\u0026#39;, ylabel=\u0026#39;Accuracy\u0026#39;) ax1.legend([\u0026#39;accuracy\u0026#39;,\u0026#39;val_accuracy\u0026#39;]) ax2.plot(history.history[\u0026#39;loss\u0026#39;]) ax2.plot(history.history[\u0026#39;val_loss\u0026#39;]) ax2.set(xlabel=\u0026#39;Epochs\u0026#39;, ylabel=\u0026#39;Loss\u0026#39;) ax2.legend([\u0026#39;loss\u0026#39;,\u0026#39;val_loss\u0026#39;]) plt.show() Training the Models # Simple Model # The first model I train is a simple GlobalAveragePooling model. I ran it for 100 epochs, however it early stopped after 39 epochs. Below is the model architecture and the results.\nmodel = tf.keras.Sequential([ tf.keras.layers.Embedding(vocab_size, embedding_dim, input_length=max_length), tf.keras.layers.GlobalAveragePooling1D(), tf.keras.layers.Dense(1, activation=\u0026#39;sigmoid\u0026#39;) ]) learning_rate = 0.0005 model.compile(loss=\u0026#39;binary_crossentropy\u0026#39;, optimizer=tf.keras.optimizers.Adam(learning_rate), metrics=[\u0026#39;accuracy\u0026#39;]) model.summary() fit_model_and_show_results(model,\u0026#39;Simple Model\u0026#39;,100) Model: \u0026#34;sequential_5\u0026#34; _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= embedding_5 (Embedding) (None, 26, 16) 64000 global_average_pooling1d_5 (None, 16) 0 (GlobalAveragePooling1D) dense_5 (Dense) (None, 1) 17 ================================================================= Total params: 64,017 Trainable params: 64,017 Non-trainable params: 0 _________________________________________________________________ Model: \u0026#34;sequential_5\u0026#34; _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= embedding_5 (Embedding) (None, 26, 16) 64000 global_average_pooling1d_5 (None, 16) 0 (GlobalAveragePooling1D) dense_5 (Dense) (None, 1) 17 ================================================================= Total params: 64,017 Trainable params: 64,017 Non-trainable params: 0 _________________________________________________________________ Epoch 1/100 8445/8445 [==============================] - 36s 4ms/step - loss: 0.5915 - accuracy: 0.6916 - val_loss: 0.5443 - val_accuracy: 0.7378 Epoch 2/100 8445/8445 [==============================] - 35s 4ms/step - loss: 0.5337 - accuracy: 0.7468 - val_loss: 0.5342 - val_accuracy: 0.7448 Epoch 3/100 8445/8445 [==============================] - 35s 4ms/step - loss: 0.5266 - accuracy: 0.7505 - val_loss: 0.5325 - val_accuracy: 0.7452 Epoch 4/100 8445/8445 [==============================] - 36s 4ms/step - loss: 0.5242 - accuracy: 0.7525 - val_loss: 0.5323 - val_accuracy: 0.7452 Epoch 5/100 8445/8445 [==============================] - 34s 4ms/step - loss: 0.5232 - accuracy: 0.7532 - val_loss: 0.5333 - val_accuracy: 0.7432 Epoch 6/100 8445/8445 [==============================] - 35s 4ms/step - loss: 0.5227 - accuracy: 0.7534 - val_loss: 0.5328 - val_accuracy: 0.7448 Epoch 7/100 8445/8445 [==============================] - 35s 4ms/step - loss: 0.5223 - accuracy: 0.7540 - val_loss: 0.5335 - val_accuracy: 0.7454 Epoch 8/100 8445/8445 [==============================] - 35s 4ms/step - loss: 0.5222 - accuracy: 0.7537 - val_loss: 0.5333 - val_accuracy: 0.7450 Epoch 9/100 8445/8445 [==============================] - 33s 4ms/step - loss: 0.5221 - accuracy: 0.7538 - val_loss: 0.5338 - val_accuracy: 0.7450 Epoch 10/100 8445/8445 [==============================] - 36s 4ms/step - loss: 0.5219 - accuracy: 0.7540 - val_loss: 0.5338 - val_accuracy: 0.7450 Epoch 11/100 8445/8445 [==============================] - 35s 4ms/step - loss: 0.5218 - accuracy: 0.7546 - val_loss: 0.5339 - val_accuracy: 0.7437 Epoch 12/100 8445/8445 [==============================] - 33s 4ms/step - loss: 0.5218 - accuracy: 0.7543 - val_loss: 0.5337 - val_accuracy: 0.7452 Epoch 13/100 8445/8445 [==============================] - 35s 4ms/step - loss: 0.5217 - accuracy: 0.7544 - val_loss: 0.5350 - val_accuracy: 0.7445 Epoch 14/100 8445/8445 [==============================] - 33s 4ms/step - loss: 0.5217 - accuracy: 0.7542 - val_loss: 0.5340 - val_accuracy: 0.7449 Epoch 15/100 8445/8445 [==============================] - 35s 4ms/step - loss: 0.5216 - accuracy: 0.7541 - val_loss: 0.5346 - val_accuracy: 0.7429 Epoch 16/100 8445/8445 [==============================] - 34s 4ms/step - loss: 0.5216 - accuracy: 0.7545 - val_loss: 0.5343 - val_accuracy: 0.7437 Epoch 17/100 8445/8445 [==============================] - 33s 4ms/step - loss: 0.5214 - accuracy: 0.7539 - val_loss: 0.5349 - val_accuracy: 0.7426 Epoch 18/100 8445/8445 [==============================] - 35s 4ms/step - loss: 0.5214 - accuracy: 0.7542 - val_loss: 0.5342 - val_accuracy: 0.7444 Epoch 19/100 8445/8445 [==============================] - 35s 4ms/step - loss: 0.5214 - accuracy: 0.7545 - val_loss: 0.5341 - val_accuracy: 0.7456 Epoch 20/100 8445/8445 [==============================] - 33s 4ms/step - loss: 0.5213 - accuracy: 0.7543 - val_loss: 0.5346 - val_accuracy: 0.7452 Epoch 21/100 8445/8445 [==============================] - 35s 4ms/step - loss: 0.5213 - accuracy: 0.7543 - val_loss: 0.5346 - val_accuracy: 0.7450 Epoch 22/100 8445/8445 [==============================] - 36s 4ms/step - loss: 0.5212 - accuracy: 0.7543 - val_loss: 0.5343 - val_accuracy: 0.7445 Epoch 23/100 8445/8445 [==============================] - 33s 4ms/step - loss: 0.5211 - accuracy: 0.7548 - val_loss: 0.5346 - val_accuracy: 0.7435 Epoch 24/100 8445/8445 [==============================] - 33s 4ms/step - loss: 0.5211 - accuracy: 0.7546 - val_loss: 0.5342 - val_accuracy: 0.7445 Epoch 25/100 8445/8445 [==============================] - 34s 4ms/step - loss: 0.5211 - accuracy: 0.7545 - val_loss: 0.5342 - val_accuracy: 0.7452 Epoch 26/100 8445/8445 [==============================] - 32s 4ms/step - loss: 0.5210 - accuracy: 0.7547 - val_loss: 0.5350 - val_accuracy: 0.7451 Epoch 27/100 8445/8445 [==============================] - 33s 4ms/step - loss: 0.5210 - accuracy: 0.7550 - val_loss: 0.5345 - val_accuracy: 0.7437 Epoch 28/100 8445/8445 [==============================] - 35s 4ms/step - loss: 0.5210 - accuracy: 0.7548 - val_loss: 0.5343 - val_accuracy: 0.7440 Epoch 29/100 8445/8445 [==============================] - 33s 4ms/step - loss: 0.5210 - accuracy: 0.7551 - val_loss: 0.5356 - val_accuracy: 0.7450 Epoch 30/100 8445/8445 [==============================] - 33s 4ms/step - loss: 0.5209 - accuracy: 0.7545 - val_loss: 0.5344 - val_accuracy: 0.7455 Epoch 31/100 8445/8445 [==============================] - 34s 4ms/step - loss: 0.5208 - accuracy: 0.7546 - val_loss: 0.5345 - val_accuracy: 0.7454 Epoch 32/100 8445/8445 [==============================] - 33s 4ms/step - loss: 0.5208 - accuracy: 0.7549 - val_loss: 0.5345 - val_accuracy: 0.7449 Epoch 33/100 8445/8445 [==============================] - 35s 4ms/step - loss: 0.5209 - accuracy: 0.7552 - val_loss: 0.5342 - val_accuracy: 0.7445 Epoch 34/100 8445/8445 [==============================] - 33s 4ms/step - loss: 0.5208 - accuracy: 0.7551 - val_loss: 0.5344 - val_accuracy: 0.7444 Epoch 35/100 8445/8445 [==============================] - 33s 4ms/step - loss: 0.5208 - accuracy: 0.7550 - val_loss: 0.5348 - val_accuracy: 0.7453 Epoch 36/100 8445/8445 [==============================] - 35s 4ms/step - loss: 0.5207 - accuracy: 0.7552 - val_loss: 0.5345 - val_accuracy: 0.7443 Epoch 37/100 8445/8445 [==============================] - 32s 4ms/step - loss: 0.5207 - accuracy: 0.7553 - val_loss: 0.5344 - val_accuracy: 0.7444 Epoch 38/100 8445/8445 [==============================] - 32s 4ms/step - loss: 0.5207 - accuracy: 0.7551 - val_loss: 0.5346 - val_accuracy: 0.7441 Epoch 39/100 8445/8445 [==============================] - 34s 4ms/step - loss: 0.5206 - accuracy: 0.7553 - val_loss: 0.5342 - val_accuracy: 0.7450 Epoch 39: early stopping CNN Model # The second model is a CNN model with 1 convolution layer. I trained it for 30 epochs, however it stopped early at 28 epochs.\nmodel_cnn = tf.keras.Sequential([ tf.keras.layers.Embedding(vocab_size, embedding_dim, input_length=max_length), tf.keras.layers.Conv1D(16, 5, activation=\u0026#39;relu\u0026#39;), tf.keras.layers.GlobalMaxPooling1D(), tf.keras.layers.Dense(1, activation=\u0026#39;sigmoid\u0026#39;) ]) learning_rate = 0.0001 model_cnn.compile(loss=\u0026#39;binary_crossentropy\u0026#39;, optimizer=tf.keras.optimizers.Adam(learning_rate), metrics=[\u0026#39;accuracy\u0026#39;]) fit_model_and_show_results(model_cnn,\u0026#39;CNN Model\u0026#39;,30) Model: \u0026#34;sequential\u0026#34; _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= embedding (Embedding) (None, 26, 16) 64000 conv1d (Conv1D) (None, 22, 16) 1296 global_max_pooling1d (Globa (None, 16) 0 lMaxPooling1D) dense (Dense) (None, 1) 17 ================================================================= Total params: 65,313 Trainable params: 65,313 Non-trainable params: 0 _________________________________________________________________ Epoch 1/30 8445/8445 [==============================] - 43s 4ms/step - loss: 0.6042 - accuracy: 0.6724 - val_loss: 0.5378 - val_accuracy: 0.7386 Epoch 2/30 8445/8445 [==============================] - 35s 4ms/step - loss: 0.5180 - accuracy: 0.7496 - val_loss: 0.5092 - val_accuracy: 0.7524 Epoch 3/30 8445/8445 [==============================] - 34s 4ms/step - loss: 0.4962 - accuracy: 0.7603 - val_loss: 0.4982 - val_accuracy: 0.7558 Epoch 4/30 8445/8445 [==============================] - 34s 4ms/step - loss: 0.4851 - accuracy: 0.7667 - val_loss: 0.4928 - val_accuracy: 0.7583 Epoch 5/30 8445/8445 [==============================] - 36s 4ms/step - loss: 0.4777 - accuracy: 0.7703 - val_loss: 0.4913 - val_accuracy: 0.7591 Epoch 6/30 8445/8445 [==============================] - 36s 4ms/step - loss: 0.4722 - accuracy: 0.7734 - val_loss: 0.4880 - val_accuracy: 0.7615 Epoch 7/30 8445/8445 [==============================] - 34s 4ms/step - loss: 0.4675 - accuracy: 0.7758 - val_loss: 0.4868 - val_accuracy: 0.7625 Epoch 8/30 8445/8445 [==============================] - 35s 4ms/step - loss: 0.4633 - accuracy: 0.7790 - val_loss: 0.4860 - val_accuracy: 0.7633 Epoch 9/30 8445/8445 [==============================] - 36s 4ms/step - loss: 0.4594 - accuracy: 0.7816 - val_loss: 0.4863 - val_accuracy: 0.7627 Epoch 10/30 8445/8445 [==============================] - 36s 4ms/step - loss: 0.4558 - accuracy: 0.7841 - val_loss: 0.4856 - val_accuracy: 0.7618 Epoch 11/30 8445/8445 [==============================] - 36s 4ms/step - loss: 0.4522 - accuracy: 0.7863 - val_loss: 0.4857 - val_accuracy: 0.7618 Epoch 12/30 8445/8445 [==============================] - 36s 4ms/step - loss: 0.4487 - accuracy: 0.7887 - val_loss: 0.4863 - val_accuracy: 0.7616 Epoch 13/30 8445/8445 [==============================] - 34s 4ms/step - loss: 0.4453 - accuracy: 0.7903 - val_loss: 0.4867 - val_accuracy: 0.7622 Epoch 14/30 8445/8445 [==============================] - 37s 4ms/step - loss: 0.4420 - accuracy: 0.7925 - val_loss: 0.4876 - val_accuracy: 0.7616 Epoch 15/30 8445/8445 [==============================] - 36s 4ms/step - loss: 0.4387 - accuracy: 0.7948 - val_loss: 0.4882 - val_accuracy: 0.7611 Epoch 16/30 8445/8445 [==============================] - 36s 4ms/step - loss: 0.4356 - accuracy: 0.7967 - val_loss: 0.4890 - val_accuracy: 0.7600 Epoch 17/30 8445/8445 [==============================] - 36s 4ms/step - loss: 0.4325 - accuracy: 0.7988 - val_loss: 0.4902 - val_accuracy: 0.7604 Epoch 18/30 8445/8445 [==============================] - 38s 4ms/step - loss: 0.4293 - accuracy: 0.8008 - val_loss: 0.4918 - val_accuracy: 0.7598 Epoch 19/30 8445/8445 [==============================] - 34s 4ms/step - loss: 0.4263 - accuracy: 0.8025 - val_loss: 0.4935 - val_accuracy: 0.7604 Epoch 20/30 8445/8445 [==============================] - 34s 4ms/step - loss: 0.4234 - accuracy: 0.8046 - val_loss: 0.4950 - val_accuracy: 0.7593 Epoch 21/30 8445/8445 [==============================] - 36s 4ms/step - loss: 0.4204 - accuracy: 0.8066 - val_loss: 0.4963 - val_accuracy: 0.7588 Epoch 22/30 8445/8445 [==============================] - 34s 4ms/step - loss: 0.4175 - accuracy: 0.8083 - val_loss: 0.4984 - val_accuracy: 0.7577 Epoch 23/30 8445/8445 [==============================] - 37s 4ms/step - loss: 0.4147 - accuracy: 0.8100 - val_loss: 0.5000 - val_accuracy: 0.7575 Epoch 24/30 8445/8445 [==============================] - 34s 4ms/step - loss: 0.4119 - accuracy: 0.8122 - val_loss: 0.5009 - val_accuracy: 0.7571 Epoch 25/30 8445/8445 [==============================] - 34s 4ms/step - loss: 0.4092 - accuracy: 0.8136 - val_loss: 0.5033 - val_accuracy: 0.7563 Epoch 26/30 8445/8445 [==============================] - 34s 4ms/step - loss: 0.4066 - accuracy: 0.8154 - val_loss: 0.5054 - val_accuracy: 0.7554 Epoch 27/30 8445/8445 [==============================] - 36s 4ms/step - loss: 0.4041 - accuracy: 0.8170 - val_loss: 0.5079 - val_accuracy: 0.7541 Epoch 28/30 8445/8445 [==============================] - 35s 4ms/step - loss: 0.4016 - accuracy: 0.8184 - val_loss: 0.5094 - val_accuracy: 0.7538 Epoch 28: early stopping GRU Model # Next I trained the data on a GRU model for 100 epochs. The model had an early stop at 42 epochs.\nmodel_gru = tf.keras.Sequential([ tf.keras.layers.Embedding(vocab_size, embedding_dim, input_length=max_length), tf.keras.layers.Bidirectional(tf.keras.layers.GRU(32)), tf.keras.layers.Dense(1, activation=\u0026#39;sigmoid\u0026#39;) ]) learning_rate = 0.0001 model_gru.compile(loss=\u0026#39;binary_crossentropy\u0026#39;, optimizer=tf.keras.optimizers.Adam(learning_rate), metrics=[\u0026#39;accuracy\u0026#39;]) fit_model_and_show_results(model_gru, \u0026#39;GRU Model\u0026#39;, 100) Model: \u0026#34;sequential_6\u0026#34; _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= embedding_6 (Embedding) (None, 26, 16) 64000 bidirectional (Bidirectiona (None, 64) 9600 l) dense_6 (Dense) (None, 1) 65 ================================================================= Total params: 73,665 Trainable params: 73,665 Non-trainable params: 0 _________________________________________________________________ Epoch 1/100 8445/8445 [==============================] - 73s 8ms/step - loss: 0.5641 - accuracy: 0.7149 - val_loss: 0.5305 - val_accuracy: 0.7424 Epoch 2/100 8445/8445 [==============================] - 65s 8ms/step - loss: 0.5173 - accuracy: 0.7495 - val_loss: 0.5155 - val_accuracy: 0.7456 Epoch 3/100 8445/8445 [==============================] - 65s 8ms/step - loss: 0.5008 - accuracy: 0.7552 - val_loss: 0.5098 - val_accuracy: 0.7488 Epoch 4/100 8445/8445 [==============================] - 66s 8ms/step - loss: 0.4944 - accuracy: 0.7586 - val_loss: 0.5032 - val_accuracy: 0.7515 Epoch 5/100 8445/8445 [==============================] - 65s 8ms/step - loss: 0.4906 - accuracy: 0.7611 - val_loss: 0.5019 - val_accuracy: 0.7517 Epoch 6/100 8445/8445 [==============================] - 65s 8ms/step - loss: 0.4877 - accuracy: 0.7628 - val_loss: 0.5012 - val_accuracy: 0.7518 Epoch 7/100 8445/8445 [==============================] - 66s 8ms/step - loss: 0.4853 - accuracy: 0.7640 - val_loss: 0.5021 - val_accuracy: 0.7528 Epoch 8/100 8445/8445 [==============================] - 64s 8ms/step - loss: 0.4831 - accuracy: 0.7659 - val_loss: 0.4997 - val_accuracy: 0.7537 Epoch 9/100 8445/8445 [==============================] - 64s 8ms/step - loss: 0.4809 - accuracy: 0.7673 - val_loss: 0.4995 - val_accuracy: 0.7534 Epoch 10/100 8445/8445 [==============================] - 65s 8ms/step - loss: 0.4786 - accuracy: 0.7694 - val_loss: 0.4978 - val_accuracy: 0.7548 Epoch 11/100 8445/8445 [==============================] - 64s 8ms/step - loss: 0.4761 - accuracy: 0.7707 - val_loss: 0.4977 - val_accuracy: 0.7551 Epoch 12/100 8445/8445 [==============================] - 64s 8ms/step - loss: 0.4731 - accuracy: 0.7728 - val_loss: 0.4943 - val_accuracy: 0.7576 Epoch 13/100 8445/8445 [==============================] - 65s 8ms/step - loss: 0.4696 - accuracy: 0.7748 - val_loss: 0.4931 - val_accuracy: 0.7575 Epoch 14/100 8445/8445 [==============================] - 64s 8ms/step - loss: 0.4656 - accuracy: 0.7771 - val_loss: 0.4913 - val_accuracy: 0.7589 Epoch 15/100 8445/8445 [==============================] - 64s 8ms/step - loss: 0.4615 - accuracy: 0.7795 - val_loss: 0.4897 - val_accuracy: 0.7608 Epoch 16/100 8445/8445 [==============================] - 65s 8ms/step - loss: 0.4576 - accuracy: 0.7816 - val_loss: 0.4895 - val_accuracy: 0.7622 Epoch 17/100 8445/8445 [==============================] - 64s 8ms/step - loss: 0.4542 - accuracy: 0.7837 - val_loss: 0.4875 - val_accuracy: 0.7632 Epoch 18/100 8445/8445 [==============================] - 63s 7ms/step - loss: 0.4512 - accuracy: 0.7853 - val_loss: 0.4890 - val_accuracy: 0.7626 Epoch 19/100 8445/8445 [==============================] - 65s 8ms/step - loss: 0.4485 - accuracy: 0.7873 - val_loss: 0.4884 - val_accuracy: 0.7615 Epoch 20/100 8445/8445 [==============================] - 63s 8ms/step - loss: 0.4461 - accuracy: 0.7888 - val_loss: 0.4865 - val_accuracy: 0.7640 Epoch 21/100 8445/8445 [==============================] - 64s 8ms/step - loss: 0.4440 - accuracy: 0.7900 - val_loss: 0.4859 - val_accuracy: 0.7641 Epoch 22/100 8445/8445 [==============================] - 65s 8ms/step - loss: 0.4418 - accuracy: 0.7913 - val_loss: 0.4861 - val_accuracy: 0.7641 Epoch 23/100 8445/8445 [==============================] - 64s 8ms/step - loss: 0.4398 - accuracy: 0.7926 - val_loss: 0.4888 - val_accuracy: 0.7638 Epoch 24/100 8445/8445 [==============================] - 64s 8ms/step - loss: 0.4380 - accuracy: 0.7936 - val_loss: 0.4882 - val_accuracy: 0.7637 Epoch 25/100 8445/8445 [==============================] - 64s 8ms/step - loss: 0.4363 - accuracy: 0.7944 - val_loss: 0.4892 - val_accuracy: 0.7635 Epoch 26/100 8445/8445 [==============================] - 64s 8ms/step - loss: 0.4346 - accuracy: 0.7953 - val_loss: 0.4892 - val_accuracy: 0.7621 Epoch 27/100 8445/8445 [==============================] - 64s 8ms/step - loss: 0.4331 - accuracy: 0.7963 - val_loss: 0.4905 - val_accuracy: 0.7631 Epoch 28/100 8445/8445 [==============================] - 64s 8ms/step - loss: 0.4315 - accuracy: 0.7971 - val_loss: 0.4909 - val_accuracy: 0.7617 Epoch 29/100 8445/8445 [==============================] - 63s 7ms/step - loss: 0.4300 - accuracy: 0.7985 - val_loss: 0.4931 - val_accuracy: 0.7614 Epoch 30/100 8445/8445 [==============================] - 63s 7ms/step - loss: 0.4284 - accuracy: 0.7988 - val_loss: 0.4930 - val_accuracy: 0.7605 Epoch 31/100 8445/8445 [==============================] - 65s 8ms/step - loss: 0.4269 - accuracy: 0.8002 - val_loss: 0.4939 - val_accuracy: 0.7611 Epoch 32/100 8445/8445 [==============================] - 64s 8ms/step - loss: 0.4253 - accuracy: 0.8009 - val_loss: 0.4959 - val_accuracy: 0.7615 Epoch 33/100 8445/8445 [==============================] - 65s 8ms/step - loss: 0.4236 - accuracy: 0.8017 - val_loss: 0.4965 - val_accuracy: 0.7613 Epoch 34/100 8445/8445 [==============================] - 64s 8ms/step - loss: 0.4221 - accuracy: 0.8028 - val_loss: 0.4967 - val_accuracy: 0.7600 Epoch 35/100 8445/8445 [==============================] - 64s 8ms/step - loss: 0.4203 - accuracy: 0.8039 - val_loss: 0.4996 - val_accuracy: 0.7604 Epoch 36/100 8445/8445 [==============================] - 65s 8ms/step - loss: 0.4185 - accuracy: 0.8051 - val_loss: 0.5014 - val_accuracy: 0.7594 Epoch 37/100 8445/8445 [==============================] - 64s 8ms/step - loss: 0.4167 - accuracy: 0.8062 - val_loss: 0.5033 - val_accuracy: 0.7590 Epoch 38/100 8445/8445 [==============================] - 64s 8ms/step - loss: 0.4149 - accuracy: 0.8071 - val_loss: 0.5061 - val_accuracy: 0.7585 Epoch 39/100 8445/8445 [==============================] - 65s 8ms/step - loss: 0.4129 - accuracy: 0.8088 - val_loss: 0.5057 - val_accuracy: 0.7585 Epoch 40/100 8445/8445 [==============================] - 64s 8ms/step - loss: 0.4112 - accuracy: 0.8096 - val_loss: 0.5074 - val_accuracy: 0.7589 Epoch 41/100 8445/8445 [==============================] - 64s 8ms/step - loss: 0.4093 - accuracy: 0.8105 - val_loss: 0.5111 - val_accuracy: 0.7579 Epoch 42/100 8445/8445 [==============================] - 65s 8ms/step - loss: 0.4073 - accuracy: 0.8119 - val_loss: 0.5112 - val_accuracy: 0.7562 Epoch 42: early stopping Bidirectional LSTM # The next model I trained was a bidirectional LSTM model with one layer. The model was trained for 70 epochs but stopped at 61.\nmodel_bidi_lstm = tf.keras.Sequential([ tf.keras.layers.Embedding(vocab_size, embedding_dim, input_length=max_length), tf.keras.layers.Bidirectional(tf.keras.layers.LSTM(embedding_dim)), tf.keras.layers.Dense(1, activation=\u0026#39;sigmoid\u0026#39;) ]) learning_rate = 0.00003 model_bidi_lstm.compile(loss=\u0026#39;binary_crossentropy\u0026#39;, optimizer=tf.keras.optimizers.Adam(learning_rate), metrics=[\u0026#39;accuracy\u0026#39;]) fit_model_and_show_results(model_bidi_lstm, \u0026#39;Bidirectional LSTM Model\u0026#39;,70) Model: \u0026#34;sequential\u0026#34; _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= embedding (Embedding) (None, 26, 16) 64000 bidirectional (Bidirectiona (None, 32) 4224 l) dense (Dense) (None, 1) 33 ================================================================= Total params: 68,257 Trainable params: 68,257 Non-trainable params: 0 _________________________________________________________________ Epoch 1/70 8445/8445 [==============================] - 76s 8ms/step - loss: 0.6338 - accuracy: 0.6332 - val_loss: 0.5873 - val_accuracy: 0.6849 Epoch 2/70 8445/8445 [==============================] - 69s 8ms/step - loss: 0.5509 - accuracy: 0.7267 - val_loss: 0.5370 - val_accuracy: 0.7370 Epoch 3/70 8445/8445 [==============================] - 67s 8ms/step - loss: 0.5274 - accuracy: 0.7449 - val_loss: 0.5291 - val_accuracy: 0.7397 Epoch 4/70 8445/8445 [==============================] - 65s 8ms/step - loss: 0.5197 - accuracy: 0.7486 - val_loss: 0.5234 - val_accuracy: 0.7459 Epoch 5/70 8445/8445 [==============================] - 66s 8ms/step - loss: 0.5146 - accuracy: 0.7520 - val_loss: 0.5200 - val_accuracy: 0.7468 Epoch 6/70 8445/8445 [==============================] - 64s 8ms/step - loss: 0.5102 - accuracy: 0.7540 - val_loss: 0.5171 - val_accuracy: 0.7482 Epoch 7/70 8445/8445 [==============================] - 64s 8ms/step - loss: 0.5067 - accuracy: 0.7555 - val_loss: 0.5165 - val_accuracy: 0.7482 Epoch 8/70 8445/8445 [==============================] - 69s 8ms/step - loss: 0.5038 - accuracy: 0.7567 - val_loss: 0.5129 - val_accuracy: 0.7489 Epoch 9/70 8445/8445 [==============================] - 65s 8ms/step - loss: 0.5017 - accuracy: 0.7578 - val_loss: 0.5114 - val_accuracy: 0.7495 Epoch 10/70 8445/8445 [==============================] - 64s 8ms/step - loss: 0.4999 - accuracy: 0.7586 - val_loss: 0.5100 - val_accuracy: 0.7502 Epoch 11/70 8445/8445 [==============================] - 66s 8ms/step - loss: 0.4983 - accuracy: 0.7593 - val_loss: 0.5088 - val_accuracy: 0.7501 Epoch 12/70 8445/8445 [==============================] - 64s 8ms/step - loss: 0.4970 - accuracy: 0.7597 - val_loss: 0.5091 - val_accuracy: 0.7497 Epoch 13/70 8445/8445 [==============================] - 64s 8ms/step - loss: 0.4956 - accuracy: 0.7602 - val_loss: 0.5067 - val_accuracy: 0.7515 Epoch 14/70 8445/8445 [==============================] - 65s 8ms/step - loss: 0.4943 - accuracy: 0.7605 - val_loss: 0.5066 - val_accuracy: 0.7508 Epoch 15/70 8445/8445 [==============================] - 64s 8ms/step - loss: 0.4929 - accuracy: 0.7614 - val_loss: 0.5049 - val_accuracy: 0.7519 Epoch 16/70 8445/8445 [==============================] - 64s 8ms/step - loss: 0.4916 - accuracy: 0.7619 - val_loss: 0.5038 - val_accuracy: 0.7524 Epoch 17/70 8445/8445 [==============================] - 65s 8ms/step - loss: 0.4901 - accuracy: 0.7625 - val_loss: 0.5037 - val_accuracy: 0.7515 Epoch 18/70 8445/8445 [==============================] - 64s 8ms/step - loss: 0.4889 - accuracy: 0.7633 - val_loss: 0.5022 - val_accuracy: 0.7532 Epoch 19/70 8445/8445 [==============================] - 64s 8ms/step - loss: 0.4877 - accuracy: 0.7634 - val_loss: 0.5018 - val_accuracy: 0.7532 Epoch 20/70 8445/8445 [==============================] - 65s 8ms/step - loss: 0.4865 - accuracy: 0.7636 - val_loss: 0.5020 - val_accuracy: 0.7529 Epoch 21/70 8445/8445 [==============================] - 64s 8ms/step - loss: 0.4853 - accuracy: 0.7643 - val_loss: 0.5004 - val_accuracy: 0.7534 Epoch 22/70 8445/8445 [==============================] - 64s 8ms/step - loss: 0.4841 - accuracy: 0.7647 - val_loss: 0.5003 - val_accuracy: 0.7533 Epoch 23/70 8445/8445 [==============================] - 65s 8ms/step - loss: 0.4827 - accuracy: 0.7653 - val_loss: 0.5002 - val_accuracy: 0.7539 Epoch 24/70 8445/8445 [==============================] - 65s 8ms/step - loss: 0.4814 - accuracy: 0.7659 - val_loss: 0.4996 - val_accuracy: 0.7540 Epoch 25/70 8445/8445 [==============================] - 64s 8ms/step - loss: 0.4801 - accuracy: 0.7666 - val_loss: 0.4997 - val_accuracy: 0.7544 Epoch 26/70 8445/8445 [==============================] - 65s 8ms/step - loss: 0.4787 - accuracy: 0.7673 - val_loss: 0.5001 - val_accuracy: 0.7547 Epoch 27/70 8445/8445 [==============================] - 63s 8ms/step - loss: 0.4775 - accuracy: 0.7679 - val_loss: 0.4991 - val_accuracy: 0.7551 Epoch 28/70 8445/8445 [==============================] - 64s 8ms/step - loss: 0.4762 - accuracy: 0.7683 - val_loss: 0.4996 - val_accuracy: 0.7550 Epoch 29/70 8445/8445 [==============================] - 68s 8ms/step - loss: 0.4749 - accuracy: 0.7692 - val_loss: 0.4996 - val_accuracy: 0.7558 Epoch 30/70 8445/8445 [==============================] - 64s 8ms/step - loss: 0.4738 - accuracy: 0.7698 - val_loss: 0.5007 - val_accuracy: 0.7558 Epoch 31/70 8445/8445 [==============================] - 64s 8ms/step - loss: 0.4727 - accuracy: 0.7703 - val_loss: 0.5011 - val_accuracy: 0.7548 Epoch 32/70 8445/8445 [==============================] - 67s 8ms/step - loss: 0.4716 - accuracy: 0.7708 - val_loss: 0.5001 - val_accuracy: 0.7556 Epoch 33/70 8445/8445 [==============================] - 65s 8ms/step - loss: 0.4707 - accuracy: 0.7710 - val_loss: 0.4996 - val_accuracy: 0.7560 Epoch 34/70 8445/8445 [==============================] - 64s 8ms/step - loss: 0.4696 - accuracy: 0.7720 - val_loss: 0.5030 - val_accuracy: 0.7546 Epoch 35/70 8445/8445 [==============================] - 63s 8ms/step - loss: 0.4687 - accuracy: 0.7722 - val_loss: 0.5012 - val_accuracy: 0.7558 Epoch 36/70 8445/8445 [==============================] - 65s 8ms/step - loss: 0.4677 - accuracy: 0.7731 - val_loss: 0.5024 - val_accuracy: 0.7554 Epoch 37/70 8445/8445 [==============================] - 65s 8ms/step - loss: 0.4668 - accuracy: 0.7736 - val_loss: 0.5022 - val_accuracy: 0.7562 Epoch 38/70 8445/8445 [==============================] - 64s 8ms/step - loss: 0.4659 - accuracy: 0.7741 - val_loss: 0.5059 - val_accuracy: 0.7538 Epoch 39/70 8445/8445 [==============================] - 64s 8ms/step - loss: 0.4649 - accuracy: 0.7746 - val_loss: 0.5034 - val_accuracy: 0.7562 Epoch 40/70 8445/8445 [==============================] - 63s 8ms/step - loss: 0.4641 - accuracy: 0.7750 - val_loss: 0.5029 - val_accuracy: 0.7553 Epoch 41/70 8445/8445 [==============================] - 63s 8ms/step - loss: 0.4633 - accuracy: 0.7759 - val_loss: 0.5030 - val_accuracy: 0.7562 Epoch 42/70 8445/8445 [==============================] - 65s 8ms/step - loss: 0.4624 - accuracy: 0.7766 - val_loss: 0.5033 - val_accuracy: 0.7555 Epoch 43/70 8445/8445 [==============================] - 63s 8ms/step - loss: 0.4616 - accuracy: 0.7770 - val_loss: 0.5051 - val_accuracy: 0.7554 Epoch 44/70 8445/8445 [==============================] - 63s 7ms/step - loss: 0.4607 - accuracy: 0.7774 - val_loss: 0.5051 - val_accuracy: 0.7556 Epoch 45/70 8445/8445 [==============================] - 64s 8ms/step - loss: 0.4599 - accuracy: 0.7775 - val_loss: 0.5043 - val_accuracy: 0.7552 Epoch 46/70 8445/8445 [==============================] - 63s 7ms/step - loss: 0.4590 - accuracy: 0.7780 - val_loss: 0.5085 - val_accuracy: 0.7536 Epoch 47/70 8445/8445 [==============================] - 64s 8ms/step - loss: 0.4582 - accuracy: 0.7789 - val_loss: 0.5077 - val_accuracy: 0.7540 Epoch 48/70 8445/8445 [==============================] - 64s 8ms/step - loss: 0.4573 - accuracy: 0.7793 - val_loss: 0.5051 - val_accuracy: 0.7561 Epoch 49/70 8445/8445 [==============================] - 63s 8ms/step - loss: 0.4565 - accuracy: 0.7795 - val_loss: 0.5057 - val_accuracy: 0.7554 Epoch 50/70 8445/8445 [==============================] - 63s 7ms/step - loss: 0.4556 - accuracy: 0.7801 - val_loss: 0.5061 - val_accuracy: 0.7547 Epoch 51/70 8445/8445 [==============================] - 63s 7ms/step - loss: 0.4547 - accuracy: 0.7811 - val_loss: 0.5057 - val_accuracy: 0.7555 Epoch 52/70 8445/8445 [==============================] - 69s 8ms/step - loss: 0.4539 - accuracy: 0.7815 - val_loss: 0.5113 - val_accuracy: 0.7541 Epoch 53/70 8445/8445 [==============================] - 67s 8ms/step - loss: 0.4528 - accuracy: 0.7823 - val_loss: 0.5081 - val_accuracy: 0.7552 Epoch 54/70 8445/8445 [==============================] - 63s 7ms/step - loss: 0.4521 - accuracy: 0.7829 - val_loss: 0.5083 - val_accuracy: 0.7560 Epoch 55/70 8445/8445 [==============================] - 68s 8ms/step - loss: 0.4512 - accuracy: 0.7833 - val_loss: 0.5087 - val_accuracy: 0.7554 Epoch 56/70 8445/8445 [==============================] - 68s 8ms/step - loss: 0.4501 - accuracy: 0.7838 - val_loss: 0.5072 - val_accuracy: 0.7556 Epoch 57/70 8445/8445 [==============================] - 64s 8ms/step - loss: 0.4494 - accuracy: 0.7845 - val_loss: 0.5099 - val_accuracy: 0.7542 Epoch 58/70 8445/8445 [==============================] - 64s 8ms/step - loss: 0.4486 - accuracy: 0.7848 - val_loss: 0.5082 - val_accuracy: 0.7557 Epoch 59/70 8445/8445 [==============================] - 64s 8ms/step - loss: 0.4476 - accuracy: 0.7853 - val_loss: 0.5116 - val_accuracy: 0.7531 Epoch 60/70 8445/8445 [==============================] - 64s 8ms/step - loss: 0.4468 - accuracy: 0.7861 - val_loss: 0.5098 - val_accuracy: 0.7550 Epoch 61/70 8445/8445 [==============================] - 64s 8ms/step - loss: 0.4461 - accuracy: 0.7861 - val_loss: 0.5128 - val_accuracy: 0.7534 Epoch 61: early stopping Multiple Bidirectional LSTMs # Finally, I trained a model with two bidirectional LSTM layers. The model was trained for 80 epochs but early stopped at 27.\nmodel_multiple_bidi_lstm = tf.keras.Sequential([ tf.keras.layers.Embedding(vocab_size, embedding_dim, input_length=max_length), tf.keras.layers.Bidirectional(tf.keras.layers.LSTM(embedding_dim, return_sequences=True)), tf.keras.layers.Bidirectional(tf.keras.layers.LSTM(embedding_dim)), tf.keras.layers.Dense(1, activation=\u0026#39;sigmoid\u0026#39;) ]) learning_rate = 0.0003 model_multiple_bidi_lstm.compile(loss=\u0026#39;binary_crossentropy\u0026#39;, optimizer=tf.keras.optimizers.Adam(learning_rate), metrics=[\u0026#39;accuracy\u0026#39;]) fit_model_and_show_results(model_multiple_bidi_lstm, \u0026#39;Mulitple Bidirectional LSTMs Model\u0026#39;, 80) Model: \u0026#34;sequential\u0026#34; _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= embedding (Embedding) (None, 26, 16) 64000 bidirectional (Bidirectiona (None, 26, 32) 4224 l) bidirectional_1 (Bidirectio (None, 32) 6272 nal) dense (Dense) (None, 1) 33 ================================================================= Total params: 74,529 Trainable params: 74,529 Non-trainable params: 0 _________________________________________________________________ Epoch 1/80 8445/8445 [==============================] - 108s 12ms/step - loss: 0.5320 - accuracy: 0.7330 - val_loss: 0.5075 - val_accuracy: 0.7502 Epoch 2/80 8445/8445 [==============================] - 93s 11ms/step - loss: 0.5004 - accuracy: 0.7529 - val_loss: 0.4983 - val_accuracy: 0.7547 Epoch 3/80 8445/8445 [==============================] - 95s 11ms/step - loss: 0.4878 - accuracy: 0.7604 - val_loss: 0.4934 - val_accuracy: 0.7569 Epoch 4/80 8445/8445 [==============================] - 94s 11ms/step - loss: 0.4755 - accuracy: 0.7682 - val_loss: 0.4893 - val_accuracy: 0.7581 Epoch 5/80 8445/8445 [==============================] - 93s 11ms/step - loss: 0.4649 - accuracy: 0.7745 - val_loss: 0.4862 - val_accuracy: 0.7633 Epoch 6/80 8445/8445 [==============================] - 95s 11ms/step - loss: 0.4573 - accuracy: 0.7803 - val_loss: 0.4870 - val_accuracy: 0.7667 Epoch 7/80 8445/8445 [==============================] - 96s 11ms/step - loss: 0.4514 - accuracy: 0.7834 - val_loss: 0.4818 - val_accuracy: 0.7671 Epoch 8/80 8445/8445 [==============================] - 93s 11ms/step - loss: 0.4465 - accuracy: 0.7870 - val_loss: 0.4843 - val_accuracy: 0.7666 Epoch 9/80 8445/8445 [==============================] - 94s 11ms/step - loss: 0.4418 - accuracy: 0.7899 - val_loss: 0.4865 - val_accuracy: 0.7651 Epoch 10/80 8445/8445 [==============================] - 93s 11ms/step - loss: 0.4374 - accuracy: 0.7927 - val_loss: 0.4861 - val_accuracy: 0.7650 Epoch 11/80 8445/8445 [==============================] - 94s 11ms/step - loss: 0.4335 - accuracy: 0.7949 - val_loss: 0.4897 - val_accuracy: 0.7647 Epoch 12/80 8445/8445 [==============================] - 94s 11ms/step - loss: 0.4293 - accuracy: 0.7973 - val_loss: 0.4921 - val_accuracy: 0.7651 Epoch 13/80 8445/8445 [==============================] - 93s 11ms/step - loss: 0.4253 - accuracy: 0.8005 - val_loss: 0.4978 - val_accuracy: 0.7637 Epoch 14/80 8445/8445 [==============================] - 93s 11ms/step - loss: 0.4212 - accuracy: 0.8030 - val_loss: 0.5027 - val_accuracy: 0.7620 Epoch 15/80 8445/8445 [==============================] - 96s 11ms/step - loss: 0.4173 - accuracy: 0.8049 - val_loss: 0.5035 - val_accuracy: 0.7580 Epoch 16/80 8445/8445 [==============================] - 93s 11ms/step - loss: 0.4135 - accuracy: 0.8072 - val_loss: 0.5147 - val_accuracy: 0.7602 Epoch 17/80 8445/8445 [==============================] - 94s 11ms/step - loss: 0.4098 - accuracy: 0.8092 - val_loss: 0.5142 - val_accuracy: 0.7593 Epoch 18/80 8445/8445 [==============================] - 93s 11ms/step - loss: 0.4059 - accuracy: 0.8109 - val_loss: 0.5228 - val_accuracy: 0.7592 Epoch 19/80 8445/8445 [==============================] - 93s 11ms/step - loss: 0.4021 - accuracy: 0.8133 - val_loss: 0.5306 - val_accuracy: 0.7563 Epoch 20/80 8445/8445 [==============================] - 94s 11ms/step - loss: 0.3982 - accuracy: 0.8158 - val_loss: 0.5338 - val_accuracy: 0.7562 Epoch 21/80 8445/8445 [==============================] - 93s 11ms/step - loss: 0.3942 - accuracy: 0.8174 - val_loss: 0.5354 - val_accuracy: 0.7542 Epoch 22/80 8445/8445 [==============================] - 93s 11ms/step - loss: 0.3906 - accuracy: 0.8202 - val_loss: 0.5426 - val_accuracy: 0.7551 Epoch 23/80 8445/8445 [==============================] - 94s 11ms/step - loss: 0.3867 - accuracy: 0.8223 - val_loss: 0.5519 - val_accuracy: 0.7488 Epoch 24/80 8445/8445 [==============================] - 93s 11ms/step - loss: 0.3826 - accuracy: 0.8245 - val_loss: 0.5540 - val_accuracy: 0.7512 Epoch 25/80 8445/8445 [==============================] - 92s 11ms/step - loss: 0.3789 - accuracy: 0.8264 - val_loss: 0.5698 - val_accuracy: 0.7537 Epoch 26/80 8445/8445 [==============================] - 94s 11ms/step - loss: 0.3756 - accuracy: 0.8286 - val_loss: 0.5584 - val_accuracy: 0.7506 Epoch 27/80 8445/8445 [==============================] - 93s 11ms/step - loss: 0.3720 - accuracy: 0.8303 - val_loss: 0.5737 - val_accuracy: 0.7462 Epoch 27: early stopping Conclusions and Next Steps # Taking a look at the results of the training, the multiple bidirectional LSTM model appeared to have the best validation accuracy at its 7th epoch with a validation accuracy of 76.71% before overfitting occurred. This is followed by the GRU, which had the second best validation accuracy at its 21st epoch with a validation accuracy of 76.41%.\nTaking a look at the data, my next steps would include seeing if I could boost the validation accuracy of the multiple bidirectional LSTM model. To do this, I can vary the dropout rate, perform a lasso and/or ridge regression, changing the vocabulary size, and adjusting the learning rate. I could also perform a GridSearch across these hyperparameters to automate this process. I could also perform several other data manipulation techniques when preparing the data, including lemmatization, tf-idf analysis, and n-gram analysis.\n","date":"18 January 2023","externalUrl":null,"permalink":"/projects/nlp-negation-sentiment/","section":"Projects","summary":"Developing a model that analyzes sentiment with negations present, by analyzing tweets that contain the words “no”, “not”, “never”, and “didn’t” and classifying them as either positive or negative.","title":"Negation Sentiment with NLP","type":"projects"},{"content":"","date":"18 January 2023","externalUrl":null,"permalink":"/tags/nlp/","section":"Tags","summary":"","title":"NLP","type":"tags"},{"content":"Write-ups of data science and machine learning projects, with code on GitHub.\n","date":"18 January 2023","externalUrl":null,"permalink":"/projects/","section":"Projects","summary":"Write-ups of data science and machine learning projects, with code on GitHub.\n","title":"Projects","type":"projects"},{"content":"","date":"18 January 2023","externalUrl":null,"permalink":"/tags/sentiment-analysis/","section":"Tags","summary":"","title":"Sentiment Analysis","type":"tags"},{"content":"","date":"18 January 2023","externalUrl":null,"permalink":"/tags/","section":"Tags","summary":"","title":"Tags","type":"tags"},{"content":"","date":"3 January 2023","externalUrl":null,"permalink":"/tags/energy/","section":"Tags","summary":"","title":"Energy","type":"tags"},{"content":"","date":"3 January 2023","externalUrl":null,"permalink":"/tags/forecasting/","section":"Tags","summary":"","title":"Forecasting","type":"tags"},{"content":" Introduction # Over the past several years, global events have rocked the energy markets: the Covid-19 pandemic caused the price of oil to crash, while a combination of inflation and the sanctions placed on Russia due to its invasion of Ukraine have caused the price of oil and natural gas to skyrocket. As a result, it is important to be able to forecast energy markets accurately despite global uncertainty on a day-to-day basis.\nWith this in mind, my project aims at developing a tool that can forecast energy markets one period into the future consistently and accurately and in a time-effective manner. This tool can help local governments allocate resources in times of volatility and help investors and high-frequency traders optimize trading futures. Specifically, by pulling publicly available data, I will train different univariate models on past values to forecast next day values.\nTools # I will be creating these models using Python. In particular, I will be using the pandas and numpy libraries to clean the data, the requests and json libraries to pull and parse the data, and the sklearn library to prepare the data. I will be using a XGBoost regressor, a Support Vector regressor, and a Random Forest regressor to forecast the data.\nimport requests import pandas as pd import numpy as np import json import matplotlib.pyplot as plt import itertools import math import datetime import xgboost as xgb from xgboost import XGBRegressor from sklearn.ensemble import RandomForestRegressor from sklearn.svm import SVR import sklearn from sklearn.metrics import mean_squared_error from sklearn.model_selection import GridSearchCV from sklearn.model_selection import cross_val_score from sklearn.model_selection import train_test_split from sklearn.metrics import mean_absolute_error from pylab import rcParams rcParams[\u0026#39;figure.facecolor\u0026#39;] = \u0026#39;white\u0026#39; rcParams[\u0026#39;axes.facecolor\u0026#39;] = \u0026#39;white\u0026#39; rcParams[\u0026#39;figure.figsize\u0026#39;] = 12,6 Pulling Data from the U.S. Energy Information Administration # The U.S. Energy Information Administration (EIA) collects and disseminates energy data to the public to promote efficient markets, sound policy making, and public education. The EIA makes its data available through dashboards available here: https://www.eia.gov/tools/. For people who want to pull data locally, the EIA provides documentation for their API here: https://www.eia.gov/opendata/documentation.php. To pull the data, one needs to register an API key. I blurred mine out, but one can easily be obtained from their website. However, their API has several problems. The first is that when trying to determine which data to pull, the options are often hidden, making querying data tedious. Additionally, the maximum number of points that can be pulled per request is 5000 datapoints. If one wants to pull more data, they would either have to paginate or filter their data in the API call, as opposed to in a dataframe tool, like Pandas, which would be typically more well known by the user. My first class, LoadQuery, addresses these issues.\nThe class LoadQuery begins by looping through the different pages available, from which the user can select which data they want while having all options available, and then choose which dataset they want to pull. Next, the user will choose the periodicity of the data, if it comes in multiple formats, as well as the type of data that they want to pull, again, if the data comes in multiple formats. Once chosen, the program will then paginate through the queries until all data is pulled. The data is then turned into a Pandas dataframe format, which can be accessed through the .df attribute of the LoadQuery object. The dataframe will be indexed by date.\ndate_switch = {\u0026#39;annual\u0026#39;:\u0026#39;%Y\u0026#39;, \u0026#39;monthly\u0026#39;:\u0026#39;%Y-%m\u0026#39;, \u0026#39;daily\u0026#39;:\u0026#39;%Y-%m-%d\u0026#39;} class LoadQuery: def __init__(self, header = \u0026#34;\u0026#34;,page = \u0026#34;\u0026#34;, key = \u0026#34;\u0026#34;,data = None, frequency = None,response = None, df = None, format = None): self.header = \u0026#39;https://api.eia.gov/v2\u0026#39; self.page = page ##Which page you want to access. Created by running load_page self.key= #XXXXXXXXX self.data = data ##The data you want to access with the query. Created by running choose_query self.frequency = frequency self.response = response self.df = df self.format = format def load_page(self): ##When running load_page, this creates api_url = self.header+self.page+self.key response = requests.get(api_url)#, verify = False) response = response.json() if \u0026#39;frequency\u0026#39; not in response[\u0026#39;response\u0026#39;].keys(): print(\u0026#39;Choose your id\u0026#39;) check = [] for i in response[\u0026#39;response\u0026#39;][\u0026#39;routes\u0026#39;]: print(\u0026#34;id: \u0026#34;, i[\u0026#39;id\u0026#39;], \u0026#34; Name: \u0026#34;, i[\u0026#39;name\u0026#39;],\u0026#39;\\r\u0026#39;) check.append(i[\u0026#39;id\u0026#39;]) next_page = input() if next_page not in check: print(\u0026#39;What you typed was not an option\u0026#39;) self.load_page() else: self.page+=\u0026#39;/\u0026#39;+next_page self.load_page() api_url = self.header+self.page+self.key response = requests.get(api_url) self.response = response.json() def choose_frequency(self): if len(self.response[\u0026#39;response\u0026#39;][\u0026#39;frequency\u0026#39;]) == 1: self.frequency = self.response[\u0026#39;response\u0026#39;][\u0026#39;frequency\u0026#39;][0][\u0026#39;id\u0026#39;] else: print(\u0026#39;Choose the frequency\u0026#39;) check = [] for i in self.response[\u0026#39;response\u0026#39;][\u0026#39;frequency\u0026#39;]: check.append(i[\u0026#39;id\u0026#39;]) print(i[\u0026#39;id\u0026#39;]) frequency = input() if frequency in check: self.frequency = frequency else: print(\u0026#39;What you typed was not an option\u0026#39;) self.choose_frequency() def choose_data(self): if len(self.response[\u0026#39;response\u0026#39;][\u0026#39;data\u0026#39;].keys()) == 1: self.data = list(self.response[\u0026#39;response\u0026#39;][\u0026#39;data\u0026#39;].keys())[0] else: print(\u0026#39;Choose the data type\u0026#39;) check = [] for i in self.response[\u0026#39;response\u0026#39;][\u0026#39;data\u0026#39;].keys(): check.append(i) print(i) data = input() if data in check: self.data = data else: print(\u0026#39;What you typed was not an option\u0026#39;) self.choose_data() def create_info(self): api_url = self.header+self.page+self.key+\u0026#39;\u0026amp;data[]=\u0026#39;+self.data+\u0026#39;\u0026amp;frequency=\u0026#39;+self.frequency response = requests.get(api_url) self.response = response.json() def create_every_response(self): self.format = date_switch[self.frequency] api_url = self.header+self.page+\u0026#34;/data\u0026#34;+self.key+\u0026#39;\u0026amp;data[]=\u0026#39;+self.data+\u0026#39;\u0026amp;frequency=\u0026#39;+self.frequency response = requests.get(api_url) data = response.json() final = data[\u0026#39;response\u0026#39;][\u0026#39;data\u0026#39;] total = data[\u0026#39;response\u0026#39;][\u0026#39;total\u0026#39;] offset = 5000 if total \u0026lt;= 5000: self.response = final else: num_loops = math.floor(total/5000) offset = 5000 for i in range(0,num_loops): api_url = self.header+self.page+\u0026#34;/data\u0026#34;+self.key+\u0026#39;\u0026amp;data[]=\u0026#39;+self.data+\u0026#39;\u0026amp;frequency=\u0026#39;+self.frequency+\u0026#39;\u0026amp;offset=\u0026#39;+str(offset) response = requests.get(api_url) data = response.json() final += data[\u0026#39;response\u0026#39;][\u0026#39;data\u0026#39;] offset += 5000 percent = (offset-5000)/total*100 print(\u0026#34;%.2f\u0026#34;%percent,\u0026#39;% of the way done\u0026#39;) self.response = final def get_total_number_of_data_points(self): api_url = self.header+self.page+\u0026#34;/data\u0026#34;+self.key+\u0026#39;\u0026amp;data[]=\u0026#39;+self.data+\u0026#39;\u0026amp;frequency=\u0026#39;+self.frequency response = requests.get(api_url) response = response.json() return response[\u0026#39;response\u0026#39;][\u0026#39;total\u0026#39;] def create_df(self): api_url = self.header+self.page+\u0026#34;/data\u0026#34;+self.key+\u0026#39;\u0026amp;data[]=\u0026#39;+self.data+\u0026#39;\u0026amp;frequency=\u0026#39;+self.frequency response = requests.get(api_url) data = response.json() self.df = pd.DataFrame(data[\u0026#39;response\u0026#39;][\u0026#39;data\u0026#39;]) def return_url(self): url = self.header+self.page+\u0026#34;/data\u0026#34;+self.key+\u0026#39;\u0026amp;data[]=\u0026#39;+self.data+\u0026#39;\u0026amp;frequency=\u0026#39;+self.frequency return url def create_data_df(self): self.df = pd.DataFrame(self.response) def create_index(self): self.df[\u0026#39;period\u0026#39;] = pd.to_datetime(self.df[\u0026#39;period\u0026#39;],format = self.format) self.df[\u0026#39;time_idx\u0026#39;] = pd.DatetimeIndex(self.df[\u0026#39;period\u0026#39;]) self.df = self.df.set_index(\u0026#39;time_idx\u0026#39;) self.df = self.df.sort_values(\u0026#39;time_idx\u0026#39;) def run_all(self): self.load_page() self.choose_frequency() self.choose_data() self.create_info() self.create_every_response() self.df = pd.DataFrame(self.response).drop_duplicates() self.create_index() print(\u0026#39;Data has been obtained\u0026#39;) Building the Models # To build and determine the best model, I built the ForecastModel class below. A ForecastModel object takes four parameters: the data one wants to forecast which was pulled from the query, the number of past periods to train the models on, the number of periods to forecast, and the name of the column in the dataframe being passed that contains the data.\nI perform a GridSearch across a Support Vector regressor, a XGBoost regressor, and a RandomForest regressor. First, I\nTo build the models, I first call the shift_values() method to shift the values being predicted up a row so that the features in each row will forecast the value for the next day. Next, I call train_models(), which calls the models and performs a GridSearch, trains each model on a 80/20 split, and then performs a forecast by using the datapoints forecasted as the features. I then store the error and the predictions for each model. Afterwards, I align the predictions with the dates they correspond to through update_df().\nFinally, I call plot_best_model() to plot with the smallest MAE, while plot_data() plots all three models trained from the GridSearch. Calling find_best_model() works through the above steps and plots the best model. Calling the regressors attribute will display the final models created for each regressor. Below, I work through two examples.\nparams = { \u0026#39;Support Vector Regressor\u0026#39;:{\u0026#34;C\u0026#34;: [0.5, 0.7], \u0026#39;kernel\u0026#39;:[\u0026#39;rbf\u0026#39;, \u0026#39;poly\u0026#39;]}, \u0026#39;RandomForest\u0026#39;:{\u0026#39;max_features\u0026#39;:[1.0,\u0026#39;sqrt\u0026#39;],\u0026#39;min_samples_split\u0026#39;:[2,5,10],\u0026#39;min_samples_leaf\u0026#39;:[1,2,4],\u0026#39;bootstrap\u0026#39;:[True,False]}, \u0026#39;XGBoost\u0026#39;:{\u0026#39;min_child_weight\u0026#39;: [5, 7, 9],\u0026#39;colsample_bytree\u0026#39;: [0.2, 0.4, 0.6],\u0026#39;max_depth\u0026#39;: [5, 7, 9]} } class ForecastModel: def __init__(self, df=None, num_shifts = None, num_predict = None, col = None, error = None, test=None, predictions = None, final_df = None, final_date = None, first_date = None, regressors=None): self.df = df self.col = col self.num_shifts = num_shifts self.num_predict = num_predict self.data = self.shift_values() self.error = error self.test = test self.predictions = predictions self.final_df = final_df self.final_date = final_date self.first_date = first_date self.regressors = regressors def create_regressors(self): regressors = { \u0026#39;Support Vector Regressor\u0026#39;: [SVR()], \u0026#39;RandomForest\u0026#39;:[RandomForestRegressor()], \u0026#39;XGBoost\u0026#39;: [XGBRegressor()] } self.regressors = regressors def shift_values(self): df_dummy = self.df[[self.col]].copy() for i in range(1,self.num_shifts+1): name = self.col + \u0026#34;_shift \u0026#34; + str(i) df_dummy[name] = df_dummy[self.col].shift(i) df_dummy[self.col] = df_dummy[self.col].shift(-1) df_dummy = df_dummy.dropna(how=\u0026#39;any\u0026#39;,axis=0) y = df_dummy[self.col] X = df_dummy.drop(self.col,axis=1) return X,y def train_models(self): X,y = self.shift_values() X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, shuffle=False) X_train = np.array(X_train) X_test = np.array(X_test) y_train = np.array(y_train) y_test = np.array(y_test) X_pred = X_test.copy() self.create_regressors() for reg in self.regressors: print(\u0026#39;Training \u0026#39;+reg) model = self.regressors[reg][0] #model.fit(X_train, y_train) grid = GridSearchCV(model, params[reg]) grid.fit(X_train, y_train) model = grid.best_estimator_ self.regressors[reg][0] = model for i in range(0,len(X_pred)-1): new = model.predict([X_pred[i]]) if i == 0: predictions = np.array([new[0]]) else: predictions = np.append(predictions,np.array([new[0]])) X_pred[i+1] = np.concatenate([new,X_test[i+1][:-1]]) error = mean_absolute_error(y_test[:-1], predictions) self.regressors[reg].append(error) self.regressors[reg].append(predictions) print(reg+\u0026#39; trained\u0026#39;) self.test = y_test def update_df(self): first_date = None final_date = None length = len(self.df[self.df[self.col] == self.test[0]].index) for i in range(0,length): x = self.df.index.get_loc(self.df[self.df[self.col] == self.test[0]].index[i]) new = x + len(self.test) - 1 if self.df.iloc[new][\u0026#39;value\u0026#39;] == self.test[-1]: first_date = x final_date = new + 1 break self.first_date = first_date self.final_date = final_date self.final_df = self.df.iloc[first_date:final_date][[self.col]].copy()[:-1] self.final_df[\u0026#39;test\u0026#39;] = self.test[:-1] for reg in self.regressors: self.final_df[reg+\u0026#39; preds\u0026#39;] = self.regressors[reg][2] def plot_best_model(self): left = 500 right = 100 min_error = self.regressors[\u0026#39;Support Vector Regressor\u0026#39;][1] best_model_name = \u0026#39;Support Vector Regressor\u0026#39; for reg in self.regressors: if self.regressors[reg][1] \u0026lt; min_error: min_error = self.regressors[reg][1] best_model_name = reg rcParams[\u0026#39;figure.figsize\u0026#39;] = 20,10 plt.plot(self.df[self.col], label = \u0026#39;Full Dataset\u0026#39;, color = \u0026#39;black\u0026#39;) plt.plot(self.final_df[best_model_name+\u0026#39; preds\u0026#39;], label = best_model_name+\u0026#39; Predictions\u0026#39;, color = \u0026#39;blue\u0026#39;) corr = str(round(self.final_df.corr()[best_model_name+\u0026#39; preds\u0026#39;][\u0026#39;value\u0026#39;],3)) min_error_name = \u0026#39; MAE: %.3f\u0026#39; % min_error plt.title(best_model_name+\u0026#39; Predictions r^2: \u0026#39;+corr+min_error_name) plt.legend() plt.show() plt.plot(self.df[self.col][self.first_date - left:self.final_date + right], label = \u0026#39;Previous\u0026#39;, color = \u0026#39;black\u0026#39;) plt.plot(self.final_df[best_model_name+\u0026#39; preds\u0026#39;], label = best_model_name+\u0026#39; Predictions\u0026#39;, color = \u0026#39;blue\u0026#39;) corr = str(round(self.final_df.corr()[reg+\u0026#39; preds\u0026#39;][\u0026#39;value\u0026#39;],3)) plt.title(best_model_name+\u0026#39; Predictions r^2: \u0026#39;+corr+min_error_name) plt.legend() plt.show() def find_best_model(self): self.train_models() self.update_df() self.plot_best_model() def plot_data(self): left = 500 right = 100 for reg in self.regressors: min_error = \u0026#39; MAE: %.3f\u0026#39; % self.regressors[reg][1] rcParams[\u0026#39;figure.figsize\u0026#39;] = 20,10 plt.plot(self.df[self.col][self.first_date - left:self.final_date + right], label = \u0026#39;Previous\u0026#39;, color = \u0026#39;black\u0026#39;) plt.plot(self.final_df[reg+\u0026#39; preds\u0026#39;], label = reg+\u0026#39; Predictions\u0026#39;, color = \u0026#39;blue\u0026#39;) corr = str(round(self.final_df.corr()[reg+\u0026#39; preds\u0026#39;][\u0026#39;value\u0026#39;],3)) plt.title(reg+\u0026#39; Predictions r^2: \u0026#39;+corr+min_error) plt.legend() plt.show() Forecasting Natural Gas Futures # Below, I walk through the process outlined above to forecast natural gas futures. Specifically, I will be forecasting the Contract 1 futures prices. Because natural gas contracts expire three business days prior to the first calendar day of the delivery month, Contract 1 contains the calendar month following the trade date. Information can be found at this link: https://www.eia.gov/dnav/ng/TblDefs/ng_pri_fut_tbldef2.asp.\nBelow, I create a LoadQuery object to pull the data that I want to forecast.\nnatural_gas_query = LoadQuery() natural_gas_query.run_all() Choose your id id: coal Name: Coal id: crude-oil-imports Name: Crude Oil Imports id: electricity Name: Electricity id: international Name: International id: natural-gas Name: Natural Gas id: nuclear-outages Name: Nuclear Outages id: petroleum Name: Petroleum id: seds Name: State Energy Data System (SEDS) id: steo Name: Short Term Energy Outlook id: densified-biomass Name: Densified Biomass id: total-energy Name: Total Energy id: aeo Name: Annual Energy Outlook id: ieo Name: International Energy Outlook id: co2-emissions Name: State CO2 Emissions natural-gas Choose your id id: sum Name: Summary id: pri Name: Prices id: enr Name: Exploration and Reserves id: prod Name: Production id: move Name: Imports and Exports/Pipelines id: stor Name: Storage id: cons Name: Consumption / End Use pri Choose your id id: sum Name: Natural Gas Prices id: fut Name: Natural Gas Spot and Futures Prices (NYMEX) id: rescom Name: Average Price of Natural Gas Delivered to Residential and Commercial Consumers by Local Distribution and Marketers in Selected States fut Choose the frequency weekly monthly daily annual daily 11.82 % of the way done 23.64 % of the way done 35.46 % of the way done 47.28 % of the way done 59.10 % of the way done 70.92 % of the way done 82.74 % of the way done 94.56 % of the way done Data has been obtained Now that the data has been obtained, I call the dataframe to see the different features and how to filter for Contract 1.\nnatural_gas_query.df[\u0026#39;series-description\u0026#39;].unique() array([\u0026#39;Natural Gas Futures Contract 4 (Dollars per Million Btu)\u0026#39;, \u0026#39;Natural Gas Futures Contract 2 (Dollars per Million Btu)\u0026#39;, \u0026#39;Natural Gas Futures Contract 1 (Dollars per Million Btu)\u0026#39;, \u0026#39;Natural Gas Futures Contract 3 (Dollars per Million Btu)\u0026#39;, \u0026#39;Henry Hub Natural Gas Spot Price (Dollars per Million Btu)\u0026#39;], dtype=object) I will filter on the series description \u0026lsquo;Natural Gas Futures Contract 1 (Dollars per Million Btu)\u0026rsquo;.\nseries_desc = \u0026#39;Natural Gas Futures Contract 1 (Dollars per Million Btu)\u0026#39; natural_gas_data = natural_gas_query.df[(natural_gas_query.df[\u0026#39;series-description\u0026#39;] == series_desc)] natural_gas_data = natural_gas_data.dropna() natural_gas_data.head() time_idx period duoarea area-name product product-name process process-name series series-description value units 1993-12-20 00:00:00 1993-12-20 00:00:00 Y35NY NEW YORK CITY EPG0 Natural Gas PE4 Future Contract 4 RNGC4 Natural Gas Futures Contract 4 (Dollars per Million Btu) 1.894 $/MMBTU 1993-12-21 00:00:00 1993-12-21 00:00:00 Y35NY NEW YORK CITY EPG0 Natural Gas PE4 Future Contract 4 RNGC4 Natural Gas Futures Contract 4 (Dollars per Million Btu) 1.83 $/MMBTU 1993-12-22 00:00:00 1993-12-22 00:00:00 Y35NY NEW YORK CITY EPG0 Natural Gas PE4 Future Contract 4 RNGC4 Natural Gas Futures Contract 4 (Dollars per Million Btu) 1.859 $/MMBTU 1993-12-23 00:00:00 1993-12-23 00:00:00 Y35NY NEW YORK CITY EPG0 Natural Gas PE4 Future Contract 4 RNGC4 Natural Gas Futures Contract 4 (Dollars per Million Btu) 1.895 $/MMBTU 1993-12-27 00:00:00 1993-12-27 00:00:00 Y35NY NEW YORK CITY EPG0 Natural Gas PE4 Future Contract 4 RNGC4 Natural Gas Futures Contract 4 (Dollars per Million Btu) 1.965 $/MMBTU I create the model by passing the dataframe natural_gas_data, train the model by having the previous 50 days predict the next day, forecast the next 150 days, and use the data in the column \u0026ldquo;value\u0026rdquo;.\nAfter training the models, the best model is graphed below. The Random Forest regressor was the best model, and had an r² value of 0.988 with the testing data and a mean average error of 0.158.\nnatural_gas_forecast = ForecastModel(df = natural_gas_data, num_shifts = 50, num_predict = 150, col = \u0026#39;value\u0026#39;) natural_gas_forecast.find_best_model() Training Support Vector Regressor Support Vector Regressor trained Training RandomForest RandomForest trained Training XGBoost XGBoost trained Below I plot the best predictions for each of the different regressors. In addition to the Random Forest regressor, the Support Vector regressor closely predicted the test data, with an r² of 0.987 and an MAE of 0.161, as well as the XGBoost regression, which had an r² of 0.986 and an MAE of 0.170.\nnatural_gas_forecast.plot_data() Forecasting Petroleum Futures # I\u0026rsquo;ll now run through the same process by forecasting New York Harbor No. 2 Heating Oil Future Contract 1.\npetroleum_forecast = LoadQuery() petroleum_forecast.run_all() Choose your id id: coal Name: Coal id: crude-oil-imports Name: Crude Oil Imports id: electricity Name: Electricity id: international Name: International id: natural-gas Name: Natural Gas id: nuclear-outages Name: Nuclear Outages id: petroleum Name: Petroleum id: seds Name: State Energy Data System (SEDS) id: steo Name: Short Term Energy Outlook id: densified-biomass Name: Densified Biomass id: total-energy Name: Total Energy id: aeo Name: Annual Energy Outlook id: ieo Name: International Energy Outlook id: co2-emissions Name: State CO2 Emissions petroleum Choose your id id: sum Name: Summary id: pri Name: Prices id: crd Name: Crude Reserves and Production id: pnp Name: Refining and Processing id: move Name: Imports/Exports and Movements id: stoc Name: Stocks id: cons Name: Consumption/Sales pri Choose your id id: gnd Name: Weekly Retail Gasoline and Diesel Prices id: spt Name: Spot Prices id: fut Name: NYMEX Futures Prices id: wfr Name: Weekly Heating Oil and Propane Prices (October - March) id: refmg Name: Refiner Gasoline Prices by Grade and Sales Type id: refmg2 Name: U.S. Refiner Gasoline Prices by Formulation, Grade, Sales Type id: refoth Name: Refiner Petroleum Product Prices by Sales Type id: allmg Name: Gasoline Prices by Formulation, Grade, Sales Type id: dist Name: No. 2 Distillate Prices by Sales Type id: prop Name: Propane (Consumer Grade) Prices by Sales Type id: resid Name: Residual Fuel Oil Prices by Sales Type id: dfp1 Name: Domestic Crude Oil First Purchase Prices by Area id: dfp2 Name: Domestic Crude Oil First Purchase Prices for Selected Crude Streams id: dfp3 Name: Domestic Crude Oil First Purchase Prices by API Gravity id: rac2 Name: Refiner Acquisition Cost of Crude Oil id: imc1 Name: F.O.B. Costs of Imported Crude Oil by Area id: imc2 Name: F.O.B. Costs of Imported Crude Oil for Selected Crude Streams id: imc3 Name: F.O.B. Costs of Imported Crude Oil by API Gravity id: land1 Name: Landed Costs of Imported Crude by Area id: land2 Name: Landed Costs of Imported Crude for Selected Crude Streams id: land3 Name: Landed Costs of Imported Crude by API Gravity id: ipct Name: Percentages of Total Imported Crude Oil by API Gravity fut Choose the frequency weekly daily monthly annual daily 3.68 % of the way done 7.36 % of the way done 11.05 % of the way done 14.73 % of the way done 18.41 % of the way done 22.09 % of the way done 25.78 % of the way done 29.46 % of the way done 33.14 % of the way done 36.82 % of the way done 40.50 % of the way done 44.19 % of the way done 47.87 % of the way done 51.55 % of the way done 55.23 % of the way done 58.91 % of the way done 62.60 % of the way done 66.28 % of the way done 69.96 % of the way done 73.64 % of the way done 77.33 % of the way done 81.01 % of the way done 84.69 % of the way done 88.37 % of the way done 92.05 % of the way done 95.74 % of the way done 99.42 % of the way done Data has been obtained petroleum_forecast.df.head() time_idx period duoarea area-name product product-name process process-name series series-description value units 1980-01-02 00:00:00 1980-01-02 00:00:00 Y35NY NEW YORK CITY EPD2F No 2 Fuel Oil / Heating Oil PE1 Future Contract 1 EER_EPD2F_PE1_Y35NY_DPG New York Harbor No. 2 Heating Oil Future Contract 1 (Dollars per Gallon) 0.821 $/GAL 1980-01-02 00:00:00 1980-01-02 00:00:00 Y35NY NEW YORK CITY EPD2F No 2 Fuel Oil / Heating Oil PE3 Future Contract 3 EER_EPD2F_PE3_Y35NY_DPG New York Harbor No. 2 Heating Oil Future Contract 3 (Dollars per Gallon) 0.89 $/GAL 1980-01-03 00:00:00 1980-01-03 00:00:00 Y35NY NEW YORK CITY EPD2F No 2 Fuel Oil / Heating Oil PE1 Future Contract 1 EER_EPD2F_PE1_Y35NY_DPG New York Harbor No. 2 Heating Oil Future Contract 1 (Dollars per Gallon) 0.827 $/GAL 1980-01-03 00:00:00 1980-01-03 00:00:00 Y35NY NEW YORK CITY EPD2F No 2 Fuel Oil / Heating Oil PE3 Future Contract 3 EER_EPD2F_PE3_Y35NY_DPG New York Harbor No. 2 Heating Oil Future Contract 3 (Dollars per Gallon) 0.866 $/GAL 1980-01-04 00:00:00 1980-01-04 00:00:00 Y35NY NEW YORK CITY EPD2F No 2 Fuel Oil / Heating Oil PE3 Future Contract 3 EER_EPD2F_PE3_Y35NY_DPG New York Harbor No. 2 Heating Oil Future Contract 3 (Dollars per Gallon) 0.88 $/GAL petroleum_forecast.df[\u0026#39;series-description\u0026#39;].unique() array([\u0026#39;New York Harbor No. 2 Heating Oil Future Contract 3 (Dollars per Gallon)\u0026#39;, \u0026#39;New York Harbor No. 2 Heating Oil Future Contract 1 (Dollars per Gallon)\u0026#39;, \u0026#39;Cushing, OK Crude Oil Future Contract 3 (Dollars per Barrel)\u0026#39;, \u0026#39;Cushing, OK Crude Oil Future Contract 1 (Dollars per Barrel)\u0026#39;, \u0026#39;New York Harbor Regular Gasoline Future Contract 3 (Dollars per Gallon)\u0026#39;, \u0026#39;New York Harbor Regular Gasoline Future Contract 1 (Dollars per Gallon)\u0026#39;, \u0026#39;Cushing, OK Crude Oil Future Contract 2 (Dollars per Barrel)\u0026#39;, \u0026#39;Cushing, OK Crude Oil Future Contract 4 (Dollars per Barrel)\u0026#39;, \u0026#39;Mont Belvieu, Tx Propane Future Contract 1 (Dollars per Gallon)\u0026#39;, \u0026#39;Mont Belvieu, Tx Propane Future Contract 4 (Dollars per Gallon)\u0026#39;, \u0026#39;New York Harbor No. 2 Heating Oil Future Contract 4 (Dollars per Gallon)\u0026#39;, \u0026#39;New York Harbor Regular Gasoline Future Contract 2 (Dollars per Gallon)\u0026#39;, \u0026#39;New York Harbor Regular Gasoline Future Contract 4 (Dollars per Gallon)\u0026#39;, \u0026#39;New York Harbor No. 2 Heating Oil Future Contract 2 (Dollars per Gallon)\u0026#39;, \u0026#39;Mont Belvieu, Tx Propane Future Contract 2 (Dollars per Gallon)\u0026#39;, \u0026#39;Mont Belvieu, Tx Propane Future Contract 3 (Dollars per Gallon)\u0026#39;, \u0026#39;New York Harbor Reformulated RBOB Regular Gasoline Future Contract 2 (Dollars per Gallon)\u0026#39;, \u0026#39;New York Harbor Reformulated RBOB Regular Gasoline Future Contract 1 (Dollars per Gallon)\u0026#39;, \u0026#39;New York Harbor Reformulated RBOB Regular Gasoline Future Contract 4 (Dollars per Gallon)\u0026#39;, \u0026#39;New York Harbor Reformulated RBOB Regular Gasoline Future Contract 3 (Dollars per Gallon)\u0026#39;], dtype=object) petreoleum_forecast_data.df = petroleum_forecast.df[petroleum_forecast.df[\u0026#39;series-description\u0026#39;] == \u0026#39;New York Harbor No. 2 Heating Oil Future Contract 1 (Dollars per Gallon)\u0026#39;] petreoleum_forecast_data.df = petreoleum_forecast_data.df.dropna() After training the different models, the Random Forest regressor again the smallest MAE at 0.058 and the largest r² at 0.99 for the testing data.\npetroleum_forecast_model = ForecastModel(df = petreoleum_forecast_data.df, num_shifts = 50, num_predict = 150, col = \u0026#39;value\u0026#39;) petroleum_forecast_model.find_best_model() Training Support Vector Regressor Support Vector Regressor trained Training RandomForest RandomForest trained Training XGBoost XGBoost trained Looking at the other the models, we see that the Support Vector regressor and the XGBoost regressor also captured the trend.\npetroleum_forecast_model.plot_data() Conclusions and Next Steps # Taking a look at the natural gas and petroleum futures predictions, we are able to see that the models captured the volatility due to the Covid-19 pandemic and the invasion of Ukraine in their forecasting fairly well. Although the petroleum forecasts had more difficulty during the invasion of Ukraine, the Random Forest model captured the overall movement. To address this, a model that includes news events could be helpful, as I discuss below. As a result, it seems that univariate time-series models can be useful and cost-effective in forecasting markets in the short term.\nI believe that there are several ways to improve this project. First, when testing the models across other datasets, some models were more seasonal than the data shown above. As a result, when data appears more seasonal, performing a seasonality decomposition could increase model performance while decreasing complexity. Secondly, there were some datasets where models had more difficulty making predictions. Creating a multivariate model that incorporates features such as GDP, local and international conflicts, different weather events, and location could improve model performance as well. Finally, this model only included 1 period forecasts. It would be interesting to see how this approach fares in predicting multi-period forecasts, such as a 7 day forecast or a 30 day forecast. To accomplish this, including an RNN neural net model could be helpful in creating these forecasts.\n","date":"3 January 2023","externalUrl":null,"permalink":"/projects/forecasting-energy-futures/","section":"Projects","summary":"A tool that pulls publicly available data and trains univariate models on past values to forecast energy markets one period into the future consistently and accurately.","title":"Predicting Energy Futures","type":"projects"},{"content":"","date":"3 January 2023","externalUrl":null,"permalink":"/tags/regression/","section":"Tags","summary":"","title":"Regression","type":"tags"}]