Today we’ll look at performing sentiment analysis using F# and ML.NET. A new version (v0.9.0) has recently been released, so we use this as an opportunity to play with some new functionality. The goal of today’s post will be to perform sentiment analysis on movie reviews from IMDB.
Note: ML.NET is still evolving, this post was written using Microsoft.ML v0.9.0.
We’ll use .NET Core version 2.2. If you don’t have it installed, head out to the .NET Core Downloads page. Select SDK for your platform. Tangential, but you can also get here by going to dot.net, then navigating to Downloads and .NET Core.
With that out of the way, create a console F# project, then add the ML.NET package.
1 2 3
dotnet new console --language F# --name MLNet-SentimentAnalysis cd MLNet-SentimentAnalysis dotnet add package Microsoft.ML --version 0.9.0
Next, it is time to get the data. The source we will use for this post is from UCI. The datafile can be found here. The zip file contains examples for IMDB, Yelp, and Amazon, but we’ll stick with IMDB for this post.
mkdir data && cd data curl -O https://archive.ics.uci.edu/ml/machine-learning-databases/00331/sentiment%20labelled%20sentences.zip
Here is a sample of what the data looks like. There is no header row. The tab separated columns represent 1) the review’s text 2) the sentiment where 1 = positive and 0 = negative.
1 2 3 4
Long, whiny and pointless. 0 But I recommend waiting for their future efforts, let this one go. 0 Excellent cast, story line, performances. 1 Totally believable. 1
Now that we have the data, time to get to the code. First there is some namespace setup.
1 2 3
open System.IO open Microsoft.ML open Microsoft.ML.Data
Here are the data types to be used. SentimentData is for loading data, SentimentPrediction is for performing predictions. Here we also get our first taste of 0.9.0. As we’ll see later we can use the SentimentData type for loading. To enable this we will add [<LoadColumn(column position)>] to the members. I have also included Probability. This is not a real column, nor is it needed for training. I have included it because it is a required field when extracting performance metrics. I feel like I shouldn’t need to include it here, but for now it’s the only way I got it to work. The CreateTextReader now accepts a datatype for driving the loading process. Once the data reader is setup, we also perform a train/test split of 70/30, respectively.
### Schema SentimentText: Text Label: Bool Probability: R4
Next we setup the training pipeline. There are other options, like FastTree, but we’ll use FastForest for today’s post. We’ll also take the defaults, but as with previous trainers we’ve looked at, we can provide custom hyperparameters. Once the pipeline is setup, we run Fit to build the model.
1 2 3 4 5 6 7
let pipeline = ml .Transforms.Text.FeaturizeText("SentimentText", "Features") .Append(ml.BinaryClassification.Trainers.FastForest()) // Example of custom hyperparameters // .Append(mlContext.BinaryClassification.Trainers.FastForest(numTrees = 500, numLeaves = 100, learningRate = 0.0001)) let model = pipeline.Fit(trainData)
Any good machine learning process requires performance evaluation. For that we’ll look at two aspects. First, ML.NET provides evaluators for the trainers. I’ve cherry-picked a couple of the available BinaryClassificationEvaluator metrics. Second, we can perform a preview of the predictions, which allows us to see the sentiment value along with the actual and predicted labels, as well as the score. There are other items in the view as well that I left in to show the extent of the reporting. Then we can run evaluation’s against the train and test sets.
As imagined, the metrics are better when run against the training data. The much better view of prediction quality is when run against the testing data. As expected, the model doesn’t perform as well against the test set, there is probably some more work that needs done here. The Preview is also useful when diagnosing more detailed problems, since it shows scores and label predictions. Not related to the results, but the stratification value is used for the train/test split.
SentimentText: Not sure who was more lost - the flat characters or the audience, nearly half of whom walked out. Label: false StratificationColumn: 0.595641375f Features: Sparse vector of size 7818, 110 explicit values PredictedLabel: false Score: -54.9804649f
SentimentText: Attempting artiness with black & white and clever camera angles, the movie disappointed - became even more ridiculous - as the acting was poor and the plot and lines almost non-existent. Label: false StratificationColumn: 0.58837676f Features: Sparse vector of size 7818, 188 explicit values PredictedLabel: false Score: -13.02876f
SentimentText: Very little music or anything to speak of. Label: false StratificationColumn: 0.753678203f Features: Sparse vector of size 7818, 52 explicit values PredictedLabel: false Score: -5.37574673f
SentimentText: The best scene in the movie was when Gerardo is trying to find a song that keeps running through his head. Label: true StratificationColumn: 0.967485666f Features: Sparse vector of size 7818, 118 explicit values PredictedLabel: true Score: 41.7043114f
SentimentText: The rest of the movie lacks art, charm, meaning... If it's about emptiness, it works I guess because it's empty. Label: false StratificationColumn: 0.929597497f Features: Sparse vector of size 7818, 119 explicit values PredictedLabel: false Score: -15.2312632f
SentimentText: Saw the movie today and thought it was a good effort, good messages for kids. Label: true StratificationColumn: 0.185497403f Features: Sparse vector of size 7818, 83 explicit values PredictedLabel: true Score: 25.0270023f
SentimentText: The movie showed a lot of Florida at it's best, made it look very appealing. Label: true StratificationColumn: 0.250951052f Features: Sparse vector of size 7818, 86 explicit values PredictedLabel: true Score: 18.1396465f
SentimentText: In other words, the content level of this film is enough to easily fill a dozen other films. Label: true StratificationColumn: 0.229808331f Features: Sparse vector of size 7818, 90 explicit values PredictedLabel: true Score: 20.8655605f
Now that model fitting and some evaluation has been performed, we need to make a prediction function. As with so many things so far, this is simple to do.
let predictor = model.CreatePredictionEngine<SentimentData, SentimentPrediction>(ml)
Once the prediction function is in place, we can run predictions and see their underlying scores.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
let tests = [ "It was cool, cute, and funny."; "It was slow and boring."; "It was the greatest thing I've seen." ]
tests |> List.iter (fun x -> let input = SentimentData() input.SentimentText <- x
Text : It was cool, cute, and funny. Prediction : true Score : 12.2628
Text : It was slow and boring. Prediction : false Score : -10.0353
Text : It was the greatest thing I've seen. Prediction : true Score : 68.4614
This is all well and good, but to be useful we need to be able to save a model to a file for later use. Here we have the ability to save and reload a model file.
1 2 3 4 5 6 7 8 9 10 11 12
// Save model to file let saveModel (ml:MLContext) trainedMode = use fsWrite = new FileStream("test-model.zip", FileMode.Create, FileAccess.Write, FileShare.Write) ml.Model.Save(model, fsWrite)
saveModel ml model
// Load model from file use fsRead = new FileStream("test-model.zip", FileMode.Open, FileAccess.Read, FileShare.Read) let mlReloaded = MLContext() let modelReloaded = TransformerChain.LoadFrom(mlReloaded, fsRead) let predictorReloaded = modelReloaded.CreatePredictionEngine<SentimentData, SentimentPrediction>(mlReloaded)
Once the model file has been reloaded, we can run a sample prediction. We just need to create the prediction function against and away we go.
1 2 3 4 5 6 7 8
let test1 = SentimentData() test1.SentimentText <- tests.
Here are the prediction results from a saved model.
1 2 3
Text : It was cool, cute, and funny. Prediction (Reloaded) : true Score (Reloaded) : 12.2628
This has been a brief look into sentiment analysis using F# and ML.NET. It has been a pleasure to see the framework progress. It is even more enjoyable performing these types of workloads using F#. Until next time. Thanks.