Repository files navigation Disaster Response Pipeline Project
This projects creates an webapp that helps classifying messsages for Disater Response.
The dataset is from Figure Eight
The webapp front-end used React, and the back-end, Flask.
We created two models, one using sklearn and nltk, the other using spaCy and transformers.
Install required libraries
pip install -r requirements.txt
Go to api directory cd api
Run flask run --no-debugger
It will take a while to download the almost 500MB spaCy model
Go to http://0.0.0.0:5000/
Then you can test messages
Run the ETL and ML pipelines
Run the following commands in the project's root directory to set up your database and model.
To run ETL pipeline that cleans data and stores in database
python data/process_data.py data/disaster_messages.csv data/disaster_categories.csv data/DisasterResponse.db
To run ML pipeline that trains classifier and saves
python models/train_classifier.py data/DisasterResponse.db models/classifier.pkl
Go to app dir
Make sure you have Node.js and yarn installed
yarn build
Train the spaCy + RoBERTa Transformer model
Make sure you have a GPU (it will take a very very long time on a CPU)
Make sure you have have installed spaCy with GPU support
Go to models/spacy folder
Run spacy project run all
api
react_app: static files for Application (took from build folder created after yarn build in the app folder)app.py: contains API functions, also serves the static filesdownload_model.py: helper function to download model
app
public: folder for public contentsrc
App.*: main application layoutClassifier.*: input box and box with classification labelsUpperBar.*: the upper barOverview.*: a table with graphs about properties of the dataset
data
DisasterResponse.db: a SQLite db with cleaned data, saved in the dataset tabledisaster_categories.csv: the raw disaster categories datadisaster_messages.csv: the raw disaster messages dataprocess_data.py: an ETL script for reading the raw data, transform it and save into SQLite db
models
nlp.py: helper functions for cleaning a text for classificationtrain_classifier.py: script for creating and training a modelspacy
configs\roberta.cfg: a spaCy configuration file of a modelscript\convert_corpus.py: a script that takes the SQLite DB and transform it for use in spaCyproject.yml: spaCy project configuration file
spaCy for building the classification network with transformersspaCy tranformers for loading transformers into modelspaCy projects for providing examples of spaCy projectsHugging Face for providing the RoBERTa transformer modelCreate React App for providing the template for Web ApplicationMaterial UI for providing a framework for the React ApplicationScikit-learn for providing machine learning toolsNLTK for providing several natural language processing toolsFlask for providing the framework to create an API that serves the Web Application
About
A Web Application to help classifying messages for Disaster Response
Resources
Stars
Watchers
Forks
You can’t perform that action at this time.
