Machine Learning with Microsoft Azure
Download as PPTX, PDF0 likes488 views
This document discusses using Microsoft Azure for machine learning with R. It covers reading data from various sources into R like local files, web URLs, Azure Blob storage, and SQL server. It then discusses preprocessing data, feature engineering, training ML models with functions like glm(), and evaluating models with metrics like AUC. It notes challenges of data and ML evolving rapidly and the need to scale. It proposes using Apache Spark on Azure via services like HDInsight and R Server to allow distributed, scalable ML in the cloud with R for enterprises.
![# { "0 10:23:26" "1 10:19:29" "1 10:20:56" } > { 0, 1, 1 }
getDay <- function(x) { strsplit(x, split = " ")[[1]][1] }
trans <- trans.raw %>%
# remove invalid rows
filter(
!is.na(amount) | amount != 0
) %>%
# transform data
mutate(
OperationType = factor(ifelse(amount > 0, "income", "withdraw")),
TransDay = as.numeric(sapply(tr_datetime, getDay)),
Amount = abs(amount)
) %>%
# remove redundant columns
select(
-c(tr_datetime, amount, term_id)
) %>%
# set column names
rename(
CustomerId = customer_id, MCC = mcc_code, TransType = tr_type
) %>%
# sort
arrange(
TransDay, Amount
)
Step 2: preprocessing data](https://image.slidesharecdn.com/devconfuture-170604090645/85/Machine-Learning-with-Microsoft-Azure-6-320.jpg)
![Step 4: training ML-model
# train model
model <- glm(formula = gender ~ ., family = binomial(link = "logit"), data = dt.train)
# score model
p <- predict(model, newdata = dt.test, type = "response")
pr <- prediction(p, dt.test$gender)
prf <- performance(pr, measure = "tpr", x.measure = "fpr")
plot(prf)
# evaluate model
auc <- performance(pr, measure = "auc")
auc <- auc@y.values[[1]]
auc](https://image.slidesharecdn.com/devconfuture-170604090645/85/Machine-Learning-with-Microsoft-Azure-9-320.jpg)
Machine Learning with Microsoft Azure
- 1. Machine Learning with Microsoft Azure #msdevcon Dmitry Petukhov, ML/DS Preacher, Coffee Addicted && Machine Intelligence Researcher @ OpenWay
- 2. R for Fun Prototyping developer PC code result RAM Data IDE RStudio or/and Visual Studio Runtime CRAN or/and Microsoft R Open Flexibility Distributed Scalable: horizontal, vertical Fault-tolerance Reliable OSS-based BigData-ready LSML Secure
- 3. R for full cycle development CRISP-DM Model evaluation Evaluate measures of quality model (ROC, RMSE, F-Score, etc.) Feature Selection** Feature Selection Feature Scaling (Normalization) Dimension Reduction Final Model Training ML algorithm Share results Revision FinalModelEvaluation Data Flow Cross-validation Training Dataset Test Dataset Source: http://0xCode.in/azure-ml-for-data-scientist This work is licensed under a Creative Commons Attribution 4.0 International License
- 4. Step 1: read data # 1. from local file system library(data.table) dt <- fread("data/transactions.csv") # > Read 6849346 rows and 6 (of 6) columns from 0.299 GB file in 00:00:31 # 2. from Web dt <- fread("https://raw.githubusercontent.com/greggles/mcc-codes/master/mcc_codes.csv", sep = ",", stringsAsFactors = F, header = T, colClasses = list(character = 2))) # > % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed # > 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0100 14872 100 14872 0 0 29744 0 --:--:-- -- :--:-- --:--:-- 31710 # 3. from Azure Blob Storage library(AzureSMR) sc <- createAzureContext(tenantID = "{TID}", clientID = "{CID}", authKey = "{KEY}") sc azureGetBlob(sc, storageAccount = "contestsdata", container = "financial", blob = "transactions.csv", type = "text")
- 5. Step 1: read data # 4. from MS SQL Server library(RODBC) # Provides database connectivity connectionString <- "Driver={ODBC Driver 13 for SQL Server};Server=tcp:msdevcon.database.windows.net,1433;Database=TransDb;Uid=..." trans.conn <- odbcDriverConnect(connectionString) # open RODBC connection sqlSave(trans.conn, mcc.raw, "MCC2", addPK = T) # save data to table mccFromDb <- sqlQuery(trans.conn, "SELECT * FROM MCC2 WHERE edited_description LIKE '%For Visa Only%'") # get data head(mccFromDb) #> rownames code edited_description combined_description #> 1 978 9700 Automated Referral Service ( For Visa Only) Automated Referral Service ( For Visa Only) #> 2 979 9701 Visa Credential Service ( For Visa Only) Visa Credential Service ( For Visa Only) #> 3 980 9702 GCAS Emergency Services ( For Visa Only) GCAS Emergency Services ( For Visa Only) #> 4 981 9950 Intra ??“ Company Purchases ( For Visa Only) Intra ??“ Company Purchases ( For Visa Only) Intra ??“ close(trans.conn) # * Excel, HDFS, Amazon S3, REST-services as data sources
- 6. # { "0 10:23:26" "1 10:19:29" "1 10:20:56" } > { 0, 1, 1 } getDay <- function(x) { strsplit(x, split = " ")[[1]][1] } trans <- trans.raw %>% # remove invalid rows filter( !is.na(amount) | amount != 0 ) %>% # transform data mutate( OperationType = factor(ifelse(amount > 0, "income", "withdraw")), TransDay = as.numeric(sapply(tr_datetime, getDay)), Amount = abs(amount) ) %>% # remove redundant columns select( -c(tr_datetime, amount, term_id) ) %>% # set column names rename( CustomerId = customer_id, MCC = mcc_code, TransType = tr_type ) %>% # sort arrange( TransDay, Amount ) Step 2: preprocessing data
- 7. Step 3: feature engineering # calculate stats library(dplyr) customers.stats <- trans.x %>% mutate(LogAmount = log(Amount)) %>% group_by(CustomerId, OperationType, Gender) %>% filter(n() > 30) %>% summarize( Min = min(LogAmount), P1 = quantile(LogAmount, probs = c(.01)), Q1 = quantile(LogAmount, probs = c(.25)), Mean = mean(LogAmount), Q3 = quantile(LogAmount, probs = c(.75)), P99 = quantile(LogAmount, probs = c(.99)), Max = max(LogAmount), Total = sum(Amount), Count = n(), StandDev = sd(LogAmount) ) %>% ungroup() # shape from long to wide table form library(reshape2) x <- dcast(customers.stats, CustomerId + Gender ~ OperationType, value.var = "Mean", fun.aggregate = mean)
- 8. Step 3: feature engineering library(ggplot2) ggplot(x, aes(x = income, y = withdraw)) + geom_point(alpha = 0.25, colour = "darkblue") + facet_grid(. ~ Gender) + xlab("Income, rub") + ylab("Withdraw, rub")
- 9. Step 4: training ML-model # train model model <- glm(formula = gender ~ ., family = binomial(link = "logit"), data = dt.train) # score model p <- predict(model, newdata = dt.test, type = "response") pr <- prediction(p, dt.test$gender) prf <- performance(pr, measure = "tpr", x.measure = "fpr") plot(prf) # evaluate model auc <- performance(pr, measure = "auc") auc <- [email protected][[1]] auc
- 10. Challenges Data Science evolve rapidly Data growing even faster Data >> Memory (now and evermore) We must scale better Complex infrastructure Zoo of frameworks May be cloud?
- 11. #msdevcon Big Data + Cloud + Machine Learning Долго, дорого, …
- 12. #msdevcon Apache Spark/Hadoop + Azure + R Server Доступен как PaaS-сервис
- 13. Application Server (Task Manager) Flexibility Distributed Large scalable Fault-tolerance Reliable OSS-based BigData-ready LSML Secure Team Head Node Worker Node DFS ML for the bloody Enterprise Version Control Distributed Execution Framework Tasks Big Data Cluster Tasks Pull code
- 14. Azure Blob Storage Microsoft R Server Team Head Node Worker Node HDFS API R for the Enterprise Apache Spark / Hadoop Tasks Azure HDInsight Tasks Pull code
- 15. Microsoft R Microsoft R Open and Microsoft R Server #R MicrosoftML #R Microsoft R Server for Azure HDInsight #PaaS R Server on Apache Spark Data Science VM #R #IaaS CNTK & GPU Instances #NN #GPU #OSS Batch AI Training preview #PaaS #NN #GPU Azure Machine Learning #PaaS R scripts, modules and models #R Jupyter Notebooks #R #SaaS R-to-cloud: AzureSMR, AzureML #R #OSS Cognitive Services #SaaS #NN SQL Server R Services #R #PaaS Power BI #R #Viz Execute R scripts Visual Studio R extensions for VS2015 R in-box-support for VS2017 MicrosoftAzure
- 16. © 2017, Dmitry Petukhov. CC BY-SA 4.0 license. Microsoft and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. Data Science must win!
- 17. Q&A Now or later (use contacts below) Ping me Habr: @codezombie All contacts: http://0xCode.in/author