Amazon SageMaker で始める機械学習
7 likes7,546 views
Amazon SageMaker is a fully managed service that enables developers and data scientists to build, train, and deploy machine learning (ML) models quickly. It provides algorithms, notebooks, APIs and scalable infrastructure for building ML models. Some key features of SageMaker include algorithms for common ML tasks, notebooks for developing models, APIs for training and deployment, and scalable infrastructure for training and hosting models. It also integrates with other AWS services like S3, EC2 and VPC.
Amazon SageMaker で始める機械学習
- 2. ?
- 3. AWS AWS Deep Learning AMIs EC2 GPUs EC2 CPUs IoT Edge Amazon SageMaker Amazon Mechanical Turk R E K O G N I T I O N R E K O G N I T I O N V I D E O P O L L Y T R A N S C R I B E T R A N S L A T E C O M P R E H E N D L E X KERAS
- 6. SageMaker
- 7. SageMaker
- 8. SageMaker
- 9. SageMaker
- 13. • • 1 1 • 1 • •
- 14. SageMaker
- 15. SageMaker 8
- 17. Jupyter Notebook • • • 4 ⎼ ml.t2 ⎼ ml.m4 ⎼ ml.p2 ⎼ ml.p3 • VPC ENI VPC
- 18. CreateTrainingJob API Docker • • 2 • m4, m5, c4, c5, p2, p3 • S3 https://docs.aws.amazon.com/sagemaker/latest/dg/API_CreateTrainingJob.html
- 19. CreateEndpoint API Docker • AB • • https://docs.aws.amazon.com/sagemaker/latest/dg/API_CreateEndpointConfig.html https://docs.aws.amazon.com/sagemaker/latest/dg/API_CreateEndpoint.html https://docs.aws.amazon.com/ja_jp/sagemaker/latest/dg/API_UpdateEndpoint.html https://docs.aws.amazon.com/ja_jp/sagemaker/latest/dg/API_UpdateEndpointWeightsAndCapacities.html
- 20. SageMaker
- 21. SageMaker 2 AWS SDK • • SageMaker SDK • • AWS SDK SageMaker SDK AWS SDK scikit-learn • Python Spark • Jupyter Notebook https://github.com/aws/sagemaker-python-sdk https://github.com/aws/sagemaker-spark
- 22. AWS SDK SageMaker SDK • SageMaker SDK Jupyter Notebook • AWS SDK SageMaker SDK AWS SDK
- 23. create-endpoint create-notebook-instance create-training-job delete-endpoint delete-notebook-instance describe-endpoint describe-notebook-instance … estimator = TensorFlow(…) estimator.set_hyperparameters(…) estimator.fit(…) predictor = estimator.deploy(…) Predictor.predict(…)
- 26. SageMaker
- 27. Linear Learner XGBoost (eXtreme Gradient Boosting) PCA k-means k-NN Factorization Machines Random Cut Forest (Amazon) LDA (Latent Dirichlet Allocation) SageMaker
- 28. SageMaker Image classification Object Detection seq2seq Neural Topic Model Blazing text (Amazon) DeepAR Forecasting (Amazon)
- 29. Image Classification ResNet • ResNet CNN • ILSVRC 2015 1 • ImageNet • use_pretrained_model 1 0 https://github.com/awslabs/amazon-sagemaker-examples/blob/master/introduction_to_amazon_algorithms/imageclassification_caltech/Image-classification-transfer-learning.ipynb https://www.cv-foundation.org/openaccess/content_cvpr_2016/papers/He_Deep_Residual_Learning_CVPR_2016_paper.pdf https://docs.aws.amazon.com/ja_jp/sagemaker/latest/dg/image-classification.html https://docs.aws.amazon.com/ja_jp/sagemaker/latest/dg/IC-Hyperparameter.html
- 30. Object Detection SSD • • VGG or ResNet • ImageNet • use_pretrained_model 1 0 https://github.com/awslabs/amazon-sagemaker-examples/blob/master/introduction_to_amazon_algorithms/object_detection_pascalvoc_coco/object_detection_image_json_format.ipynb https://arxiv.org/pdf/1512.02325.pdf https://docs.aws.amazon.com/sagemaker/latest/dg/object-detection.html https://docs.aws.amazon.com/ja_jp/sagemaker/latest/dg/IC-Hyperparameter.html
- 31. SageMaker S3
- 35. SageMaker
- 40. Tensorflow • model_fn • estimator_fn tensorflow.estimator • keras_model_fn tf.keras • train_input_fn • eval_input_fn • serving_input_fn • input_fn • output_fn https://docs.aws.amazon.com/sagemaker/latest/dg/tf-training-inference-code-template.html
- 41. 5
- 45. MXNet • train • save • model_fn: • transform_fn: • https://docs.aws.amazon.com/sagemaker/latest/dg/mxnet-training-inference-code-template.html
- 47. docker run IMAGE_ID train • train • estimator.fit() docker run train docker run IMAGE_ID serve • serve • estimator.fit() serve • predictor.predict() /invocations • https://docs.aws.amazon.com/sagemaker/latest/dg/your-algorithms-training-algo.html https://docs.aws.amazon.com/sagemaker/latest/dg/your-algorithms-inference-code.html
- 53. SageMaker
- 54. SageMaker 3 1: • SageMaker 2: GPU • AWS
- 57. • → • VPC
- 59. AWS SageMaker Presigned URL API • CreatePresignedNotebookInstanceUrl • AWS AWS https://docs.aws.amazon.com/sagemaker/latest/dg/API_CreatePresignedNotebookInstanceUrl.html CreatePresignedNotebookInstanceUrl CreateNotebookInstance Presigned Instance URL Returned Presigned Instance URL Notebook Instance Request Instance Created
- 61. SageMaker → instance_count 2 Tensorflow/Chainer/PyTorch/MXNet → instance_count → Docker SageMaker /opt/ml/input/config/resourceConfig.json
- 64. Tensorflow AWS Tensorflow Docker model_fn source_dir Tensorflow 1.4-1.9 Keras https://docs.aws.amazon.com/ja_jp/sagemaker/latest/dg/tf-examples.html
- 65. SageMaker Tensorflow/Chainer/PyTorch/MXNet github • SageMaker pull / • instance_type ‘local’ https://github.com/aws/sagemaker-python-sdk#local-mode https://github.com/aws/sagemaker-tensorflow-containers https://github.com/aws/sagemaker-mxnet-containers https://github.com/aws/sagemaker-chainer-container https://github.com/aws/sagemaker-pytorch-container
- 66. PIPE 2 • FILE: • PIPE: S3 API PIPE • Tensorflow TFRecord • MXNet RecordIO Chainer PyTorch PIPE
- 68. = 2 • SageMakerVariantInvocationsPerInstance ⎼ 1 1 • https://docs.aws.amazon.com/sagemaker/latest/dg/endpoint-auto-scaling.html#endpoint-auto-scaling-add-policy https://docs.aws.amazon.com/ja_jp/autoscaling/application/userguide/application-auto-scaling-target-tracking.html
- 69. Tensorflow AWS Tensorflow Docker Tensorflow Serving input_fn, output_fn https://docs.aws.amazon.com/ja_jp/sagemaker/latest/dg/tf-examples.html
- 73. : KMS key ID SSE-KMS • CreateTrainingJob / • CreateEndpointConfig • • Cloudtrail PCI DSS HIPPA https://aws.amazon.com/about-aws/whats-new/2018/01/aws-kms-based-encryption-is-now-available-in-amazon-sagemaker-training-and-hosting/ https://aws.amazon.com/about-aws/whats-new/2018/01/aws-cloudtrail-integration-is-now-available-in-amazon-sagemaker/ https://aws.amazon.com/about-aws/whats-new/2018/01/amazon-sagemaker-achieves-pci-dss-compliance/ https://aws.amazon.com/about-aws/whats-new/2018/04/access-amazon-vpc-resources-for-training-and-hosting-with-amazon-sageMaker/ https://aws.amazon.com/about-aws/whats-new/2018/05/Amazon-SageMaker-Achieves-HIPAA-Eligibility/ https://aws.amazon.com/jp/about-aws/whats-new/2018/06/amazon-sagemaker-inference-calls-are-supported-on-aws-privatelink/
- 74. : SageMaker S3 S3 VPC • S3 • S3 SageMaker API PrivateLink • SageMaker Service API • SageMaker Runtime API https://aws.amazon.com/about-aws/whats-new/2018/04/access-amazon-vpc-resources-for-training-and-hosting-with-amazon-sageMaker/ https://aws.amazon.com/jp/about-aws/whats-new/2018/06/amazon-sagemaker-inference-calls-are-supported-on-aws-privatelink/ https://aws.amazon.com/about-aws/whats-new/2018/08/amazon-sagemaker-apis-supported-on-aws-privatelink/
- 75. ML • SageMaker 1 ML • • 0.14 USD/GB/ • • 0.016 USD/GB https://aws.amazon.com/jp/sagemaker/pricing/
- 76. 3 SageMaker Example Notebooks • https://github.com/awslabs/amazon-sagemaker-examples SageMaker SDK • https://github.com/aws/sagemaker-python-sdk (Doc : https://readthedocs.org/projects/sagemaker/) SageMaker • https://docs.aws.amazon.com/ja_jp/sagemaker/latest/dg/whatis.html
- 78. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Web Services Japan, K. K. Amazon SageMaker
- 79. SageMaker https://aws.amazon.com/jp/console/ (Chrome, Firefox IE, Safari ) SageMaker
- 81. SageMaker
- 83. IAM S3 S3
- 84. • VPC, •
- 85. Jupyter Notebook • InService Jupyter Notebook • New → Terminal
- 86. Terminal cd SageMaker/ wget http://bit.ly/sm-handson -O handson.zip unzip handson.zip + (O) URL DL
- 88. • • XGBoost MNIST • Chainer on SageMaker • Chainer MLP MNIST • MNIST Factorization Machines