![北京/上海/广州 0xFF Life's pathetic, go Pythonic!
Interface - Message
● message (<= 1000)
● params
>>> {
>>> "message": "My raw message
with interpreted strings like %s",
>>> "params": ["this"]
>>> }](https://image.slidesharecdn.com/diveintosentry-151228050419/85/Dive-into-sentry-26-320.jpg)
![北京/上海/广州 0xFF Life's pathetic, go Pythonic!
Interface - Template
● A rendered template (generally
used like a single frame in a
stacktrace).
● The attributes ``filename``,
``context_line``, and ``lineno`` are
required.
>>> {
>>> "abs_path": "/real/file/name.html"
>>> "filename": "file/name.html",
>>> "pre_context": [
>>> "line1",
>>> "line2"
>>> ],
>>> "context_line": "line3",
>>> "lineno": 3,
>>> "post_context": [
>>> "line4",
>>> "line5"
>>> ],
>>> }](https://image.slidesharecdn.com/diveintosentry-151228050419/85/Dive-into-sentry-29-320.jpg)
![北京/上海/广州 0xFF Life's pathetic, go Pythonic!
Interface - Stacktrace
● Python Frame
>>> {
>>> "frames": [{
>>> "abs_path": "/real/file/name.py"
>>> "filename": "file/name.py",
>>> "function": "myfunction",
>>> "vars": {
>>> "key": "value"
>>> },
>>> "pre_context": [
>>> "line1",
>>> "line2"
>>> ],
>>> "context_line": "line3",
>>> "lineno": 3,
>>> "in_app": true,
>>> "post_context": [
>>> "line4",
>>> "line5"
>>> ],
>>> }],
>>> "frames_omitted": [13, 56]
>>> }](https://image.slidesharecdn.com/diveintosentry-151228050419/85/Dive-into-sentry-31-320.jpg)
![北京/上海/广州 0xFF Life's pathetic, go Pythonic!
Rollups & Sampling
● Rollups
○ Raven.captureException(ex, {fingerprint: ['my', 'custom', 'fingerprint']})
○ Raven.captureException(ex, {fingerprint: ['{{ default }}', 'other', 'data']})
● Sampling
○ Count
○ Time](https://image.slidesharecdn.com/diveintosentry-151228050419/85/Dive-into-sentry-43-320.jpg)
![北京/上海/广州 0xFF Life's pathetic, go Pythonic!
TIPS
● Webhooks:
○ 默认禁止访问内网 IP, 需要更改一下配置
● Timezone
○ SENTRY_DEFAULT_TIME_ZONE = 'Asia/Shanghai' 设置用户默认时区
● Public
○ SENTRY_PUBLIC = False 这个权限有点问题，不要开启
● Register
○ SENTRY_FEATURES['auth:register'] = False 禁止自己注册](https://image.slidesharecdn.com/diveintosentry-151228050419/85/Dive-into-sentry-63-320.jpg)
Dive into sentry
- 1. Dive into Sentry The modern error logging and aggregation platform XTao 09.19.2015 Beijing
- 2. 北京/上海/广州 0xFF Life's pathetic, go Pythonic! 徐涛 ● @ 豆瓣 ● (?:产品开发|运维)工程师 ● (?:CODE|DevOps|Git|Python) ● 2014 PyConChina Beijing ● https://blog.xtao.me ❏ Douban: @xtaooooo ❏ Twitter: @xtao ❏ Github: @xtao
- 3. Sentry 概述 Sentry 毕业于 Disqus https://engineering.disqus.com/ Sentry 历史 Sentry 是什么 DEMO
- 4. 北京/上海/广州 0xFF Life's pathetic, go Pythonic! 起源 ● 2010 ● http://disqus.com/ ● django-db-log (祖父) ● tl;dr Sentry and Raven are StarCraft 2（星际争 霸 2） units. ● driven-by-open-source commit 3c2e87573d3bd16f61cf08fece0638cc47a4fc22 Author: David Cramer <[email protected]> Date: Mon May 12 16:26:19 2008 +0000 initial working code djangodblog/__init__.py | 35 +++++++++++++++++++++++++++++++++++ djangodblog/models.py | 36 ++++++++++++++++++++++++++++++++++++ 2 files changed, 71 insertions(+)
- 5. 北京/上海/广州 0xFF Life's pathetic, go Pythonic! Sentry 5 ● 2012 ● Protocol Version 3 ● branch: 5.4.x-maint
- 6. 北京/上海/广州 0xFF Life's pathetic, go Pythonic! Sentry 6 ● 2013 ● Protocol Version 4 ● Protocol Version 5 ● Alerts ● Filters
- 7. 北京/上海/广州 0xFF Life's pathetic, go Pythonic! Sentry 7 ● 2014 ● Organizations ● TSDB ● Rules ● Web API ● Protocol Version 6 ● BIGINT ● Help Pages
- 8. 北京/上海/广州 0xFF Life's pathetic, go Pythonic! Senty 8 ● 2015 ? ● Most of the application has been overhauled and rewritten on top of React and our web API. ● beta
- 9. 北京/上海/广州 0xFF Life's pathetic, go Pythonic! Sentry 是什么 ● 一个错误记录和汇聚平台 ○ Server: Sentry (The Sentry Open Source Server) ○ Client: The Raven Clients.
- 10. 北京/上海/广州 0xFF Life's pathetic, go Pythonic! 为什么要用 Sentry ● 详细的错误息 ○ 某一行代码 (Python) ○ 某一个变量 (Python) ● 详细错误分类 ○ Tag ● 提醒 ● 合理的重复错误处理 ● 支持多种语言 ○ 对 Python 支持好 ❏ 额外的收获 ❏ 入门 ❏ 一个很好的 Django 项目，如果你要 学习如何使用 Django 的话，可以读 一下 Sentry 的源码 ❏ 进阶 ❏ Sentry 应该算是一个中型 Web 项目 了，如果你缺少 Web 项目开发经 验，也可以从源码中获取一些经验 ❏ 开源 ❏ 一个 Python 开源 Web 应用的示 例，数据迁移还是靠谱的
- 11. 北京/上海/广州 0xFF Life's pathetic, go Pythonic! Sentry - 服务端(7.x) ● Backend ○ Python ○ Django ○ Celery ● Frontend ○ JQuery ○ Backbone ○ Underscore ○ Bootstrap ○ Moment ● Database ○ MySQL ○ PostgreSQL ● KV ○ Cassandra ○ Riak ○ Redis ● Queue ○ Redis ○ RabbitMQ
- 12. 北京/上海/广州 0xFF Life's pathetic, go Pythonic! Raven - 官方支持的 Client ● Python ● JavaScript ● Node.js ● PHP ● Ruby ● Objective-C ● Java ● C# ● Go
- 13. DEMO 1. Hosted Sentry a. https://www.getsentry. com/signup/ b. 14-day Free Trial 2. Sentry On Promise a. https://docs.getsentry. com/on- premise/server/installati on/ b. Sentry Internal
- 15. 北京/上海/广州 0xFF Life's pathetic, go Pythonic! Raven 101 pip install raven --upgrade from raven import Client client = Client('___DSN___') try: 1 / 0 except ZeroDivisionError: client.captureException() def handle_request(request): client.context.merge({'user': { 'email': request.user.email }}) try: ... finally: client.context.clear()
- 16. 北京/上海/广州 0xFF Life's pathetic, go Pythonic! Raven 102 ● WSGI middleware ● raven/middleware.py ``` A WSGI middleware which will attempt to capture any uncaught exceptions and send them to Sentry. >>> from raven.base import Client >>> application = Sentry (application, Client()) ```
- 17. 北京/上海/广州 0xFF Life's pathetic, go Pythonic! DSN 101 '{PROTOCOL}://{PUBLIC_KEY}:{SECRET_KEY}@{HOST}/{PATH}{PROJECT_ID}' http://c44a73655e50454581da995bbedd392a: [email protected]/10 udp://c44a73655e50454581da995bbedd392a: [email protected]: 4008/10
- 19. 北京/上海/广州 0xFF Life's pathetic, go Pythonic! Event ● HTTP(DATA) ● UDP(DATA) ● EventManager ● Project ● Event ● Group ● EventMapping ○ event_id: uuid.uuid4(). hex ● UserReport ○ 用户反馈 Sentry 问题 ● post_process_group.delay ● index_event.delay
- 20. 北京/上海/广州 0xFF Life's pathetic, go Pythonic! Group ● hashes ○ checksum (provided by client) ○ fingerprint / (default + fingerprint) ○ default (first interface ordered by score) ● find group ○ find group at GroupHash by hash ○ first matched group ● sample event (count, time) ● regression (resolved event)
- 21. 北京/上海/广州 0xFF Life's pathetic, go Pythonic! Protocol ● CLIENT_RESERVED_ATTRS = ( ● 'project', ● 'event_id', ● 'message', ● 'checksum', ● 'culprit', ● 'fingerprint', ● 'level', ● 'time_spent', ● 'logger', { "event_id": "fc6d8c0c43fc4630ad850ee518f1b9d0", "culprit": "my.module.function_name", "timestamp": "2011-05-02T17:41:36", "message": "SyntaxError: Wattttt!" "sentry.interfaces.Exception": { "type": "SyntaxError": "value": "Wattttt!", "module": "__builtins__" } } ● 'server_name', ● 'site', ● 'timestamp', ● 'extra', ● 'modules', ● 'tags', ● 'platform', ● 'release', ● )
- 22. 北京/上海/广州 0xFF Life's pathetic, go Pythonic! UDP Protocol "AUTHnnDATA" ● AUTH ○ "Sentry key=value, key=value, …" ● DATA ○ json string ○ zlib ○ base64
- 23. 北京/上海/广州 0xFF Life's pathetic, go Pythonic! HTTP Protocol ● 用户认证跟 Web 复用了 ● /api/store ● GET/POST DATA
- 24. 北京/上海/广州 0xFF Life's pathetic, go Pythonic! Interface An interface is a structured representation of data, which may render differently than the default ``extra`` metadata in an event. ● to_python ● get_api_context ● to_json ● get_path ● get_alias ● get_hash ● get_score ● ...
- 25. 北京/上海/广州 0xFF Life's pathetic, go Pythonic! Interface - Exception ● 标准的 Python 异常 ● type, value, module ● stacktrace == sentry.interfaces. Stacktrace >>> { >>> "type": "ValueError", >>> "value": "My exception value", >>> "module": "__builtins__" >>> "stacktrace": { >>> # see sentry.interfaces.Stacktrace >>> } >>> }
- 26. 北京/上海/广州 0xFF Life's pathetic, go Pythonic! Interface - Message ● message (<= 1000) ● params >>> { >>> "message": "My raw message with interpreted strings like %s", >>> "params": ["this"] >>> }
- 27. 北京/上海/广州 0xFF Life's pathetic, go Pythonic! Interface - HTTP ● 常用的 HTTP 参数 >>> { >>> "url": "http://absolute.uri/foo", >>> "method": "POST", >>> "data": { >>> "foo": "bar" >>> }, >>> "query_string": "hello=world", >>> "cookies": "foo=bar", >>> "headers": { >>> "Content-Type": "text/html" >>> }, >>> "env": { >>> "REMOTE_ADDR": "192.168.0.1" >>> } >>> }
- 28. 北京/上海/广州 0xFF Life's pathetic, go Pythonic! Interface - Query ● 用于记录 SQL >>> { >>> "query": "SELECT 1" >>> "engine": "psycopg2" >>> }
- 29. 北京/上海/广州 0xFF Life's pathetic, go Pythonic! Interface - Template ● A rendered template (generally used like a single frame in a stacktrace). ● The attributes ``filename``, ``context_line``, and ``lineno`` are required. >>> { >>> "abs_path": "/real/file/name.html" >>> "filename": "file/name.html", >>> "pre_context": [ >>> "line1", >>> "line2" >>> ], >>> "context_line": "line3", >>> "lineno": 3, >>> "post_context": [ >>> "line4", >>> "line5" >>> ], >>> }
- 30. 北京/上海/广州 0xFF Life's pathetic, go Pythonic! Interface - User ● 定义一个用户 >>> { >>> "id": "unique_id", >>> "username": "my_user", >>> "email": "[email protected]" >>> "ip_address": "127.0.0.1", >>> "optional": "value" >>> }
- 31. 北京/上海/广州 0xFF Life's pathetic, go Pythonic! Interface - Stacktrace ● Python Frame >>> { >>> "frames": [{ >>> "abs_path": "/real/file/name.py" >>> "filename": "file/name.py", >>> "function": "myfunction", >>> "vars": { >>> "key": "value" >>> }, >>> "pre_context": [ >>> "line1", >>> "line2" >>> ], >>> "context_line": "line3", >>> "lineno": 3, >>> "in_app": true, >>> "post_context": [ >>> "line4", >>> "line5" >>> ], >>> }], >>> "frames_omitted": [13, 56] >>> }
- 32. 北京/上海/广州 0xFF Life's pathetic, go Pythonic! TSDB - 时序数据库 ● Dummy (none) ● InMemory (defaultdict) ● Redis (hashes) Redis: { "TSDBModel:epoch:shard": { "Key": Count } } # rollups must be ordered from highest granularity to lowest SENTRY_TSDB_ROLLUPS = ( # (time in seconds, samples to keep) (10, 360), # 60 minutes at 10 seconds (3600, 24 * 7), # 7 days at 1 hour (3600 * 24, 60), # 60 days at 1 day )
- 33. 北京/上海/广州 0xFF Life's pathetic, go Pythonic! NodeStore - KV 数据库 ● riak ● cassandra ● django (node table) ● 用于和数据一起存储一些特殊信息(比如不 适合存在数据库里的大文本等) ● validate ● create ● delete ● delete_multi ● get ● get_multi ● set ● set_multi ● generate_id ● cleanup
- 34. 北京/上海/广州 0xFF Life's pathetic, go Pythonic! Example class Event(Model): """ An individual event. """ __core__ = False ... time_spent = BoundedIntegerField(null=True) data = NodeField(blank=True, null=True)
- 35. 北京/上海/广州 0xFF Life's pathetic, go Pythonic! Cache ● django ○ filesystem ○ memcached ○ local memory ○ dummy ● redis ● set ● get ● delete ● redis: ○ from nydus.db import create_cluster ○ 支持 cluster ○ 重写了 rb，但还没有在已发布的版本里使用
- 36. 北京/上海/广州 0xFF Life's pathetic, go Pythonic! Example ● Cache 与 Model ○ db/models/manager.py ○ class BaseManager(Manager) ● get_from_cache ● updated by signal ● deleted by signal
- 37. 北京/上海/广州 0xFF Life's pathetic, go Pythonic! Buffer This is useful in situations where a single event might be happening so fast that the queue can't keep up with the updates. ● InProcess (no buffer) ● Redis ● 降低 MySQL 数据的 QPS (写) ● 支持 Cluster Redis ● Redis 2.6.12 or newer
- 38. 北京/上海/广州 0xFF Life's pathetic, go Pythonic! Buffer Internal ● 生产者 ● incr ● ● 'b:k:%s:%s' (hashmap, key_expire = 60 * 60 # 1 hour) ○ 'm' ○ 'f' ○ 'l+%s' ○ 'e+%s' ● 'b:p' (Sorted sets) ● 消费者 ● process pending ● process 'flush-buffers': { 'task': 'sentry.tasks.process_buffer.process_pending', 'schedule': timedelta(seconds=10), 'options': { 'expires': 10, 'queue': 'counters-0', } },
- 39. 北京/上海/广州 0xFF Life's pathetic, go Pythonic! Membership Roles ● Member *:read ● Admin *:write ● Owner *:delete Scoping has access to all teams ● 跟 GitHub 类似的组织结构以及权限控制 ● Organization - Owner, Admin, Member ● Team - (Role, Project) ● Project
- 40. 北京/上海/广州 0xFF Life's pathetic, go Pythonic! Sensitive Data ● 'password', ● 'secret', ● 'passwd', ● 'authorization', ● 'api_key', ● 'apikey', ● 'access_token', ● DEFAULT_SCRUBBED_FIELDS
- 41. 北京/上海/广州 0xFF Life's pathetic, go Pythonic! Notifications ● Rules ○ An event is first seen (the first event in a rollup) ○ An event changes state from resolved to unresolved ● State ○ Unresolved ○ Resolved ○ Muted ● Condition ● Action
- 42. 北京/上海/广州 0xFF Life's pathetic, go Pythonic! Tagging Events ● Event 分类 ● We’ll automatically index all tags for an event, as well as the frequency and the last time a value has been seen. ● TagValue ● GroupTagValue ● Added by buffer
- 43. 北京/上海/广州 0xFF Life's pathetic, go Pythonic! Rollups & Sampling ● Rollups ○ Raven.captureException(ex, {fingerprint: ['my', 'custom', 'fingerprint']}) ○ Raven.captureException(ex, {fingerprint: ['{{ default }}', 'other', 'data']}) ● Sampling ○ Count ○ Time
- 44. 北京/上海/广州 0xFF Life's pathetic, go Pythonic! Web Profile ● ?prof=1 ● DEBUG ● super user ● src/sentry/utils/debug.py def can(self, request): if 'prof' not in request.GET: return False if settings.DEBUG: return True if hasattr(request, 'user') and request.user.is_superuser: return True return False
- 45. 北京/上海/广州 0xFF Life's pathetic, go Pythonic! Sentry to Sentry ● 自举 ● DISABLE_RAVEN ● default: project id == 1 ● src/sentry/utils/raven.py
- 46. 北京/上海/广州 0xFF Life's pathetic, go Pythonic! PostgreSQL & Gevent ● psycopg2 ● src/sentry/utils/gevent.py ● Sentry 官方使用的应该是这个数据库，有非阻塞的 patch，支持异步
- 49. 北京/上海/广州 0xFF Life's pathetic, go Pythonic! Monitor
- 50. 北京/上海/广州 0xFF Life's pathetic, go Pythonic! 豆瓣 ● 应用 ○ Python (大部分) ○ Javascript (前端) ○ Go (少部分) ○ C++/C/Java (少量) ● 错误 ○ devtools (DIY, 已废弃) ○ Sentry
- 51. 北京/上海/广州 0xFF Life's pathetic, go Pythonic! 问题 ● 已经部署了一套 Sentry ● 5.x 使用 UDP 协议 ● 测试以及线上，有丢错误情况 ● 且比较明显 ● 但是这时还没有针对 Sentry 的监控 ● 开始研究黑盒 ● UDP Worker CPU 使用率比较高 ● UDP 是用 DNS 做负载均衡 ● DNS 因为 cache 问题，导致负载不平衡 ● 使用 Random 改善了 cache 带来的隐患 ● 查看 Worker 代码 ● Gevent / Eventlet 使用错误，没有 Monkey Patch ● 5.x 数据库压力较大，需要做合并写
- 52. 北京/上海/广州 0xFF Life's pathetic, go Pythonic! 升级 ● 5.x-maint support UDP ● but 7.x not ● We have to backport UDP to 7.x ● 幸好原来的接口还在 ❏ src/sentry/conf/server.py: ❏ #socket.setdefaulttimeout(5) ❏ src/sentry/coreapi.py: ❏ insert_data_to_database_sync (async to sync，Redis 内存放大太厉害，因为 cache 原因)
- 53. 北京/上海/广州 0xFF Life's pathetic, go Pythonic! Insert Queue ● insert_data_to_database - cache ● preprocess_event - queue ● save_event - queue ● ● insert_data_to_database_sync - queue
- 54. 北京/上海/广州 0xFF Life's pathetic, go Pythonic! 部署情况 ● HTTP x 4 (默认 Sentry 是用 Gunicorn 管理 Worker 的) ● UDP x 4 (开启了 Gevent，收到包后，扔到队列) ● Celery x 4 (Task consumer, 默认是开启 CPU_NUM 个 Worker) ● Celery Beat x 1 ● Cron：cleanup 21 (只保留 21 天的数据) ● HTTP 前面用 LVS + Nginx 做负载均衡 ● UDP 用 DNS 做负载均衡
- 55. 北京/上海/广州 0xFF Life's pathetic, go Pythonic! 内部配置 ● LDAP ○ 我们用的用户帐号系统 ○ 配置一下 Sentry 即可 ● MAIL ○ 配置 Sentry 邮件服务器 ● IRC ○ sentry-irc ○ 因为我们使用了 ircbot，稍微改了一下这个插件代码
- 56. 北京/上海/广州 0xFF Life's pathetic, go Pythonic! Why UDP ● 快 ● 应用不需要关心 Sentry 服务是否正常 ● 即使 Sentry 出问题，也不影响应用 ● 可以观察系统 UDP 丢包情况，判断 UDP 服务是否正常
- 57. 北京/上海/广州 0xFF Life's pathetic, go Pythonic! Celery ● 芹菜 ● 还没有吃透
- 58. 北京/上海/广州 0xFF Life's pathetic, go Pythonic! DBA ● Redis ○ Memory ○ QPS ○ CPU ○ Queue Size ● MySQL ○ QPS ■ update ■ insert ■ delete ■ select ○ thread
- 59. 北京/上海/广州 0xFF Life's pathetic, go Pythonic! Sentry ● Statsd ○ celery worker cpu ○ udp worker cpu ○ http worker cpu ● App 内统计 ○ task 执行时间 ○ task 执行数量
- 60. 北京/上海/广州 0xFF Life's pathetic, go Pythonic! UDP Received Packet (Sentry) ● udp worker ● 收包后记录 ● d = sock.recvfrom(self.BUF_SIZE) ● statsd.increment(STATSD_KEY_RECV)
- 61. 北京/上海/广州 0xFF Life's pathetic, go Pythonic! UDP Received Packet (Kernel) ● UDP Server 收到的包数 (by diamond) ● ~ $ sudo /sbin/iptables -t filter -I INPUT -i lan -p udp --dport 4008 -j ACCEPT ● ~ $ sudo /sbin/iptables -L INPUT 1 -nvx ● 52308183 414772916064 ACCEPT udp -- lan * 0.0.0.0/0 0.0.0.0/0 udp dpt:4008 ● pkts (UDP 完整包数，底层已经处理了分包问题) ● bts
- 62. 北京/上海/广州 0xFF Life's pathetic, go Pythonic! UDP Dropped Packets (Kernel) ● cat /proc/net/udp ● sl local_address rem_address st tx_queue rx_queue tr tm->when retrnsmt uid timeout inode ref pointer drops ● 41: 00000000:80CE 00000000:0000 07 00000000:00000000 00:00000000 00000000 6561 0 4110825944 2 ffff8809c23e5e40 0 ● cat /proc/net/snmp ● Udp: InDatagrams NoPorts InErrors OutDatagrams RcvbufErrors SndbufErrors ● Udp: 5416706536 993028 290598311 22725578190 4662160 1318 ● UdpLite: InDatagrams NoPorts InErrors OutDatagrams RcvbufErrors SndbufErrors ● UdpLite: 0 0 0 0 0 0
- 63. 北京/上海/广州 0xFF Life's pathetic, go Pythonic! TIPS ● Webhooks: ○ 默认禁止访问内网 IP, 需要更改一下配置 ● Timezone ○ SENTRY_DEFAULT_TIME_ZONE = 'Asia/Shanghai' 设置用户默认时区 ● Public ○ SENTRY_PUBLIC = False 这个权限有点问题，不要开启 ● Register ○ SENTRY_FEATURES['auth:register'] = False 禁止自己注册
- 64. 北京/上海/广州 0xFF Life's pathetic, go Pythonic! 下一步计划 ● 项目错误统计(QPS，Sentry 提供的图还不能满足需求) ● Profile 工具，有助于分析 Worker 瓶颈 (Celery) ● 应对雪崩式错误的处理方案(压测 Sentry) ● 尝试一下 MySQL + Redis + Gevent
- 65. Jobs ● 2016 校招 ● always 社招 ● TO: [email protected] ● 当然 python 也可以 ● TO: [email protected] ● 如果你想试试 js 也可以尝试 ● TO: [email protected] ● 详情: http://jobs.douban.com
- 66. 北京/上海/广州 0xFF Life's pathetic, go Pythonic!