Infrastructure as code might be literally impossible / Joe Domato (packagecloud.io)
0 likes•126 views
The document discusses the challenges and complexities of implementing infrastructure as code, highlighting issues such as coding language difficulties, security flaws in package management, and the cognitive load on developers. It emphasizes that achieving reproducible infrastructure is still elusive and suggests that the tech industry must take responsibility to improve software systems. The author expresses a need for greater innovation and collaboration to find better solutions for infrastructure management.
![sigprocmask
a syscall used
via
[sg]etcontext](https://image.slidesharecdn.com/3-171113151533/85/Infrastructure-as-code-might-be-literally-impossible-Joe-Domato-packagecloud-io-39-320.jpg)
![[sg]etcontext
used for threading
and exception
handling](https://image.slidesharecdn.com/3-171113151533/85/Infrastructure-as-code-might-be-literally-impossible-Joe-Domato-packagecloud-io-40-320.jpg)
![cognitive load
“People changing our Chef recipes to
make something work for them, but
then breaking everyone else’s [stuff] is
practically constant at [company] right
now.”
- my friend who works at [company]](https://image.slidesharecdn.com/3-171113151533/85/Infrastructure-as-code-might-be-literally-impossible-Joe-Domato-packagecloud-io-80-320.jpg)
Infrastructure as code might be literally impossible / Joe Domato (packagecloud.io)
- 1. infrastructure as code might be literally impossible joe damato packagecloud.io
- 2. hi, i’m joe i like computers i once had a blog called timetobleed.com @joedamato
- 5. BTW bit.ly/linux-networking literally everything there is to know about linux networking
- 6. infrastructure as code might be impossible because nothing works.
- 8. code makes Computer do complicated stuff in small steps
- 9. code each small step is made up of a keyword (and other stuff)
- 10. code and so the keywords let you use Computer
- 14. so, you need to use them defensively
- 15. Story Time opteron revision E mysql +
- 16. code some languages are perceived as easy, but are terribly difficult
- 18. An Aside You must be an expert in C to write good, fast Ruby/etc
- 19. that’s part of a different talk called: “high level languages don’t exist” But
- 20. hard or impossible to use these languages defensively enough
- 21. Story Time MRI segfaults MRI threading
- 23. Thus Your code does things outside of your reference frame
- 24. Unless You’ve read every line all the way down (you haven’t).
- 25. OK.
- 26. infra code makes Computer do complicated stuff in small steps
- 27. infra code has really high level ‘keywords’
- 28. package “blah-pkg" do version "1:1.2.8-1" action :install end
- 29. package { “blah-pkg“: ensure => present, source => “https://packagecloud.io/...”, provider => rpm, }
- 30. what if i told you
- 31. infra code operates outside of your reference frame, too
- 32. meaning
- 33. unless you’ve read every line all the way down…
- 34. you haven’t
- 35. OK.
- 36. some things you (probably) didn’t know
- 37. what if i told you…
- 38. an MRI bug once made puppet peg CPU usage
- 40. [sg]etcontext used for threading and exception handling
- 41. “The “puppet” process spends 40-60% time in “system time”, which lengthens the time a single puppet run takes from a few minutes to > 20 minutes.”
- 42. I wrote a fix for this bug that was never accepted upstream
- 44. a friend working at a huge company told me that without that patch, they couldn’t run puppet.
- 45. coincidence? “We’re working on rebuilding our entire client-side technology stack, so it takes fewer resources, runs faster, and is more maintainable.” - puppet blog
- 46. supposively they are supposively rebuilding (some/ all?) client side stuff in C++
- 47. similarly OHAI-330 Ohai crashes on Solaris 11, Ubuntu 12.04 in mixins/ command.rb: popen4
- 49. (The work around is to disable a major feature of the language.) workaround
- 50. what if i told you…
- 51. it’s impossible to install a program securely on most linuxes
- 53. No
- 54. YUM + GPG tl;dr: doesn’t work most of the time and is nearly impossible to get it working
- 55. Story Time pygpgme repo_gpgcheck gpg v3 signatures
- 56. gpg v3 signatures %__gpg_sign_cmd %{__gpg} gpg --force-v3-sigs --digest-algo=sha1 --batc verbose --no-armor --passphrase-fd 3 --no-secm warning -u "%{_gpg_name}" -sbo % {__signature_filename} %{__plaintext_filename}
- 58. APT + GPG tl;dr: doesn’t work most of the time and is nearly impossible to get it working
- 59. Story Time debsigs vs dpkg-sig gpg signing deb packages is pointless XML policy documents
- 60. /etc/debsig/policies/ DDDF2F4CE732A79A/hi.pol <?xml version="1.0"?> <!DOCTYPE Policy SYSTEM "http://www.debian.org/debsig/1.0/policy.dtd"> <Policy xmlns="http://www.debian.org/debsig/1.0/"> <Origin Name="test" id="DDDF2F4CE732A79A" Description="Test package"/> <Selection> <Required Type="origin" File="debsig.gpg" id="DDDF2F4CE732A79A"/> </Selection> <Verification MinOptional="0"> <Required Type="origin" File="debsig.gpg" id="DDDF2F4CE732A79A"/> </Verification> </Policy>
- 61. oh, and, um…
- 62. Both are vulnerable to replay attacks
- 63. Neither deal with key revocation
- 64. Both are vulnerable to several GPG related attacks
- 65. what if i told you…
- 66. the CA certificate bundle you use revoked AWS’s SSL CA ?
- 67. but before i explain that, periodic reminder that trusted CA certs come from this URL
- 69. OK, anw…
- 70. bento, vagrant, kitchen opscode-centos-5.11 on or around 2015-02-23 updated the CA cert bundle
- 71. bento, vagrant, kitchen resulting in a bundle with AWS’s CA being revoked
- 72. curl.haxx.se “We in the curl project didn't anticipate anything of this. We get the data from the Mozilla project and they changed the properties. We've run the same script daily since a long time. One day the output changed to this.” - http://curl.haxx.se/mail/ archive-2014-10/0068.html
- 73. bento, vagrant, kitchen and then accessing S3 from vagrant boxes produced by bento stopped working
- 74. read more on chef’s blog: “Bento Box Update for CentOS and Fedora” https://www.chef.io/blog/2015/02/26/bento-box-update-for- centos-and-fedora/
- 75. PS debugging SSL is really difficult
- 76. BTW QUICK THING
- 77. cognitive load “cognitive load refers to the total amount of mental effort being used in the working memory"
- 78. cognitive load at some point you have to wonder: when does it become too much?
- 79. cognitive load “just read the code” is impossible because you need to read millions of lines of code
- 80. cognitive load “People changing our Chef recipes to make something work for them, but then breaking everyone else’s [stuff] is practically constant at [company] right now.” - my friend who works at [company]
- 81. this is all part of a different talk called: “the effect of capitalism on computing” But
- 82. anw
- 83. what if i told you…
- 84. you can DoS a machine with yum/apt metadata?
- 85. when apt/yum request metadata, just reply with a never ending file.
- 86. ya but i’m not an official mirror lol ??
- 87. “Debian and CentOS listed the mirror within a few hours, and Fedora listed the mirror in minutes.” - academic paper
- 88. what if i told you…
- 90. rpmUtils bug rpmUtils uses python’s find method instead of rfind when splitting version strings
- 91. rpmUtils bug version strings with two ‘-‘ in them are split on the first, not the last (incorrect)
- 92. rpmUtils bug resulting in incorrect package metadata
- 93. rpmUtils bug this is live on the official mirrors right now
- 94. rpmUtils bug i filed a bug about it, but from the looks of it, it won’t be fixed.
- 95. OK, these are all cool stories, but…
- 96. people are using infrastructure as code today though? what gives?
- 97. indeed they are, with varying levels of success and in many cases great pain what gives?
- 98. IN MY OPINION
- 99. opinion 1 we won’t be able to have truly reproducible infrastructure until we figure out better ways of building computer systems.
- 100. opinion 2 each time you move to a higher level of abstraction, you need to know more stuff. maybe cutting out some layers in between can make this more easily solvable?
- 101. opinion 3 we need to be more honest and responsible about our choices and analysis of technology.
- 102. opinion 4 huge companies making billions of dollars on top of these software systems should take the initiative to invest in making them better.
- 103. opinion 5 we haven’t found the “answer” yet. what we have is better than what we had, but we need to think bigger.