From 76aed6ef732de38d82245b3d674f70bab30221e5 Mon Sep 17 00:00:00 2001 From: Owen Jacobson Date: Fri, 3 Jul 2015 22:31:49 -0400 Subject: Fuck it, serve the files directly. --- .html/devops/_list.html | 96 ++++++++++++ .html/devops/autodeploy.html | 131 ++++++++++++++++ .html/devops/continuous-signing.html | 93 ++++++++++++ .html/devops/glassfish-and-upstart.html | 231 +++++++++++++++++++++++++++++ .html/devops/index.html | 96 ++++++++++++ .html/devops/puppet-2.7-to-3.1.html | 147 ++++++++++++++++++ .html/devops/self-daemonization-sucks.html | 162 ++++++++++++++++++++ 7 files changed, 956 insertions(+) create mode 100644 .html/devops/_list.html create mode 100644 .html/devops/autodeploy.html create mode 100644 .html/devops/continuous-signing.html create mode 100644 .html/devops/glassfish-and-upstart.html create mode 100644 .html/devops/index.html create mode 100644 .html/devops/puppet-2.7-to-3.1.html create mode 100644 .html/devops/self-daemonization-sucks.html (limited to '.html/devops') diff --git a/.html/devops/_list.html b/.html/devops/_list.html new file mode 100644 index 0000000..f4f4354 --- /dev/null +++ b/.html/devops/_list.html @@ -0,0 +1,96 @@ + + + + + The Codex » + ls /devops + + + + + + + + +
+ + + + + + + + + + + + + + +
+ + \ No newline at end of file diff --git a/.html/devops/autodeploy.html b/.html/devops/autodeploy.html new file mode 100644 index 0000000..67644a2 --- /dev/null +++ b/.html/devops/autodeploy.html @@ -0,0 +1,131 @@ + + + + + The Codex » + Notes towards automating deployment + + + + + + + + +
+ + + + + +
+

Notes towards automating deployment

+

This is mostly aimed at the hosted-apps folks; deploying packaged software for +end users requires a slightly different approach.

+

Assumptions

+
    +
  1. +

    You have one or more services to deploy. (If not, what are you doing +here?)

    +
    2. +
  2. +

    Your services are tracked in source control. (If not, go sort that out, +then come back. No, seriously, now.)

    +
    3. +
  3. +

    You will be deploying your services to one or more environments. An +environment is an abstract thing: think “production,” not +“web01.public.example.com.” (If not, where, exactly, will your service run?)

    +
    4. +
  4. +

    For each service, in each environment, there are one or more servers to +host the service. These servers are functionally identical. (If not, go pave +them and rebuild them using Puppet, Chef, CFengine, or, hell, shell scripts +and duct tape. An environment full of one-offs is the kind of hell I wouldn't +wish on my worst enemy.)

    +
    5. +
  5. +

    For each service, in each environment, there is a canonical series of steps +that produce a “deployed” system.

    +
    6. +
+
+
    +
  1. Decide what code should be deployed. (This is a version control activity.)
    2. +
  2. Get the code onto the fucking server.
    3. +
  3. Decide what configuration values should be deployed. (This is also a + version control activity, though possibly not in the same repositories as + the code.)
    4. +
  4. Get the configuration onto the fucking server.
    5. +
  5. Get the code running with the configuration.
    6. +
  6. Log to fucking syslog.
    7. +
  7. When the machine reboots, make sure the code comes back running the same + configuration.
    8. +
+
+ + + +
+
+ + +comments powered by Disqus +
+ + + + + +
+ + \ No newline at end of file diff --git a/.html/devops/continuous-signing.html b/.html/devops/continuous-signing.html new file mode 100644 index 0000000..5f61000 --- /dev/null +++ b/.html/devops/continuous-signing.html @@ -0,0 +1,93 @@ + + + + + The Codex » + Code Signing on Build Servers + + + + + + + + +
+ + + + + +
+

Code Signing on Build Servers

+

We sign things so that we can authenticate them later, but authentication is +largely a conscious function. Computers are bad at answering "is this real".

+

Major signing systems (GPG, jarsigner) require presentation of credentials at +signing time. CI servers don't generally have safe tools for this.

+
+ + + +
+
+ + +comments powered by Disqus +
+ + + + + +
+ + \ No newline at end of file diff --git a/.html/devops/glassfish-and-upstart.html b/.html/devops/glassfish-and-upstart.html new file mode 100644 index 0000000..0d03620 --- /dev/null +++ b/.html/devops/glassfish-and-upstart.html @@ -0,0 +1,231 @@ + + + + + The Codex » + Glassfish and Upstart + + + + + + + + +
+ + + + + +
+

Glassfish and Upstart

+

Warning: the article you're about to read is largely empirical. Take +everything in it in a grain of salt, and verify it yourself before putting +it into production. You have been warned.

+

The following observations apply to Glassfish 3.1.2.2. Other versions probably +act similarly, but check the docs.

+

asadmin create-service

+

Glassfish is capable of emitting SysV init scripts for the DAS, or for any +instance. These init scripts wrap asadmin start-domain and asadmin +start-local-instance. However, the scripts it emits are (justifiably) +minimalist, and it makes some very strong assumptions about the layout of your +system's rc.d trees and about your system's choice of runlevels. The minimal +init scripts avoid any integration with platform “enhancements” (such as +Redhat's /var/lock/subsys mechanism and condrestart convention, or +Debian's start-stop-daemon helpers) in the name of portability, and the +assumptions it makes about runlevels and init layout are becoming +incrementally more fragile as more distributions switch to alternate init +systems with SysV compatiblity layers.

+

Fork and expect

+

Upstart's process tracking mechanism relies on services following one of three +forking models, so that it can accurately track which children of PID 1 are +associated with which services:

+
    +
  • +

    No expect stanza: The service's “main” process is expected not to fork at + all, and to remain running. The process started by upstart is the “main” + process.

    +
    • +
  • +

    expect fork: The service is expected to call fork() or clone() once. + The process started by upstart itself is not the “main” process, but its + first child process is.

    +
    • +
  • +

    expect daemon: The service is expected to call fork() or clone() + twice. The first grandchild process of the one started by upstart itself is + the “main” process. This corresponds to classical Unix daemons, which fork + twice to properly dissociate themselves from the launching shell.

    +
    • +
+

Surprisingly, asadmin-launched Glassfish matches none of these models, and +using asadmin start-domain to launch Glassfish from Upstart is not, as far +as I can tell, possible. It's tricky to debug why, since JVM thread creation +floods strace with chaff, but I suspect that either asadmin or Glassfish +itself is forking too many times.

+

From this mailing list +thread, +though, it appears to be safe to launch Glassfish directly, using java -jar +GLASSFISH_ROOT/modules/glassfish.jar -domain DOMAIN. This fits nicely into +Upstart's non-forking expect mode, but you lose the ability to pass VM +configuration settings to Glassfish during startup. Any memory settings or +Java environment properties you want to pass to Glassfish have to be passed to +the java command manually.

+

You also lose asadmin's treatment of Glassfish's working directory. Since +Upstart can configure the working directory, this isn't a big deal.

+

SIGTERM versus asadmin stop-domain

+

Upstart always stops services by sending them a signal. While you can dictate +which signal it uses, you cannot replace signals with another mechanims. +Glassfish shuts down abruptly when it recieves SIGTERM or SIGINT, leaving +some ugly noise in the logs and potentially aborting any transactions and +requests in flight. The Glassfish developers believe this is harmless and that +the server's operation is correct, and that's probably true, but I've not +tested its effect on outward-facing requests or on in-flight operations far +enough to be comfortable with it.

+

I chose to run a “clean”(er) shutdown using asadmin stop-domain. This fits +nicely in Upstart's pre-stop step, provided you do not use Upstart's +respawn feature. Upstart will correctly notice that Glassfish has already +stopped after pre-stop finishes, but when respawn is enabled Upstart will +treat this as an unexpected termination, switch goals from stop to +respawn, and restart Glassfish.

+

(The Upstart documentation claims that respawn does not apply if the tracked +process exits during pre-stop. This may be true in newer versions of +Upstart, but the version used in Ubuntu 12.04 does restart Glassfish if it +stops during pre-stop.)

+

Yes, this does make it impossible to stop Glassfish, ever, unless you set a +respawn limit.

+

Fortunately, you don't actually want to use respawn to manage availability. +The respawn mode cripples your ability to manage the service “out of band” +by forcing Upstart to restart it as a daemon every time it stops for any +reason. This means you cannot stop a server with SIGTERM or SIGKILL; it'll +immediately start again.

+

initctl reload

+

It sends SIGHUP. This does not reload Glassfish's configuration. Deal with +it; use initctl restart or asadmin restart-domain instead. Most of +Glassfish's configuration can be changed on the fly with asadmin set or +other commands anyways, so this is not a big limitation.

+

Instances

+

Upstart supports “instances” of a service. This slots nicely into Glassfish's +ability to host multiple domains and instances on the same physical hardware. +I ended up with a generic glassfish-domain.conf Upstart configuration:

+
description "Glassfish DAS"
+console log
+
+instance $DOMAIN
+
+setuid glassfish
+setgid glassfish
+umask 0022
+chdir /opt/glassfish3
+
+exec /usr/bin/java -jar /opt/glassfish3/glassfish/modules/glassfish.jar -domain "${DOMAIN}"
+
+pre-stop exec /opt/glassfish3/bin/asadmin stop-domain "${DOMAIN}"
+
+

Combined with a per-domain wrapper:

+
description "Glassfish 'example' domain"
+console log
+
+# Consider using runlevels here.
+start on started networking
+stop on deconfiguring-networking
+
+pre-start script
+    start glassfish-domain DOMAIN=example
+end script
+
+post-stop script
+    stop glassfish-domain DOMAIN=example
+end script
+
+

Possible refinements

+
    +
  • +

    Pull system properties and VM flags from the domain's own domain.xml + correctly. It might be possible to abuse the (undocumented, unsupported, but + helpful) --_dry-run argument from asadmin start-domain for this, or it + might be necessary to parse domain.xml manually, or it may be possible to + exploit parts of Glassfish itself for this.

    +
    • +
  • +

    The asadmin cwd is actually the domain's config dir, not the Glassfish + installation root.

    +
    • +
  • +

    Something something something password files.

    +
    • +
  • +

    Syslog and logrotate integration would be useful. The configurations above + spew Glassfish's startup output and stdout to + /var/log/upstart/glassfish-domain-FOO.log, which may not be rotated by + default.

    +
    • +
+
+ + + +
+
+ + +comments powered by Disqus +
+ + + + + +
+ + \ No newline at end of file diff --git a/.html/devops/index.html b/.html/devops/index.html new file mode 100644 index 0000000..f4f4354 --- /dev/null +++ b/.html/devops/index.html @@ -0,0 +1,96 @@ + + + + + The Codex » + ls /devops + + + + + + + + +
+ + + + + + + + + + + + + + +
+ + \ No newline at end of file diff --git a/.html/devops/puppet-2.7-to-3.1.html b/.html/devops/puppet-2.7-to-3.1.html new file mode 100644 index 0000000..ca53067 --- /dev/null +++ b/.html/devops/puppet-2.7-to-3.1.html @@ -0,0 +1,147 @@ + + + + + The Codex » + Notes on upgrading Puppet from 2.7 to 3.1 + + + + + + + + +
+ + + + + +
+

Notes on upgrading Puppet from 2.7 to 3.1

+

Bad

+
    +
  • +

    As usual, you have to upgrade the puppet master first. 2.7 agents can speak + to 3.1 masters just fine, but 3.1 agents cannot speak to 2.7 masters.

    +
    • +
  • +

    I tried to upgrade the Puppet master using both puppet agent (failed when + package upgrades shut down the puppet master) and puppet apply (failed for + Ubuntu-specific reasons outlined below)

    +
    • +
  • +

    This bug.

    +
    • +
  • +

    You more or less can't upgrade Puppet using Puppet.

    +
    • +
+

Good

+
    +
  • +

    My 2.7 manifests worked perfectly under 3.1.

    +
    • +
  • +

    Puppet's CA and SSL certs survived intact and required no maintenance after + the upgrade.

    +
    • +
  • +

    The Hiera integration into class parameters works as advertised and really + does help a lot.

    +
    • +
  • +

    Once I figured out how to execute it, the upgrade was pretty smooth.

    +
    • +
  • +

    No Ruby upgrade!

    +
    • +
  • +

    Testing the upgrade in a VM sandbox meant being able to fuck up safely. + Vagrant is super awesome.

    +
    • +
+

Package Management Sucks

+

Asking Puppet to upgrade Puppet went wrong on Ubuntu because of the way Puppet +is packaged: there are three (ish) Puppet packages, and Puppet's resource +evaluation bits try to upgrade and install one package at a time. Upgrading +only “puppetmaster” upgraded “puppet-common” but not “puppet,” causing Apt to +remove “puppet”; upgrading only “puppet” similarly upgraded “puppet-copmmon” +but not “puppetmaster,” causing Apt to remove “puppetmaster.”

+

The Puppet aptitude provider (which I use instead of apt-get) for Package +resources also doesn't know how to tell aptitude what to do with config files +during upgrades. This prevented Puppet from being able to upgrade pacakges +even when running standalone (via puppet apply).

+

Finally, something about the switchover from Canonical's Puppet .debs to +Puppetlabs' .debs caused aptitude to consider all three packages “broken” +after a manual upgrade ('aptitude upgrade puppet puppetmaster'). Upgrading the +packages a second time corrected it; this is the path I eventually took with +my production puppetmaster and nodes.

+
+ + + +
+
+ + +comments powered by Disqus +
+ + + + + +
+ + \ No newline at end of file diff --git a/.html/devops/self-daemonization-sucks.html b/.html/devops/self-daemonization-sucks.html new file mode 100644 index 0000000..14e2c01 --- /dev/null +++ b/.html/devops/self-daemonization-sucks.html @@ -0,0 +1,162 @@ + + + + + The Codex » + Self-daemonizing code is awful + + + + + + + + +
+ + + + + +
+

Self-daemonizing code is awful

+

The classical UNIX approach to services is to implement them as “daemons,” +programs that run without a terminal attached and provide some service. The +key feature of a classical daemon is that, when started, it carefully +detaches itself from its initial environment and terminal, then continues +running in the background.

+

This is awful and I'm glad modern init replacements discourage it.

+

Process Tracking

+

Daemons don't exist in a vacuum. Administrators and owners need to be able to +start and stop daemons reliably, and check their status. The classic +self-daemonization approach makes this impossible.

+

Traditionally, daemons run as children of init (pid 1), even if they start +out as children of some terminal or startup process. Posix only provides +deterministic APIs for processes to manage their children and their immediate +parents; the classic daemonisation protocol hands the newly-started daemon +process off from its original parent process, which knows how to start and +stop it, to an unsuspecting init, which has no idea how this specific +daemon is special.

+

The standard workaround has daemons write their own PIDs to a file, but a +file is “dead” data: it's not automatically updated if the daemon dies, and +can linger long enough to contain the PID of some later, unrelated program. +PID file validity checks generally suffer from subtle (or, sometimes, quite +gross) race conditions.

+

Complexity

+

The actual code to correctly daemonize a process is surprisingly complex, +given the individual interfaces' relative simplicity:

+
    +
  • +

    The daemon must start its own process group

    +
    • +
  • +

    The daemon must detach from its controlling terminal

    +
    • +
  • +

    The daemon should close (and may reopen) file handles inherited from its + parent process (generally, a shell)

    +
    • +
  • +

    The daemon should ensure its working directory is predictable and + controllable

    +
    • +
  • +

    The daemon should ensure its umask is predictable and controllable

    +
    • +
  • +

    If the daemon uses privileged resources (such as low-numbered ports), it + should carefully manage its effective, real, and session UID and GIDs

    +
    • +
  • +

    Daemons must ensure that all of the above steps happen in signal-safe ways, + so that a daemon can be shut down sanely even if it's still starting up

    +
    • +
+

See this list +for a longer version. It's worse than you think.

+

All of this gets even more complicated if the daemon has its own child +processes, a pattern common to network services. Naturally, a lot of daemons +in the real world get some of these steps wrong.

+

The Future

+

Supervisord, +Foreman, +Upstart, +Launchd, +systemd, and daemontools all +encourage services not to self-daemonize by providing a sane system for +starting the daemon with the right parent process and the right environment +in the first place.

+

This is a great application of +DRY, as the daemon management +code only needs to be written once (in the daemon-managing daemon) rather +than many times over (in each individual daemon). It also makes daemon +execution more predictable, since daemons “in production” behave more like +they do when run attached to a developer's console during debugging or +development.

+
+ + + +
+
+ + +comments powered by Disqus +
+ + + + + +
+ + \ No newline at end of file -- cgit v1.2.3