Date: Wed, 9 Dec 2015 20:40:42 -0500 Subject: Remove HTML from the project. (We're no longer using Dokku.) --- .html/devops/_list.html | 98 --------- .html/devops/autodeploy.html | 131 ------------ .html/devops/continuous-signing.html | 93 --------- .html/devops/glassfish-and-upstart.html | 231 --------------------- .html/devops/index.html | 98 --------- .../notes-on-bootstrapping-grimoire-dot-ca.html | 166 --------------- .html/devops/puppet-2.7-to-3.1.html | 147 ------------- .html/devops/self-daemonization-sucks.html | 162 --------------- 8 files changed, 1126 deletions(-) delete mode 100644 .html/devops/_list.html delete mode 100644 .html/devops/autodeploy.html delete mode 100644 .html/devops/continuous-signing.html delete mode 100644 .html/devops/glassfish-and-upstart.html delete mode 100644 .html/devops/index.html delete mode 100644 .html/devops/notes-on-bootstrapping-grimoire-dot-ca.html delete mode 100644 .html/devops/puppet-2.7-to-3.1.html delete mode 100644 .html/devops/self-daemonization-sucks.html (limited to '.html/devops') diff --git a/.html/devops/_list.html b/.html/devops/_list.html deleted file mode 100644 index b121d4f..0000000 --- a/.html/devops/_list.html +++ /dev/null @@ -1,98 +0,0 @@ - - - - - The Codex » - ls /devops - - - - - - - - -

- -

- - index - -
- - devops - -
- - list - -

- - - -

`ls /devops`

- - - - -

Pages

- - - -

- - - - - - - - -

- - \ No newline at end of file diff --git a/.html/devops/autodeploy.html b/.html/devops/autodeploy.html deleted file mode 100644 index 67644a2..0000000 --- a/.html/devops/autodeploy.html +++ /dev/null @@ -1,131 +0,0 @@ - - - - - The Codex » - Notes towards automating deployment - - - - - - - - -

- -

- - index - -
- - devops - -
- - autodeploy - -

- - - -

Notes towards automating deployment

This is mostly aimed at the hosted-apps folks; deploying packaged software for -end users requires a slightly different approach.

Assumptions

-
You have one or more services to deploy. (If not, what are you doing -here?)
-
-
Your services are tracked in source control. (If not, go sort that out, -then come back. No, seriously, now.)
-
-
You will be deploying your services to one or more environments. An -environment is an abstract thing: think “production,” not -“web01.public.example.com.” (If not, where, exactly, will your service run?)
-
-
For each service, in each environment, there are one or more servers to -host the service. These servers are functionally identical. (If not, go pave -them and rebuild them using Puppet, Chef, CFengine, or, hell, shell scripts -and duct tape. An environment full of one-offs is the kind of hell I wouldn't -wish on my worst enemy.)
-
-
For each service, in each environment, there is a canonical series of steps -that produce a “deployed” system.
-

Decide what code should be deployed. (This is a version control activity.)
Get the code onto the fucking server.
Decide what configuration values should be deployed. (This is also a - version control activity, though possibly not in the same repositories as - the code.)
Get the configuration onto the fucking server.
Get the code running with the configuration.
Log to fucking syslog.
When the machine reboots, make sure the code comes back running the same - configuration.

- - - -

- - -comments powered by Disqus -

- - - - - -

- - \ No newline at end of file diff --git a/.html/devops/continuous-signing.html b/.html/devops/continuous-signing.html deleted file mode 100644 index 5f61000..0000000 --- a/.html/devops/continuous-signing.html +++ /dev/null @@ -1,93 +0,0 @@ - - - - - The Codex » - Code Signing on Build Servers - - - - - - - - -

- -

- - index - -
- - devops - -
- - continuous-signing - -

- - - -

Code Signing on Build Servers

We sign things so that we can authenticate them later, but authentication is -largely a conscious function. Computers are bad at answering "is this real".

Major signing systems (GPG, jarsigner) require presentation of credentials at -signing time. CI servers don't generally have safe tools for this.

- - - -

- - -comments powered by Disqus -

- - - - - -

- - \ No newline at end of file diff --git a/.html/devops/glassfish-and-upstart.html b/.html/devops/glassfish-and-upstart.html deleted file mode 100644 index 0d03620..0000000 --- a/.html/devops/glassfish-and-upstart.html +++ /dev/null @@ -1,231 +0,0 @@ - - - - - The Codex » - Glassfish and Upstart - - - - - - - - -

- -

- - index - -
- - devops - -
- - glassfish-and-upstart - -

- - - -

Glassfish and Upstart

Warning: the article you're about to read is largely empirical. Take -everything in it in a grain of salt, and verify it yourself before putting -it into production. You have been warned.

The following observations apply to Glassfish 3.1.2.2. Other versions probably -act similarly, but check the docs.

`asadmin create-service`

Glassfish is capable of emitting SysV init scripts for the DAS, or for any -instance. These init scripts wrap asadmin start-domain and asadmin -start-local-instance. However, the scripts it emits are (justifiably) -minimalist, and it makes some very strong assumptions about the layout of your -system's rc.d trees and about your system's choice of runlevels. The minimal -init scripts avoid any integration with platform “enhancements” (such as -Redhat's /var/lock/subsys mechanism and condrestart convention, or -Debian's start-stop-daemon helpers) in the name of portability, and the -assumptions it makes about runlevels and init layout are becoming -incrementally more fragile as more distributions switch to alternate init -systems with SysV compatiblity layers.

Fork and `expect`

Upstart's process tracking mechanism relies on services following one of three -forking models, so that it can accurately track which children of PID 1 are -associated with which services:

-
No expect stanza: The service's “main” process is expected not to fork at - all, and to remain running. The process started by upstart is the “main” - process.
-
-
expect fork: The service is expected to call fork() or clone() once. - The process started by upstart itself is not the “main” process, but its - first child process is.
-
-
expect daemon: The service is expected to call fork() or clone() - twice. The first grandchild process of the one started by upstart itself is - the “main” process. This corresponds to classical Unix daemons, which fork - twice to properly dissociate themselves from the launching shell.
-

Surprisingly, asadmin-launched Glassfish matches none of these models, and -using asadmin start-domain to launch Glassfish from Upstart is not, as far -as I can tell, possible. It's tricky to debug why, since JVM thread creation -floods strace with chaff, but I suspect that either asadmin or Glassfish -itself is forking too many times.

From this mailing list -thread, -though, it appears to be safe to launch Glassfish directly, using java -jar -GLASSFISH_ROOT/modules/glassfish.jar -domain DOMAIN. This fits nicely into -Upstart's non-forking expect mode, but you lose the ability to pass VM -configuration settings to Glassfish during startup. Any memory settings or -Java environment properties you want to pass to Glassfish have to be passed to -the java command manually.

You also lose asadmin's treatment of Glassfish's working directory. Since -Upstart can configure the working directory, this isn't a big deal.

`SIGTERM` versus `asadmin stop-domain`

Upstart always stops services by sending them a signal. While you can dictate -which signal it uses, you cannot replace signals with another mechanims. -Glassfish shuts down abruptly when it recieves SIGTERM or SIGINT, leaving -some ugly noise in the logs and potentially aborting any transactions and -requests in flight. The Glassfish developers believe this is harmless and that -the server's operation is correct, and that's probably true, but I've not -tested its effect on outward-facing requests or on in-flight operations far -enough to be comfortable with it.

I chose to run a “clean”(er) shutdown using asadmin stop-domain. This fits -nicely in Upstart's pre-stop step, provided you do not use Upstart's -respawn feature. Upstart will correctly notice that Glassfish has already -stopped after pre-stop finishes, but when respawn is enabled Upstart will -treat this as an unexpected termination, switch goals from stop to -respawn, and restart Glassfish.

(The Upstart documentation claims that respawn does not apply if the tracked -process exits during pre-stop. This may be true in newer versions of -Upstart, but the version used in Ubuntu 12.04 does restart Glassfish if it -stops during pre-stop.)

Yes, this does make it impossible to stop Glassfish, ever, unless you set a -respawn limit.

Fortunately, you don't actually want to use respawn to manage availability. -The respawn mode cripples your ability to manage the service “out of band” -by forcing Upstart to restart it as a daemon every time it stops for any -reason. This means you cannot stop a server with SIGTERM or SIGKILL; it'll -immediately start again.

`initctl reload`

It sends SIGHUP. This does not reload Glassfish's configuration. Deal with -it; use initctl restart or asadmin restart-domain instead. Most of -Glassfish's configuration can be changed on the fly with asadmin set or -other commands anyways, so this is not a big limitation.

Instances

Upstart supports “instances” of a service. This slots nicely into Glassfish's -ability to host multiple domains and instances on the same physical hardware. -I ended up with a generic glassfish-domain.conf Upstart configuration:

description "Glassfish DAS"
-console log
-
-instance $DOMAIN
-
-setuid glassfish
-setgid glassfish
-umask 0022
-chdir /opt/glassfish3
-
-exec /usr/bin/java -jar /opt/glassfish3/glassfish/modules/glassfish.jar -domain "${DOMAIN}"
-
-pre-stop exec /opt/glassfish3/bin/asadmin stop-domain "${DOMAIN}"
-

Combined with a per-domain wrapper:

description "Glassfish 'example' domain"
-console log
-
-# Consider using runlevels here.
-start on started networking
-stop on deconfiguring-networking
-
-pre-start script
-    start glassfish-domain DOMAIN=example
-end script
-
-post-stop script
-    stop glassfish-domain DOMAIN=example
-end script
-

- -

-
Pull system properties and VM flags from the domain's own domain.xml - correctly. It might be possible to abuse the (undocumented, unsupported, but - helpful) --_dry-run argument from asadmin start-domain for this, or it - might be necessary to parse domain.xml manually, or it may be possible to - exploit parts of Glassfish itself for this.
-
-
The asadmin cwd is actually the domain's config dir, not the Glassfish - installation root.
-
-
Something something something password files.
-
-
Syslog and logrotate integration would be useful. The configurations above - spew Glassfish's startup output and stdout to - /var/log/upstart/glassfish-domain-FOO.log, which may not be rotated by - default.
-

- - - -

- - -comments powered by Disqus -

- - - - - -

- - \ No newline at end of file diff --git a/.html/devops/index.html b/.html/devops/index.html deleted file mode 100644 index b121d4f..0000000 --- a/.html/devops/index.html +++ /dev/null @@ -1,98 +0,0 @@ - - - - - The Codex » - ls /devops - - - - - - - - -

- -

- - index - -
- - devops - -
- - list - -

- - - -

`ls /devops`

- - - - -

Pages

- - - -

- - - - - - - - -

- - \ No newline at end of file diff --git a/.html/devops/notes-on-bootstrapping-grimoire-dot-ca.html b/.html/devops/notes-on-bootstrapping-grimoire-dot-ca.html deleted file mode 100644 index e5a1b47..0000000 --- a/.html/devops/notes-on-bootstrapping-grimoire-dot-ca.html +++ /dev/null @@ -1,166 +0,0 @@ - - - - - The Codex » - Notes on Bootstrapping This Host - - - - - - - - -

- -

- - index - -
- - devops - -
- - notes-on-bootstrapping-grimoire-dot-ca - -

- - - -

Notes on Bootstrapping This Host

Presented without comment:

-
Package updates:
-
```
apt-get update
-apt-get upgrade
-
```
-
-
Install Git:
-
```
apt-get install git
-
```
-

Set hostname:

echo 'grimoire' > /etc/hostname
-sed -i -e $'s,ubuntu,grimoire.ca\tgrimoire,' /etc/hosts
-poweroff
-

To verify:

hostname -f # => grimoire.ca
-hostname    # => grimoire
-

Add owen user:

adduser owen
-adduser owen sudo
-

To verify:

id owen # => uid=1000(owen) gid=1000(owen) groups=1000(owen),27(sudo)
-

Install Puppetlabs Repos:

wget https://apt.puppetlabs.com/puppetlabs-release-pc1-trusty.deb
-dpkg -i puppetlabs-release-pc1-trusty.deb
-apt-get update
-

Install Puppet server:

apt-get install puppetserver
-sed -i \
-    -e '/^JAVA_ARGS=/ s,2g,512m,g' \
-    -e '/^JAVA_ARGS=/ s, -XX:MaxPermSize=256m,,' \
-    /etc/default/puppetserver
-service puppetserver start
-

Test Puppet agent:

/opt/puppetlabs/bin/puppet agent --test --server grimoire.ca
-

This should output the following:

Info: Retrieving pluginfacts
-Info: Retrieving plugin
-Info: Caching catalog for grimoire.ca
-Info: Applying configuration version '1446415926'
-Info: Creating state file /opt/puppetlabs/puppet/cache/state/state.yaml
-Notice: Applied catalog in 0.01 seconds
-

Install environment:

git init --bare /root/puppet.git
-# From workstation, `git push root@grimoire.ca:puppet.git master` to populate the repo
-rm -rf /etc/puppetlabs/code/environments/production
-git clone /root/puppet.git /etc/puppetlabs/code/environments/production
-

Bootstrap puppet:

/opt/puppetlabs/bin/puppet agent --test --server grimoire.ca
-

- - - -

- - -comments powered by Disqus -

- - - - - -

- - \ No newline at end of file diff --git a/.html/devops/puppet-2.7-to-3.1.html b/.html/devops/puppet-2.7-to-3.1.html deleted file mode 100644 index ca53067..0000000 --- a/.html/devops/puppet-2.7-to-3.1.html +++ /dev/null @@ -1,147 +0,0 @@ - - - - - The Codex » - Notes on upgrading Puppet from 2.7 to 3.1 - - - - - - - - -

- -

- - index - -
- - devops - -
- - puppet-2.7-to-3.1 - -

- - - -

Notes on upgrading Puppet from 2.7 to 3.1

Bad

-
As usual, you have to upgrade the puppet master first. 2.7 agents can speak - to 3.1 masters just fine, but 3.1 agents cannot speak to 2.7 masters.
-
-
I tried to upgrade the Puppet master using both puppet agent (failed when - package upgrades shut down the puppet master) and puppet apply (failed for - Ubuntu-specific reasons outlined below)
-
-
This bug.
-
-
You more or less can't upgrade Puppet using Puppet.
-

Good

-
My 2.7 manifests worked perfectly under 3.1.
-
-
Puppet's CA and SSL certs survived intact and required no maintenance after - the upgrade.
-
-
The Hiera integration into class parameters works as advertised and really - does help a lot.
-
-
Once I figured out how to execute it, the upgrade was pretty smooth.
-
-
No Ruby upgrade!
-
-
Testing the upgrade in a VM sandbox meant being able to fuck up safely. - Vagrant is super awesome.
-

Package Management Sucks

Asking Puppet to upgrade Puppet went wrong on Ubuntu because of the way Puppet -is packaged: there are three (ish) Puppet packages, and Puppet's resource -evaluation bits try to upgrade and install one package at a time. Upgrading -only “puppetmaster” upgraded “puppet-common” but not “puppet,” causing Apt to -remove “puppet”; upgrading only “puppet” similarly upgraded “puppet-copmmon” -but not “puppetmaster,” causing Apt to remove “puppetmaster.”

The Puppet aptitude provider (which I use instead of apt-get) for Package -resources also doesn't know how to tell aptitude what to do with config files -during upgrades. This prevented Puppet from being able to upgrade pacakges -even when running standalone (via puppet apply).

Finally, something about the switchover from Canonical's Puppet .debs to -Puppetlabs' .debs caused aptitude to consider all three packages “broken” -after a manual upgrade ('aptitude upgrade puppet puppetmaster'). Upgrading the -packages a second time corrected it; this is the path I eventually took with -my production puppetmaster and nodes.

- - - -

- - -comments powered by Disqus -

- - - - - -

- - \ No newline at end of file diff --git a/.html/devops/self-daemonization-sucks.html b/.html/devops/self-daemonization-sucks.html deleted file mode 100644 index 14e2c01..0000000 --- a/.html/devops/self-daemonization-sucks.html +++ /dev/null @@ -1,162 +0,0 @@ - - - - - The Codex » - Self-daemonizing code is awful - - - - - - - - -

- -

- - index - -
- - devops - -
- - self-daemonization-sucks - -

- - - -

Self-daemonizing code is awful

The classical UNIX approach to services is to implement them as “daemons,” -programs that run without a terminal attached and provide some service. The -key feature of a classical daemon is that, when started, it carefully -detaches itself from its initial environment and terminal, then continues -running in the background.

This is awful and I'm glad modern init replacements discourage it.

Process Tracking

Daemons don't exist in a vacuum. Administrators and owners need to be able to -start and stop daemons reliably, and check their status. The classic -self-daemonization approach makes this impossible.

Traditionally, daemons run as children of init (pid 1), even if they start -out as children of some terminal or startup process. Posix only provides -deterministic APIs for processes to manage their children and their immediate -parents; the classic daemonisation protocol hands the newly-started daemon -process off from its original parent process, which knows how to start and -stop it, to an unsuspecting init, which has no idea how this specific -daemon is special.

The standard workaround has daemons write their own PIDs to a file, but a -file is “dead” data: it's not automatically updated if the daemon dies, and -can linger long enough to contain the PID of some later, unrelated program. -PID file validity checks generally suffer from subtle (or, sometimes, quite -gross) race conditions.

Complexity

The actual code to correctly daemonize a process is surprisingly complex, -given the individual interfaces' relative simplicity:

-
The daemon must start its own process group
-
-
The daemon must detach from its controlling terminal
-
-
The daemon should close (and may reopen) file handles inherited from its - parent process (generally, a shell)
-
-
The daemon should ensure its working directory is predictable and - controllable
-
-
The daemon should ensure its umask is predictable and controllable
-
-
If the daemon uses privileged resources (such as low-numbered ports), it - should carefully manage its effective, real, and session UID and GIDs
-
-
Daemons must ensure that all of the above steps happen in signal-safe ways, - so that a daemon can be shut down sanely even if it's still starting up
-

See this list -for a longer version. It's worse than you think.

All of this gets even more complicated if the daemon has its own child -processes, a pattern common to network services. Naturally, a lot of daemons -in the real world get some of these steps wrong.

The Future

Supervisord, -Foreman, -Upstart, -Launchd, -systemd, and daemontools all -encourage services not to self-daemonize by providing a sane system for -starting the daemon with the right parent process and the right environment -in the first place.

This is a great application of -DRY, as the daemon management -code only needs to be written once (in the daemon-managing daemon) rather -than many times over (in each individual daemon). It also makes daemon -execution more predictable, since daemons “in production” behave more like -they do when run attached to a developer's console during debugging or -development.

- - - -

- - -comments powered by Disqus -

- - - - - -

- - \ No newline at end of file -- cgit v1.2.3

ls /devops

Pages

Notes towards automating deployment

Assumptions

Code Signing on Build Servers

Glassfish and Upstart

asadmin create-service

Fork and expect

SIGTERM versus asadmin stop-domain

initctl reload

Instances

Possible refinements

ls /devops

Pages

Notes on Bootstrapping This Host

Notes on upgrading Puppet from 2.7 to 3.1

Bad

Good

Package Management Sucks

Self-daemonizing code is awful

Process Tracking

Complexity

The Future

`ls /devops`

`asadmin create-service`

Fork and `expect`

`SIGTERM` versus `asadmin stop-domain`

`initctl reload`

`ls /devops`