From 0d6f58c54a7af6c8b4e6cd98663eb36ec4e3accc Mon Sep 17 00:00:00 2001 From: Owen Jacobson Date: Tue, 28 Jan 2020 20:49:17 -0500 Subject: Editorial pass & migration to mkdocs. There's a lot in grimoire.ca that I either no longer stand behind or feel pretty weird about having out there. --- wiki/12factor/3-config.md | 22 - wiki/12factor/7-port-binding.md | 31 - wiki/12factor/index.md | 19 - wiki/authnz/users-rolegraph-privs.md | 110 --- wiki/chat/notes.md | 39 -- wiki/cool-urls-can-change.md | 66 -- wiki/dev/buffers.md | 99 --- wiki/dev/builds.md | 194 ------ wiki/dev/comments.md | 8 - wiki/dev/commit-messages.md | 70 -- wiki/dev/configuring-browser-apps.md | 108 --- wiki/dev/debugger-101.md | 86 --- wiki/dev/entry-points.md | 56 -- wiki/dev/gnu-collective-action-license.md | 51 -- wiki/dev/go.md | 112 ---- wiki/dev/liquibase.md | 77 --- wiki/dev/merging-structural-changes.md | 85 --- wiki/dev/on-rights.md | 21 - wiki/dev/papers.md | 36 - wiki/dev/rich-shared-models.md | 102 --- wiki/dev/shutdown-hooks.md | 29 - .../stop-building-synchronous-web-containers.md | 41 -- wiki/dev/trackers-from-first-principles.md | 219 ------ wiki/dev/twigs.md | 24 - wiki/dev/webapp-versions.md | 27 - wiki/dev/webapps.md | 5 - wiki/dev/webpack.md | 236 ------- wiki/dev/whats-wrong-with-jenkins.md | 108 --- wiki/dev/why-scm.md | 73 -- wiki/devops/autodeploy.md | 38 -- wiki/devops/continuous-signing.md | 7 - wiki/devops/glassfish-and-upstart.md | 153 ----- .../notes-on-bootstrapping-grimoire-dot-ca.md | 71 -- wiki/devops/puppet-2.7-to-3.1.md | 51 -- wiki/devops/self-daemonization-sucks.md | 78 --- wiki/email.md | 19 - wiki/ethics/lg-smart-tv.md | 98 --- wiki/ethics/linkedin-intro.md | 187 ------ wiki/ethics/musings.md | 76 --- wiki/games/dark-souls.md | 57 -- wiki/git/config.md | 58 -- wiki/git/detached-sigs.md | 298 --------- wiki/git/integrate.md | 41 -- wiki/git/pull-request-workflow.md | 101 --- wiki/git/scratch.md | 55 -- wiki/git/stop-using-git-pull-to-deploy.md | 98 --- wiki/git/survival.md | 81 --- wiki/git/theory-and-practice/index.md | 42 -- wiki/git/theory-and-practice/objects.md | 125 ---- wiki/git/theory-and-practice/refs-and-names.md | 94 --- wiki/github-nomic/notes.md | 118 ---- wiki/github-nomic/rules.md | 180 ----- wiki/gossamer/coda.md | 19 - wiki/gossamer/index.md | 435 ------------ wiki/gossamer/mistakes.md | 81 --- wiki/gpg/cool.md | 67 -- wiki/gpg/keys.md | 727 -------------------- wiki/gpg/terrible.md | 139 ---- wiki/hire-me.md | 110 --- wiki/if/messages-and-announcements.md | 78 --- wiki/if/narrative-in-muds.md | 115 ---- wiki/java/a-new-kind-of.md | 137 ---- wiki/java/install/centos.md | 57 -- wiki/java/install/index.md | 11 - wiki/java/install/ubuntu.md | 84 --- wiki/java/kwargs.md | 152 ----- wiki/java/stop-using-class-dot-forname.md | 69 -- wiki/muds/tinyfugue-on-yosemite.md | 21 - wiki/mysql/broken-xa.md | 29 - wiki/mysql/choose-something-else.md | 736 --------------------- wiki/packaging-ideas.md | 20 - wiki/people/co-op-social-media.md | 51 -- wiki/people/community-norms.md | 23 - wiki/people/public-compensation.md | 101 --- wiki/people/rape-culture-and-men.md | 39 -- wiki/people/rincewind.md | 32 - wiki/people/why-twitter.md | 37 -- wiki/toronto/pan-am-carding-lab.md | 49 -- 78 files changed, 7699 deletions(-) delete mode 100644 wiki/12factor/3-config.md delete mode 100644 wiki/12factor/7-port-binding.md delete mode 100644 wiki/12factor/index.md delete mode 100644 wiki/authnz/users-rolegraph-privs.md delete mode 100644 wiki/chat/notes.md delete mode 100644 wiki/cool-urls-can-change.md delete mode 100644 wiki/dev/buffers.md delete mode 100644 wiki/dev/builds.md delete mode 100644 wiki/dev/comments.md delete mode 100644 wiki/dev/commit-messages.md delete mode 100644 wiki/dev/configuring-browser-apps.md delete mode 100644 wiki/dev/debugger-101.md delete mode 100644 wiki/dev/entry-points.md delete mode 100644 wiki/dev/gnu-collective-action-license.md delete mode 100644 wiki/dev/go.md delete mode 100644 wiki/dev/liquibase.md delete mode 100644 wiki/dev/merging-structural-changes.md delete mode 100644 wiki/dev/on-rights.md delete mode 100644 wiki/dev/papers.md delete mode 100644 wiki/dev/rich-shared-models.md delete mode 100644 wiki/dev/shutdown-hooks.md delete mode 100644 wiki/dev/stop-building-synchronous-web-containers.md delete mode 100644 wiki/dev/trackers-from-first-principles.md delete mode 100644 wiki/dev/twigs.md delete mode 100644 wiki/dev/webapp-versions.md delete mode 100644 wiki/dev/webapps.md delete mode 100644 wiki/dev/webpack.md delete mode 100644 wiki/dev/whats-wrong-with-jenkins.md delete mode 100644 wiki/dev/why-scm.md delete mode 100644 wiki/devops/autodeploy.md delete mode 100644 wiki/devops/continuous-signing.md delete mode 100644 wiki/devops/glassfish-and-upstart.md delete mode 100644 wiki/devops/notes-on-bootstrapping-grimoire-dot-ca.md delete mode 100644 wiki/devops/puppet-2.7-to-3.1.md delete mode 100644 wiki/devops/self-daemonization-sucks.md delete mode 100644 wiki/email.md delete mode 100644 wiki/ethics/lg-smart-tv.md delete mode 100644 wiki/ethics/linkedin-intro.md delete mode 100644 wiki/ethics/musings.md delete mode 100644 wiki/games/dark-souls.md delete mode 100644 wiki/git/config.md delete mode 100644 wiki/git/detached-sigs.md delete mode 100644 wiki/git/integrate.md delete mode 100644 wiki/git/pull-request-workflow.md delete mode 100644 wiki/git/scratch.md delete mode 100644 wiki/git/stop-using-git-pull-to-deploy.md delete mode 100644 wiki/git/survival.md delete mode 100644 wiki/git/theory-and-practice/index.md delete mode 100644 wiki/git/theory-and-practice/objects.md delete mode 100644 wiki/git/theory-and-practice/refs-and-names.md delete mode 100644 wiki/github-nomic/notes.md delete mode 100644 wiki/github-nomic/rules.md delete mode 100644 wiki/gossamer/coda.md delete mode 100644 wiki/gossamer/index.md delete mode 100644 wiki/gossamer/mistakes.md delete mode 100644 wiki/gpg/cool.md delete mode 100644 wiki/gpg/keys.md delete mode 100644 wiki/gpg/terrible.md delete mode 100644 wiki/hire-me.md delete mode 100644 wiki/if/messages-and-announcements.md delete mode 100644 wiki/if/narrative-in-muds.md delete mode 100644 wiki/java/a-new-kind-of.md delete mode 100644 wiki/java/install/centos.md delete mode 100644 wiki/java/install/index.md delete mode 100644 wiki/java/install/ubuntu.md delete mode 100644 wiki/java/kwargs.md delete mode 100644 wiki/java/stop-using-class-dot-forname.md delete mode 100644 wiki/muds/tinyfugue-on-yosemite.md delete mode 100644 wiki/mysql/broken-xa.md delete mode 100644 wiki/mysql/choose-something-else.md delete mode 100644 wiki/packaging-ideas.md delete mode 100644 wiki/people/co-op-social-media.md delete mode 100644 wiki/people/community-norms.md delete mode 100644 wiki/people/public-compensation.md delete mode 100644 wiki/people/rape-culture-and-men.md delete mode 100644 wiki/people/rincewind.md delete mode 100644 wiki/people/why-twitter.md delete mode 100644 wiki/toronto/pan-am-carding-lab.md (limited to 'wiki') diff --git a/wiki/12factor/3-config.md b/wiki/12factor/3-config.md deleted file mode 100644 index 5d6c6c6..0000000 --- a/wiki/12factor/3-config.md +++ /dev/null @@ -1,22 +0,0 @@ -# Factor 3: Config - -[This section](http://www.12factor.net/config) advises using environment -variables for everything. - -> [Owen J](https://twitter.com/derspiny): I think I disagree with -> 12factor's conclusions on config even though I agree with the premises -> and rationale in general -> -> [Owen J](https://twitter.com/derspiny): environment variables -> are neither exceptionally portable, exceptionally standard, nor -> exceptionally easy to manage -> -> [Owen J](https://twitter.com/derspiny): and therefore should not be -> the exceptional configuration mechanism :) -> -> [Kit L](https://twitter.com/wlonk): that's exactly the critique i have - -Frustratingly, the config section doesn't provide any guidance on sensible -ways to _manage_ environment variables. In any real-world deployment, they're -going to have to be stored somewhere; where's appropriate? `.bash_profile`? -`httpd.con` as `SetEnv` directives? Per-release `rc` files? `/etc/init.d`? diff --git a/wiki/12factor/7-port-binding.md b/wiki/12factor/7-port-binding.md deleted file mode 100644 index a756496..0000000 --- a/wiki/12factor/7-port-binding.md +++ /dev/null @@ -1,31 +0,0 @@ -# Factor 7: Port Binding - -[This](http://www.12factor.net/port-binding) is the exact point where the -Heroku-specific features of the approach overwhelm the general features. - -Factor 7 is over-specific: - -* It presupposes the existence of a front-end routing layer, without providing - any insight into how to deploy, configure, provision, or manage one. - -* It demands HTTP (by name) rather than a more flexible “any well-standardized - protocol,” without explaining why. (Web apps can have non-HTTP internal - components.) - -* It dismisses the value of “pre-existing” container ecosystems that don't - work the way Heroku does. Have a giant, well-managed - [Glassfish](http://glassfish.org) cluster that you deploy components to? TOO - BAD, not Heroku-like enough for these guys even though many aspects run - along similar philosophical lines. - -* It dismisses the value of unix-as-a-container. Unix domain sockets with - controlled permissions? Psh, let's go through the network stack instead. - SysV IPC? (Yeah, I know.) Network. Pipes? Network. There's an implicit - exception for “intra-process” communication, but it's never really - identified or reasoned about. - -* Have you _seen_ the kinds of process control interfaces developers invent, - when left to their own devices? Signals and PID files are well-established - conventions, and smart, competent people still fuck those up all the time. - Command-line arguments are another frequent case of NIH stupidity. Do you - really want every app to have its own startup API? diff --git a/wiki/12factor/index.md b/wiki/12factor/index.md deleted file mode 100644 index 6e75732..0000000 --- a/wiki/12factor/index.md +++ /dev/null @@ -1,19 +0,0 @@ -# 12-Factor Apps - -Some folks over at [Heroku](http://heroku.com/) wrote up their perceived best -practices for building “software as a service”-style applications and called -it [The Twelve-Factor App](http://www.12factor.net). It's a good read, and has -lots of good advice in it. - -I have a few thoughts on it. - ------ - -* [III. Config](3-config) -* [VII. Port Binding](7-port-binding) - ------ - -At some point around sections 6 or 7, the goodness of the advice is overtaken -by the “be more like Heroku specifically”-ness of the advice, to the detriment -of their point. diff --git a/wiki/authnz/users-rolegraph-privs.md b/wiki/authnz/users-rolegraph-privs.md deleted file mode 100644 index fdbf52d..0000000 --- a/wiki/authnz/users-rolegraph-privs.md +++ /dev/null @@ -1,110 +0,0 @@ -# A Users, Roles & Privileges Scheme Using Graphs - -The basic elements: - -* Every agent that can interact with a system is represented by a **user**. -* Every capability the system has is authorized by a distinct **privilege**. -* Each user has a list of zero or more **roles**. - * Roles can **imply** further roles. This relationship is transitive: if - role A implies role B, then a member of role A is a member of role B; if - role B also implies role C, then a member of role A is also a member of - role C. It helps if the resulting role graph is acyclic, but it's not - necessary. - * Roles can **grant** privileges. - -A user's privileges are the union of the privileges granted by the transitive -closure of their roles. - -## In SQL - - create table "user" ( - username varchar - primary key - -- credentials &c - ); - - create table role ( - name varchar - primary key - ); - - create table role_member ( - role varchar - not null - references role, - member varchar - not null - references "user", - primary key (role, member) - ); - - create table role_implies ( - role varchar - not null - references role, - implied_role varchar - not null - ); - - create table privilege ( - privilege varchar - primary key - ); - - create table role_grants ( - role varchar - not null - references role, - privilege varchar - not null - references privilege, - primary key (role, privilege) - ); - -If your database supports recursive CTEs, querying this isn't awful, since we -can have the database do all the graph-walking along roles: - - with recursive user_roles (role) AS ( - select - role - from - role_member - where - member = 'SOME USERNAME' - union - select - implied_role as role - from - user_roles - join role_implies on - user_roles.role = role_implies.role - ) - select distinct - role_grants.privilege as privilege - from - user_roles - join role_grants on - user_roles.role = role_grants.role - order by privilege; - -If not, get a better database. Recursive graph walking with network round -trips at each step is stupid and you shouldn't do it. - -Realistic uses should have fairly simple graphs: elemental privileges are -grouped into abstract roles, which are in turn grouped into meaningful roles -(by department, for example), which are in turn granted to users. In -PostgreSQL, the above schema handles ~10k privileges and ~10k roles with -randomly-generated graph relationships in around 100ms on my laptop, which is -pretty slow but not intolerable. Perverse cases (interconnected total -subgraphs, deeply-nested linear graphs) can take absurd time but do not -reflect any likely permissions scheme. - -## What Sucks - -* Graph theory in my authorization system? It's more likely than you think. -* There's no notion of revoking a privilege. If you have a privilege by any - path through your roles, then it cannot be revoked except by removing all of - the paths that lead back to that privilege. -* Not every system has an efficient way to compute these graphs. - * PostgreSQL, as given above, has a hard time with unrealistically-deep - nested roles. diff --git a/wiki/chat/notes.md b/wiki/chat/notes.md deleted file mode 100644 index 84f60f6..0000000 --- a/wiki/chat/notes.md +++ /dev/null @@ -1,39 +0,0 @@ -# Notes towards a Chat Service - -Now: - -* Chat tools divide discussion by "channel"/"room" -* A channel is an undifferentiated sequence of remarks. -* Social dynamics in small channels: don't interrupt the current channel discussion even if you have another discussion to raise that would be within the channel's purpose. - * Conversations are bimodal: short bursts of generally-interesting remarks, or long chains of interrun responses. Not much middle ground. (Think meme channels vs discussion channels.) - * Small groups + robots: the robots interrupt things anyways, because they're robots. -* Social dynamics in large channels: it's moving too fast to really track, unless it's the _only_ thing you're doing. - -Slack specifically: - -* Per-social-circle UI modality makes it awkward to engage with multiple discussions at a time unless they all happen in the same place. -* Universally poor respect for consent. -* Pricing/business model issues: - -Instead: - -* A channel is a group of distinct discussions, plus a jumping-off point for new discussions. -* A user viewing a channel sees an overview of the ongoing discussions (maintained automatically or semi-automatically) along with lists of their active participants, and any initial remarks that could lead to a new discussion. -* A user can join an ongoing discussion and see the remarks to date, or duck out of it to see the summary again. -* A user can leave an ongoing discussion to indicate that they no longer expect to participate and may not respond to things said. -* Conversations "age out" of channels after they fall silent. -* Aged out conversations are still visible in archives and in the participants' clients, and necroposting brings them back. - -* New remarks to the channel appear as "prompts." -* Responding to a prompt creates a conversation. -* Prompts age out (quickly) if not responded to. - -![A channel overview. On the left is a list of channels and groups. On the right, dominating the screen, is an area showing two converation previews, with avatar lists and a response button. At the bottom is a callout for John Doe, showing an un-responded-to prompt. Below that is a text field with the legend "Say anything!"](/media/chat/notes/channel-overview.png) - -![A conversation overview. On the left is a list of channels and groups. On the right, dominating the screen, is an area showing a single conversation between two participants as a list of chat lines marked by speaker. At the bottom is a callout for John Doe, showing an un-responded-to prompt. Below that is a text field with the legend "Say anything!"](/media/chat/notes/conversation.png) - -Why: - -* Allow multiple concurrent discussions within the same nominal channel with minimal crosstalk/confusion. -* Insulate conversations from accidental interruptions, while making it easy to intentionally participate. -* Closer model to rooms full of people. diff --git a/wiki/cool-urls-can-change.md b/wiki/cool-urls-can-change.md deleted file mode 100644 index b0c489b..0000000 --- a/wiki/cool-urls-can-change.md +++ /dev/null @@ -1,66 +0,0 @@ -# Cool URLs Do Change (Sometimes) - -Required reading: [Cool URLs don't -change](http://www.w3.org/Provider/Style/URI.html). - -When I wrote [Nobody Cares About Your -Build](http://codex.grimoire.ca/2008/09/24/nobody-cares-about-your-build/), I -set up a dedicated publishing platform - Wordpress, as it happens - to host -it, and as part of that process I put some real thought into the choice of -“permalink” schemes to use. I opted to use a “dated” scheme, baking the -publication date of each article into its name - into its URL - for all -eternity. I'm a big believer in the idea that a URL should be a long-term name -for the appropriate bit of data or content, and every part of a dated scheme -“made sense” at the time. - -This turned out to be a mistake. - -The web is not, much, like print media. Something published may be amended; -you don't even have to publish errata or a correction, since you can correct -the original mistake “seamlessly.” This has its good and its -[bad](http://en.wikipedia.org/wiki/Memory_hole) parts, but with judicious use -and [a public history](https://github.com/ojacobson/grimoiredotca), amendment -is more of a win than a loss. However, this plays havoc with the idea of a -“publication” date, even for data that takes the form of an article: is the -publication date the date it was first made public, the date of its most -recent edit, or some other date? - -Because the name - the URL - of an article was set when I first published it, -the date in the name had to be its initial publication date. _This has -actually stopped me from making useful amendments to old articles_ because the -effort of writing a full, free-standing followup article is more than I'm -willing to commit to. Had I left the date out of the URLs, I'd feel more free -to judiciously amend articles in place and include, in the content, a short -amendment summary. - -The W3C's informal suggestions on the subject state that “After the creation -date, putting any information in the name is asking for trouble one way or -another.” I'm starting to believe that this doesn't go far enough: _every_ -part of a URL must have some semantic justification for being there, dates -included: - -1. *Each part must be meaningful*. While - `http://example.com/WW91IGp1c3QgbG9zdCB0aGUgZ2FtZQ==` is fairly easy to - render stable, the meaningless blob renders the name immemorable. - -2. *Each part must be stable*. This is where I screwed up worst: I did not - anticipate that the “date” of an article could be a fluid thing. It's - tempting to privilege the first date, and it's not an unreasonable - solution, but it didn't fit how I wanted to address the contents of - articles. - -Running a web server gives you one namespace to play with. Use it wisely. - -## Ok, But I've Already Got These URLs - -Thankfully, there's a way out - for _some_ URLs. URLs inherently name -resources _accessed using some protocol_, and some protocols provide support -for resources that are, themselves, references to other URLs. HTTP is a good -example, providing a fairly rich set of responses that all, fundamentally, -tell a client to check a second URL for the content relevent to a given URL. -In protocols like this, you can easily replace the content of a URL with a -reference to its new, “better” URL rather than abandoning it entirely. - -Names can evolve organically as the humans that issue them grow a better -understanding of the problem, and don't always have to be locked in stone from -the moment they're first used. diff --git a/wiki/dev/buffers.md b/wiki/dev/buffers.md deleted file mode 100644 index 62bcad6..0000000 --- a/wiki/dev/buffers.md +++ /dev/null @@ -1,99 +0,0 @@ -# Observations on Buffering - -None of the following is particularly novel, but the reminder has been useful: - -* All buffers exist in one of two states: full (writes outpace reads), or empty - (reads outpace writes). There are no other stable configurations. - -* Throughput on an empty buffer is dominated by the write rate. Throughput on a - full buffer is dominated by the read rate. - -* A full buffer imposes a latency penalty equal to its size in bits, divided by - the read rate in bits per second. An empty buffer imposes (approximately) no - latency penalty. - -The previous three points suggest that **traffic buffers should be measured in -seconds, not in bytes**, and managed accordingly. Less obviously, buffer -management needs to be considerably more sophisticated than the usual "grow -buffer when full, up to some predefined maximum size." - -Point one also implies a rule that I see honoured more in ignorance than in -awareness: **you can't make a full buffer less full by making it bigger**. Size -is not a factor in buffer fullness, only in buffer latency, so adjusting the -size in response to capacity pressure is worse than useless. - -There are only three ways to make a full buffer less full: - -1. Increase the rate at which data exits the buffer. - -2. Slow the rate at which data enters the buffer. - -3. Evict some data from the buffer. - -In actual practice, most full buffers are upstream of some process that's -already going as fast as it can, either because of other design limits or -because of physics. A buffer ahead of disk writing can't drain faster than the -disk can accept data, for example. That leaves options two and three. - -Slowing the rate of arrival usually implies some variety of _back-pressure_ on -the source of the data, to allow upstream processes to match rates with -downstream processes. Over-large buffers delay this process by hiding -back-pressure, and buffer growth will make this problem worse. Often, -back-pressure can happen automatically: failing to read from a socket, for -example, will cause the underlying TCP stack to apply back-pressure to the peer -writing to the socket by delaying TCP-level message acknowledgement. Too often, -I've seen code attempt to suppress these natural forms of back-pressure without -replacing them with anything, leading to systems that fail by surprise when -some other resource – usually memory – runs out. - -Eviction relies on the surrounding environment, and must be part of the -protocol design. Surprisingly, most modern application protocols get very -unhappy when you throw their data away: the network age has not, sadly, brought -about protocols and formats particularly well-designed for distribution. - -If neither back-pressure nor eviction are available, the remaining option is to -fail: either to start dropping data unpredictably, or to cease processing data -entirely as a result of some resource or another running out, or to induce so -much latency that the data is useless by the time it arrives. - ------ - -Some uncategorized thoughts: - -* Some buffers exist to trade latency against the overhead of coordination. A - small buffer in this role will impose more coordination overhead; a large - buffer will impose more latency. - - * These buffers appear where data transits between heterogenous system: for - example, buffering reads from the network for writes to disk. - - * Mismanaged buffers in this role will tend to cause the system to spend - an inordinate proportion of latency and throughput negotiating buffer - sizes and message readiness. - - * A coordination buffer is most useful when _empty_; in the ideal case, the - buffer is large enough to absorb one message's worth of data from the - source, then pass it along to the sink as quickly as possible. - -* Some buffers exist to trade latency against jitter. A small buffer in this - role will expose more jitter to the upstream process. A large buffer in this - role will impose more latency. - - * These tend to appear in _homogenous_ systems with differing throughputs, - or as a consequence of some other design choice. Store-and-forward - switching in networks, for example, implies that switches must buffer at - least one full frame of network data. - - * Mis-managed buffers in this role will _amplify_ rather than smoothing out - jitter. Apparent throughput will be high until the buffer fills, then - change abruptly when full. Upstream processes are likely to throttle - down, causing them to under-deliver if the buffer drains, pushing the - system back to a high-throughput mode. [This problem gets worse the - more buffers are present in a system](http://www.bufferbloat.net). - - * An anti-jitter buffer is most useful when _full_; in exchange for a - latency penalty, sudden changes in throughput will be absorbed by data - in the buffer rather than propagating through to the source or sink. - -* Multimedia people understand this stuff at a deep level. Listen to them when - designing buffers for other applications. diff --git a/wiki/dev/builds.md b/wiki/dev/builds.md deleted file mode 100644 index abe3d19..0000000 --- a/wiki/dev/builds.md +++ /dev/null @@ -1,194 +0,0 @@ -# Nobody Cares About Your Build - -Every software system, from simple Python packages to huge enterprise-grade -systems spanning massive clusters, has a build—a set of steps that must be -followed to go from a source tree or a checked-out project to a ready-to-use -build product. A build system's job is to automate these steps. - -Build systems are critical to software development. - -They're also one of the most common avoidable engineering failures. - -A reliable, comfortable build system has measurable benefits for software -development. Being able to build a testable, deployable system at any point -during development lets the team test more frequently. Frequent testing -isolates bugs and integration problems earlier, reducing their impact. Simple, -working builds allow new team members to ramp up more quickly on a project: -once they understand how one piece of the system is constructed, they can -apply that knowledge to the entire system and move on to doing useful work. If -releases, the points where code is made available outside the development -team, are done using the same build system that developers use in daily life, -there will be fewer surprises during releases as the “release” build process -will be well-understood from development. - -## Builds Have Needs, Too - -In 1947, Abraham Maslow described a [hierarchy of -needs](http://en.wikipedia.org/wiki/Maslow's_hierarchy_of_needs) for a -person's physical and mental well-being on the premise that all the items at -the lowest level of the hierarchy must be met before a person will be able to -focus usefully on higher-level needs. Maslow's hierarchy begins with a set of -needs that, without which, you do not have a person (for long)—physiological -needs like “breathing,” “food,” and “water.” At the peak, there are extremely -high-level needs that are about being a happy and enlightened -person—“creativity,” “morality,” “curiosity,” and so on. - -![A three-tier pyramid. At the bottom: Automatable. Repeatable. Standardized. -Extensible. Understood. In the middle tier: Simple. Fast. Unit tests. Part of -the project. Environment independent. At the top: Metrics. Parallel builds. -Acceptance tests. Product caching. IDE -integration.](/media/dev/builds/buildifesto-pyramid.png) - -Builds, and software engineering as a whole, can be described the same way: at -the top of the hierarchy is a working system that solves a problem, and at the -bottom are the things you need to have software at all. If you don't meet -needs at a given level, you will eventually be forced to stop what you're -doing at a higher level and face them. - -Before a build is a build, there are five key needs to meet: - -* **It must be repeatable**. Every time you start your build on a given source - tree, it must build exactly the same products without any further - intervention. Without this, you can't reliably decide whether a given build - is “good,” and can easily wind up with a build that needs to be run several - times, or a build that relies on running several commands in the right - order, to produce a build. -* **It must be automatable**. Build systems are used by developers sitting at - their desks, but they’re also used by automatic build systems for nightly - builds and continuous integration, and they can be made into parts of other - builds. A build system that can only be run by having someone sit down at a - keyboard and mouse and kicking it off can’t be integrated into anything - else. -* **It must be standardized**. If you have multiple projects that build - similar things—for example, several Java libraries—all of them must be built - the same way. Without this, it's difficult for a developer to apply - knowledge from one project to another, and it's difficult to debug problems - with individual builds. -* **It must be extensible**. Not all builds are created equal. Where one build - compiles a set of source files, another needs five libraries and a WSDL - descriptor before it can compile anything. There must be affordances within - the standard build that allow developers to describe the ways their build is - different. Without this, you have to write what amounts to a second build - tool to ensure that all the “extra” steps for certain projects happen. -* **Someone must understand it**. A build nobody understands is a time bomb: - when it finally breaks (and it will), your project will be crippled until - someone fixes it or, more likely, hacks around it. - -If you have these five things, you have a working build. The next step is to -make it comfortable. Comfortable builds can be used daily for development -work, demonstrations, and tests as well as during releases; builds that are -used constantly don't get a chance to “rust” as developers ignore them until a -release or a demo and don’t hide surprises for launch day. - -* **It must be simple**. When a complicated build breaks, you need someone who - understands it to fix it for you. Simple builds mean more people can - understand it and fewer things can break. -* **It must be fast**. A slow build will be hacked around or ignored entirely. - Ideally, someone creating a local build for a small change should have a - build ready in seconds. -* **It must be part of the product**. The team responsible for developing a - project must be in control of and responsible for its build. Changes to it - and bugs against it must be treated as changes to the product or bugs in the - product. -* **It must run unit tests**. Unit tests, which are completely isolated tests - written by and for developers, can catch a large number of bugs, but they're - only useful if they get run. The build must run the unit test suite for the - product it's building every build. -* **It must build the same thing in any environment**. A build is no good if - developers can only get a working build from a specific machine, or where a - build from one developer's machine is useless anywhere else. If the build is - uniform on any environment, any developer can cook up a build for a test or - demo at any time. - -Finally, there are “chrome” features that take a build from effective to -excellent. These vary widely from project to project and from organization to -organization. Here are some common chrome needs: - -* **It should integrate with your IDEs**. This goes both directions: it should - be possible to run the build without leaving your IDE or editor suite, and - it should be possible to translate the build system into IDE-specific - configurations to reduce duplication between IDE settings and the build - configuration. -* **It should generate metrics**. If you gather metrics for test coverage, - common bugs, complexity analysis, or generate reports or documentation, the - build system should be responsible for it. This keeps all the common - administrative actions for the project in the same place as the rest of the - configuration, and provides the same consistency that the system gives the - rest of the build. -* **It should support multiple processors**. For medium-sized builds that - aren’t yet large enough to merit breaking down into libraries, being able to - perform independent build steps in parallel can be a major time-saver. This - can extend to distributed build systems, where idle CPU time can be donated - to other peoples’ builds. -* **It should run integration and acceptance tests**. Taking manual work from - the quality control phase of a project and running it automatically during - builds amplifies the benefits of early testing and, if your acceptance tests - are good, when your project is done. -* **It should not need repeating**. Once you declare a particular set of build - products “done,” you should be able to use those products as-is any time you - need them. Without this, you will eventually find yourself rebuilding the - same code from the same release over and over again. - -## What Doesn’t Work - -Builds, like any other part of software development, have -antipatterns—recurring techniques for solving a problem that introduce more -problems. - -* **One Source Tree, Many Products**. Many small software projects that - survive to grow into large, monolithic projects are eventually broken up - into components. It's easy to do this by taking the existing source tree and - building parts of it, and it's also wrong. Builds that slice up a single - source tree require too much discipline to maintain and too much mental - effort to understand. Break your build into separate projects that are built - separately, and have each build produce one product. -* **The Build And Deploy System**. Applications that have a server component - often choose to automate deployment and setup using the same build system - that builds the project. Too often, the extra build steps that set up a - working system from the built project are tacked onto the end of an existing - build. This breaks standardization, making that build harder to understand, - and means that that one build is producing more than one thing—it's - producing the actual project, and a working system around the project. -* **The Build Button**. IDEs are really good at editing code. Most of them - will produce a build for you, too. Don't rely on IDE builds for your build - system, and don't let the IDE reconfigure the build process. Most IDEs don't - differentiate between settings that apply to the project and settings that - apply to the local environment, leading to builds that rely on libraries or - other projects being in specific places and on specific IDE settings that - are often buried in complex settings dialogs. -* **Manual Steps**. Anything that gets done by hand will eventually be done - wrong. Automate every step. - -## What Does Work - -Similarly, there are patterns—solutions that recur naturally and can be -applied to many problems. - -* **Do One Thing Well**. The UNIX philosophy of small, cohesive tools works - for build systems, too: if you need to build a package, and then install it - on a server, write three builds: one that builds the package, one that takes - a package and installs it, and a third that runs the first two builds in - order. The individual builds will be small enough to easily understand and - easy to standardize, and the package ends up installed on the server when - the main build finishes. -* **Dependency Repositories**. After a build is done, make the built product - available to other builds and to the user for reuse rather than rebuilding - it every time you need it. Similarly, libraries and other inward - dependencies for a build can be shared between builds, reducing duplication - between projects. -* **Convention Over Extension**. While it's great that your build system is - extensible, think hard about whether you really need to extend your build. - Each extension makes that project’s build that much harder to understand and - adds one more point of failure. - -## Pick A Tool, Any Tool - -Nothing here is new. The value of build systems has been -[discussed](http://www.joelonsoftware.com/articles/fog0000000043.html) -[in](http://www.gamesfromwithin.com/articles/0506/000092.html) -[great](http://c2.com/cgi/wiki?BuildSystem) -[detail](http://www.codinghorror.com/blog/archives/000988.html) elsewhere. -Much of the accumulated build wisdom of the software industry has already been -incorporated to one degree or another into build tools. What matters is that -you pick one, then use it with the discipline needed to get repeatable results -without thinking. diff --git a/wiki/dev/comments.md b/wiki/dev/comments.md deleted file mode 100644 index 7dc1a68..0000000 --- a/wiki/dev/comments.md +++ /dev/null @@ -1,8 +0,0 @@ -# Comment Maturity Model - -> * Beginners comment nothing -> * Apprentices comment the obvious -> * Journeymen comment the reason for doing it -> * Masters comment the reason for not doing it another way - -Richard C. Haven, via [cluefire.net](http://cluefire.net/) diff --git a/wiki/dev/commit-messages.md b/wiki/dev/commit-messages.md deleted file mode 100644 index 6b3702d..0000000 --- a/wiki/dev/commit-messages.md +++ /dev/null @@ -1,70 +0,0 @@ -# Writing Good Commit Messages - -Rule zero: “good” is defined by the standards of the project you're on. Have a -look at what the existing messages look like, and try to emulate that first -before doing anything else. - -Having said that, here are some things that will help your commit messages be -useful later: - -* Treat the first line of the message as a one-sentence summary. Most SCM - systems have an “overview” command that shows shortened commit messages in - bulk, so making the very beginning of the message meaningful helps make - those modes more useful for finding specific commits. _It's okay for this to - be a “what” description_ if the rest of the message is a “why” description. - -* Fill out the rest of the message with prose outlining why you made the - change. The guidelines for a good “why” message are the same as [the - guidelines for good comments](comments), but commit messages can be - signifigantly longer. Don't bother reiterating the contents of the change in - detail; anyone who needs that can read the diff themselves. - -* If you use an issue tracker (and you should), include whatever issue-linking - notes it supports right at the start of the message, where it'll be visible - even in shortlogs. If your tracker has absurdly long issue-linking syntax, - or doesn't support issue links in commits at all, include a short issue - identifier at the front of the message and put the long part somewhere out - of the way, such as on a line of its own at the end of the message. - -* Pick a tense and a mood and stick with them. Reading one commit with a - present-tense imperative message (“Add support for PNGs”) and another commit - with a past-tense narrative message (“Fixed bug in PNG support”) is - distracting. - -* If you need rich commit messages (links, lists, and so on), pick one markup - language and stick with it. It'll be easier to write useful commit - formatters if you only have to deal with one syntax, rather than four. - (Personally, I use Markdown on projects I control.) - - * This also applies to line-wrapping: either hard-wrap everywhere, or - hard-wrap nowhere. - -## An Example - - commit 842e6c5f41f6387781fcc84b59fac194f52990c7 - Author: Owen Jacobson - Date: Fri Feb 1 16:51:31 2013 -0500 - - DS-37: Add support for privileges, and create a default privileged user. - - This change gives each user a (possibly empty) set of privileges. Privileges - are mediated by roles in the following ways: - - * Each user is a member of zero or more roles. - * Each role implies membership in zero or more roles. If role A implies role - B, then a member of role A is also a transitive member of role B. This - relationship is transitive: if A implies B and B implies C, then A implies - C. This graph should not be cyclic, but it's harmless if it is. - * Each role grants zero or more privileges. - - A user's privileges are the union of all privileges of all roles the user is a - member of, either directly or transitively. - - Obviously, a role that implies no other roles and grants no priveleges is - meaningless to the authorization system. This may be useful for "advisory" - roles meant for human consumption. - - This also introduces a user with the semi-magical name '*admin' (chosen - because asterisks cannot collide with player-chosen usernames), and the group - '*superuser' that is intended to hold all privileges. No privileges are yet - defined. diff --git a/wiki/dev/configuring-browser-apps.md b/wiki/dev/configuring-browser-apps.md deleted file mode 100644 index 8bba0b2..0000000 --- a/wiki/dev/configuring-browser-apps.md +++ /dev/null @@ -1,108 +0,0 @@ -# Configuring Browser Apps - -I've found myself in he unexpected situation of having to write a lot of -browser apps/single page apps this year. I have some thoughts on configuration. - -## Why Bother - -* Centralize environment-dependent facts to simplify management & testing -* Make it easy to manage app secrets. - - [@wlonk](https://twitter.com/wlonk) adds: - - > “Secrets”? What this means in a browser app is a bit different. - - Which is unpleasantly true. In a freestanding browser app, a “secret” is only as secret as your users and their network connections choose to make it, i.e., not very secret at all. Maybe that should read “make it easy to manage app _tokens_ and _identities_,” instead. - -* Keep config data & API tokens out of app's source control -* Integration point for external config sources (Aerobatic, Heroku, etc) -* The forces described in [12 Factor App: - Dependencies](http://12factor.net/dependencies) and, to a lesser extent, [12 - Factor App: Configuration](http://12factor.net/config) apply just as well to - web client apps as they do to freestanding services. - -## What Gets Configured - -Yes: - -* Base URLs of backend services -* Tokens and client IDs for various APIs - -No: - -* “Environments” (sorry, Ember folks - I know Ember thought this through carefully, but whole-env configs make it easy to miss settings in prod or test, and encourage patterns like “all devs use the same backends”) - -## Delivering Configuration - -There are a few ways to get configuration into the app. - -### Globals - - - - - - -* Easy to consume: it's just globals, so `window.appConfig.foo` will read them. - * This requires some discipline to use well. -* Have to generate a script to set them. - * This can be a `` tag or a - standalone config script loaded with `` in them. That may - be an unlikely edge case, but that only makes it a nastier trap for - administrators. - - [There are more edge - cases](https://developer.mozilla.org/en/docs/Web/JavaScript/Reference/Global_Objects/JSON/stringify). - I strongly suspect that a hazard-free implementation requires a - full-blown JS source generator. I had a look at building something out of - [escodegen](https://github.com/estools/escodegen) and - [estemplate](https://github.com/estools/estemplate), but - - 1. `escodegen`'s node version [doesn't generate browser-safe - code](https://github.com/estools/escodegen/issues/298), so string literals - with `` or `` in them still break the page, and - 2. converting javascript values into parse trees to feed to `estemplate` - is some seriously tedious code. - -### Data Attributes and Link Elements - - - - - - -* Flat values only. This is probably a good thing in the grand, since flat configurations are easier to reason about and much easier to document, but it makes namespacing trickier than it needs to be for groups of related config values (URL + token for a single service, for example). -* Have to generate the DOM to set them. - * This is only practical given server-side templates or DOM rendering. You can't do this with bare nginx, unless you pre-generate pages at deployment time. - -### Config API Endpoint - - fetch('/config') /* {"FOO_URL": …, "FOO_TOKEN": …} */ - .then(response => response.json()) - .then(json => someConfigurableService); - -* Works even with “dumb” servers (nginx, CloudFront) as the endpoint can be a generated JSON file on disk. If you can generate files, you can generate a JSON endpoint. -* Requires an additional request to fetch the configuration, and logic for injecting config data into all the relevant configurable places in the code. - * This request can't happen until all the app code has loaded. - * It's very tempting to write the config to a global. This produces some hilarious race conditions. - -### Cookies - -See for example [clientconfig](https://github.com/henrikjoreteg/clientconfig): - - var config = require('clientconfig'); - -* Easy to consume given the right tools; tricky to do right from scratch. -* Requires server-side support to send the correct cookie. Some servers will allow you to generate the right cookie once and store it in a config file; others will need custom logic, which means (effectively) you need an app server. -* Cookies persist and get re-sent on subsequent requests, even if the server stops delivering config cookies. Client code has to manage the cookie lifecycle carefully (clientconfig does this automatically) -* Size limits constrain how much configuration you can do. diff --git a/wiki/dev/debugger-101.md b/wiki/dev/debugger-101.md deleted file mode 100644 index 6d7e773..0000000 --- a/wiki/dev/debugger-101.md +++ /dev/null @@ -1,86 +0,0 @@ -# Intro to Debuggers - -(Written largely because newbies in [##java](http://evanchooly.com) never seem -to have this knowledge.) - -A “debugger” is a mechanism for monitoring and controlling the execution of -your program, usually interactively. Using a debugger, you can stop your -program at known locations and examine the _actual_ values of its variables -(to compare against what you expected), monitor variables for changes (to see -where they got the values they have, and why), and step through code a line at -a time (to watch control flow and verify that it matches your expectations). - -Pretty much every worthwhile language has debugging support of some kind, -whether it's via IDE integration or via a command-line debugger. - -(Of course, none of this helps if you don't have a mental model of the -“expected” behaviour of the program. Debuggers can help you read, but can't -replace having an understanding of the code.) - -## Debugging Your First Program - -Generally, you start running a debugger because you have a known problem -- an -exception, or code behaving strangely -- somewhere in your program that you -want to investigate more closely. Start by setting a _breakpoint_ in your -program at a statement slightly before the problem area. - -Breakpoints are instructions to the debugger, telling it to stop execution -when the program reaches the statement the breakpoint is set on. - -Run the program in the debugger. When it reaches your breakpoint, execution -will stop (and your program will freeze, rather than exiting). You can now -_inspect_ values and run expressions in the context of your program in its -current state. Depending on the debugger and the platform, you may be able to -modify those values, too, to quickly experiment with the problem and attempt -to solve it. - -Once you've looked at the relevant variables, you can resume executing your -program - generally in one of five ways: - -* _Continue_ execution normally. The debugger steps aside until the program - reaches the next breakpoint, or exits, and your program executes normally. - -* Execute the _next_ statement. Execution proceeds for one statement in the - current function, then stops again. If the statement is, for example, a - function or method call, the call will be completely evaluated (unless it - contains breakpoints of its own). (In some debuggers, this is labelled “step - over,” since it will step “over” a function call.) - -* _Step_ forward one operation. Execution proceeds for one statement, then - stops again. This mode can single-step into function calls, rather than - letting them complete uninterrupted. - -* _Continue to end of function_. The debugger steps aside until the program - reaches the end of the current function, then halts the program again. - -* _Continue to a specific statement_. Some debuggers support this mode as a - way of stepping over or through “uninteresting” sections of code quickly and - easily. (You can implement this yourself with “Continue” and normal - breakpoints, too.) - -Whenever the debugger halts your program, you can do any of several things: - -* Inspect the value of a variable or field, printing a useful representation - to the debugger. This is a more flexible version of the basic idea of - printing debug output as you go: because the program is stopped, you can - pick and choose which bits of information to look at on the fly, rather than - having to rerun your code with extra debug output. - -* Inspect the result of an expression. The debugger will evaluate an - expression “as if” it occurred at the point in the program where the - debugger is halted, including any local variables. In languages with static - visibility controls like Java, visibility rules are often relaxed in the - name of ease of use, allowing you to look at the private fields of objects. - The result of the expression will be made available for inspection, just - like a variable. - -* Modify a variable or field. You can use this to quickly test hypotheses: for - example, if you know what value a variable “should” have, you can set that - value directly and observe the behaviour of the program to check that it - does what you expected before fixing the code that sets the variable in a - non-debug run. - -* In some debuggers, you can run arbitrary code in the context of the halted - program. - -* Abort the program. diff --git a/wiki/dev/entry-points.md b/wiki/dev/entry-points.md deleted file mode 100644 index 0e56ce0..0000000 --- a/wiki/dev/entry-points.md +++ /dev/null @@ -1,56 +0,0 @@ -# Entry Points - -The following captures a conversation from IRC: - -> [Owen J](https://twitter.com/derspiny): Have you run across the idea -> of an "entry point" in a runtime yet? (You've definitely used it, just -> possibly not known it had a name.) -> -> [Alex L](https://twitter.com/aeleitch): I have not! -> -> [Owen J](https://twitter.com/derspiny): It's the point where the -> execution of the outside system -- the OS, the browser, the Node -> runtime, whatever -- stops and the execution of your code starts. Some -> platforms only give you one: C on Unix is classic, where there's only -> two entry points: main and signal handlers (and a lot of apps only use -> main). JS gives you _a shit fucking ton_ of entry points. -> -> [Owen J](https://twitter.com/derspiny): In a browser, the pageload -> process is an entry point: your code gets run when the browser -> encounters a `