The infinite stream
If you run
euca_conf --get-credentialson eucalyptus 4.2 you will see the following warning:
warning: euca_conf is deprecated; use ``eval `clcadmin-assume-system-credentials`'' or ``euare-useraddkey''`
There are numerous reasons for that command’s deprecation, but what causes confusion is the fact that it has two replacements. Setting up a new cloud now involves more than just one set of credentials, and if you’re used to having fully-functional credentials immediately this is likely to trip you up.
One of the most common complaints about
euca_confis that it tries to be everything to everybody. It combines multiple types of functionality that need to run in different places, adding excess dependencies and requiring one to log into systems that one normally shouldn’t have to. Eucalyptus 4.2 introduces new administration tools that break
euca_conf’s functionality down into three groups with more specific purposes:
- Whole-cloud administration tools
- Cloud controller (CLC) support scripts
- Cluster controller (CC) support scripts
Cloud controller and cluster controller support scripts can run only on those specific systems, and thus are only installed alongside them. The rest of the administration tools are web service clients, similar to euca2ools, that can run from anywhere. All they need are access keys.
But where do those access keys come from?
Out with the Old
In the old regime, access keys and other credentials come in the form of a zip file containing a bunch of certificates as well as
eucarc, a shell script that sets a bunch of environment variables that include service URLs and the access keys themselves. The first zip file it creates is missing several service URLs because those services have yet to be set up, and it doesn’t use DNS either because that has yet to be set up as well.
Once DNS and all of the services are ready, we then have the cloud generate a new zip file. Everything seems fine until something changes for whatever reason and we need to obtain a third one. Since we can only have two certificates at a time, though, this third zip file will not include one. This causes countless problems for automation that relies on them, including eucalyptus’s own QA scripts.
That said, the zip file still has some particularly useful properties:
- It’s a single file for the administrator to e-mail to new users
- It contains both access keys and service URLs
- It (usually) contains all of the certificates needed to bundle images
euca2ools.inifile also has the first two of those properties, while also managing to be more flexible. Any euca2ools commands that can create access keys, such as euare-useraddkey and euare-usercreate, can generate euca2ools.ini files automatically. That leaves just certificates, which we dealt with by making them all optional or possible to obtain automatically.
In with the New
In isolation, euca2ools commands alone have a chicken-and-egg problem: they require access keys to run, but a new cloud doesn’t have any access keys. We break this loop by splitting eucalyptus installation into two phases, each with different credentials.
A cloud controller support script,
clcadmin-assume-system-credentials, provides temporary setup credentials. This script works similarly to
euare-assumerole, but it is much more limited and it only works on a cloud controller. Setup credentials cannot be used for normal system operation; they provide access only to service registration, service configuration, and IAM services – the minimum necessary to get up and running with euca2ools.
# eval `clcadmin-assume-system-credentials` # euserv-register-service -t user-api -h 198.51.100.2 ufs-1 # euctl system.dns.dnsdomain=mycloud.example.com
Once DNS and an IAM service are set up, you can use euca2ools to create long-lived admin credentials that let you access the cloud’s full functionality. It is these credentials that are the replacements for the zip file. Once you create them, you are unlikely to ever need setup credentials again.
# euare-usercreate -wld mycloud.example.com gholms > ~gholms/.euca/mycloud.ini
Here is an explanation of the various parts of that command:
gholms: Create a user named gholms
-w: Write out a euca2ools.ini file
-l: In that file, make that user the default for this cloud
-d mycloud.example.com: Use the domain
mycloud.example.comas the cloud’s DNS domain
Normally, when this command writes a configuration file it will pull the DNS domain from the IAM service’s URL, but since this is the very first user we have to supply it by hand because it has not yet been set.
If you are interested in using eucalyptus’s administration roles instead of full-blown admin credentials, you would create a new account here and add it to whatever roles you need.
Once you have a set of admin credentials you can use this for day-to-day cloud administration the same way you would with a classic
$ export AWS_DEFAULT_REGION=mycloud.example.com $ euare-accountcreate -wl alice > alice.ini $ mail -s "Try out this shiny, new cloud" -a alice.ini ...
You can change the name of the region in the configuration file if you want. The domain name is simply the default.
Normally, when someone asks me to merge something in git I need to add his or her repository using
git remote add, fetch the branch I need, and then merge it. When someone submits a pull request to a project hosted on GitHub, however, GitHub additionally publishes it as something I can fetch from my own repository:
$ git ls-remote origin | grep pull/6 03d7fb7af91a74bb7658a0742fa68bfeb5d50a3f refs/pull/6/head f8ebbc62555143019947f1255b064cda38fd239f refs/pull/6/merge 8a42021dbabcfc222b4d47c5ada47e344154d934 refs/pull/60/head f4d8052000601e59e4e7d4dec4aa4094df4e39a0 refs/pull/61/head 8a519986f5b59721692ec75608edf0f404f88e87 refs/pull/62/head 9c03e723e8b50ca56a1257659d686a68a69e6e40 refs/pull/62/merge f3a983c6fc3ff236c2bc678cfec3885da609f79a refs/pull/63/head e7953c21a77fb37fb7158dd87fc0c156dd8f97ae refs/pull/63/merge
With this I can use one line of configuration to create a convenient shortcut that lets me immediately check out any pull request:
$ git config --add remote.origin.fetch "+refs/pull/*/head:refs/remotes/origin/pull/*" $ git checkout pull/63 Branch pull/63 set up to track remote branch pull/63 from upstream. Switched to a new branch 'pull/63'
People have to use SSH or HTTPS to push to GitHub, but when fetching one can use git’s own network protocol because it is generally faster. You can make a specific repository on your machine use SSH only for pushing by cloning it with the faster
git://URL and running something like this:
$ git config remote.upstream.pushurl email@example.com:gholms/boto
That works nicely, but you have to do it once for every single repository you want to interact with. That quickly becomes annoying. Thankfully, you can leverage git’s URL-rewriting mechanism to make this easier:
$ git config --global url.git://github.com/.insteadOf github: $ git config --global firstname.lastname@example.org:.pushInsteadOf github:
This adds two new rules to your git configuration:
- If a URL starts with
github:then replace that with
- If a URL starts with
github:and you are pushing then replace that with
After you do that you can simply use URLs like
github:gholms/botowhen cloning. They will get rewritten to
email@example.com:gholms/botowhen pushing, and
git://github.com/gholms/botothe rest of the time, speeding things up without creating additional work in the future.
This should work if you prefer HTTPS for pushing to GitHub, or if you use other servers, too. Just tweak the commands.
- If a URL starts with
When looking for a summary of a git repository’s history, the output of
git logisn’t always as informative as one might like. It displays every commit in chronological order, which effectively hides the changes that merges bring in. It is also quite verbose, showing complete log messages, author info, commit hashes, and so on, drowning us with so much info that only a few commits will fit on the screen at once. After supplying the command with the right cocktail of options, though, its output becomes a significantly better summary:
The output above came from a command that is long enough that I made an alias, for it,
git graph, in my
[alias] graph = log --graph --abbrev-commit --date=relative --pretty=format:'%Cred%h%Creset -%C(yellow)%d%Creset %s %Cgreen(%cr) %C(blue)<%an>%Creset'
Don’t forget that
git logaccepts a list of things to show logs for as well, so if you want to look at the logs for
branch-2you can simply use
git graph branch-1 branch-2to make them both show up in the graph.
To unlock a disk that is encrypted with OS X’s FileVault feature one needs to type in the password that belongs to any user on the machine who is allowed to unlock the disk. The system then boots and helpfully logs you in as that user. In general that is probably a convenient little feature, but for me it just makes things awkward – I want to use different passwords for unlocking the disk and logging into my user account. To make that work I have to create a second account dedicated to unlocking the disk, get logged into that one when the system boots, then immediately log back out so I can log in as the user I actually want to use.
Or do I?
The system that powers FileVault, Core Storage, combines full disk encryption and some logical volume management features in a manner similar to LVM and LUKS on Linux. As a dedicated user of those features on my Linux-based machines, I jumped at the chance to read more about OS X’s version only to discover next to no official documentation whatsoever. On the bright side, after searching some more and then playing around with it for a while I finally figured out how to make it do everything I wanted it to do, including using a disk password. You just have to boot into recovery (or, in my case, the OS X 10.9 install DVD that I threw together) and use the command line.
Create a partition
Since this is a laptop I want to put everything on one huge partition. The utility that handles nearly everything related to disk management on OS X,
diskutil, requires us to choose a type of filesystem for every partition we tell it to create, so for now we’ll just tell it to mark the partition as HFS+ and not bother to format it.
# diskutil partitiondisk disk0 1 gpt jhfs+ %noformat% 100% Started partitioning on disk0 Unmounting disk Creating the partition map Waiting for the disks to reappear Finished partitioning on disk0 /dev/disk0 #: TYPE NAME SIZE IDENTIFIER 0: GUID_partition_scheme *256.1 GB disk0 1: EFI EFI 209.7 MB disk0s1 2: Apple_HFS 255.7 GB disk0s2
The system actually created two partitions for us. The first is a small EFI system partition, which helps the system start, and the second is the big partition for our data that spans the rest of the disk.
Or does it?
# gpt show disk0 start end index contents 0 1 PMBR 1 1 Pri GPT header 2 32 Pri GPT header 34 6 40 409600 1 GPT part - C12A7328-F81F-11D2-BA4B-00A0C93EC93B 409640 499446368 2 GPT part - 48465300-0000-11AA-AA11-00306543ECAC 499856008 262151 500118159 32 Sec GPT table 500118191 1 Sec GPT header
It seems to have left a little empty space in between the last partition and the boilerplate bits at the end of the disk. More on that later.
Create a volume group
The next step is quite familiar to LVM users. Before we can add encryption we need to create a logical volume group that combines one or more partitions (physical volumes) into a single virtual “disk” that we can slice up.
Why more than one disk? That’s how Fusion Drive works.
# diskutil cs create vg_sodium disk0s2 Started CoreStorage operation Touching partition type on disk0s2 Adding disk0s2 to Logical Volume Group Creating Core Storage Logical Volume Group Switching disk0s2 to Core Storage Waiting for Logical Volume Group to appear Discovered new Logical Volume Group "A642DBC3-644C-4C23-8337-ADBCDD9C85F2" Core Storage LVG UUID: A642DBC3-644C-4C23-8337-ADBCDD9C85F2 Finished CoreStorage operation # diskutil cs list CoreStorage logical volume groups (1 found) | +-- Logical Volume Group A642DBC3-644C-4C23-8337-ADBCDD9C85F2 ========================================================= Name: vg_sodium Status: Online Size: 255716540416 B (255.7 GB) Free Space: 255380987904 B (255.4 GB) | +-< Physical Volume A39FA7E9-F52F-4FFA-9A70-F07304111115 ---------------------------------------------------- Index: 0 Disk: disk0s2 Status: Online Size: 255716540416 B (255.7 GB) # diskutil list disk0 /dev/disk0 #: TYPE NAME SIZE IDENTIFIER 0: GUID_partition_scheme *256.1 GB disk0 1: EFI EFI 209.7 MB disk0s1 2: Apple_CoreStorage 255.7 GB disk0s2 3: Apple_Boot Boot OS X 134.2 MB disk0s3
This changed the type of the second partition and constructed a volume group out of it, but it also created a third partition on which it can put whatever minimal data are needed to allow the system to boot from a Core Storage volume. With a quick look at the partition table we can see that it carved this out of the space it reserved at the end of the disk earlier:
# gpt show disk0 start end index contents 0 1 PMBR 1 1 Pri GPT header 2 32 Pri GPT header 34 6 40 409600 1 GPT part - C12A7328-F81F-11D2-BA4B-00A0C93EC93B 409640 499446368 2 GPT part - 48465300-0000-11AA-AA11-00306543ECAC 499856008 262144 3 GPT part - 426F6F74-0000-11AA-AA11-00306543ECAC 500118152 7 500118159 32 Sec GPT table 500118191 1 Sec GPT header
Create a logical volume
LVM has three layers: physical volumes, volume groups, and logical volumes. Since Core Storage includes encryption it adds a fourth layer, the logical volume family, between the volume group and logical volume levels. When you boot a FileVault-enabled system it’s the volume family that your password unlocks. The final step before we can install OS X is to create an encrypted volume family and the logical volume that we will actually install onto.
# diskutil cs createvolume vg_sodium jhfs+ 'Macintosh HD' 100% -passphrase Passphrase for new volume: Confirm new passphrase: Started CoreStorage operation Waiting for Logical Volume to appear Formatting file system for Logical Volume Initialized /dev/rdisk13 as a 238 GB case-insensitive HFS Plus volume with a 24576k journal Mounting disk Core Storage LV UUID: 46A6CA83-9CDC-4978-ADF2-E4DC3F203DBD Core Storage disk: disk13 Finished CoreStorage operation # diskutil cs list CoreStorage logical volume groups (1 found) | +-- Logical Volume Group A642DBC3-644C-4C23-8337-ADBCDD9C85F2 ========================================================= Name: vg_sodium Status: Online Size: 255716540416 B (255.7 GB) Free Space: 0 B (0 B) | +-< Physical Volume A39FA7E9-F52F-4FFA-9A70-F07304111115 | ---------------------------------------------------- | Index: 0 | Disk: disk0s2 | Status: Online | Size: 255716540416 B (255.7 GB) | +-> Logical Volume Family AC7B483C-0524-4ACF-8083-9EFD963F81A5 ---------------------------------------------------------- Encryption Status: Unlocked Encryption Type: AES-XTS Conversion Status: Complete Conversion Direction: -none- Has Encrypted Extents: Yes Fully Secure: Yes Passphrase Required: Yes | +-> Logical Volume 46A6CA83-9CDC-4978-ADF2-E4DC3F203DBD --------------------------------------------------- Disk: disk13 Status: Online Size (Total): 255380987904 B (255.4 GB) Conversion Progress: -none- Revertable: No LV Name: Macintosh HD Volume Name: Macintosh HD Content Hint: Apple_HFS
Now we have a disk called “Macintosh HD” that is encrypted with its own non-user-specific password, formatted, and ready to go. Just switch back to the OS X installer and let it run.
Create a recovery key
After the installer finishes and the system reboots we see the familiar FileVault login screen with just one option: “Disk Password.” Success! Now we just enter that, run through the usual first-time boot stuff, and finally open up a terminal to create a recovery key:
$ sudo fdesetup changerecovery -personal Enter a password for '/': New recovery key = 'XPO6-E4OL-XQG6-TGV9-GFOZ-GB8M'
Add more users
Since I generally don’t log in with an administrative account I created a second, regular user, but then the next time I rebooted I discovered a problem: the FileVault password screen asked for the disk password or the new user’s password. Thankfully, removing that user from the list of choices is simple:
$ sudo fdesetup remove -user gholms
Now that I have gone through all this my machine finally seems to be working exactly the way I want it to, but a few unanswered questions still remain in the back of my mind:
- How am I supposed to do this on a newer Mac that ships with neither an install DVD nor an upgrade application that I can turn into one?
- How do I make OS X stop automatically adding new users to FileVault’s password list?
- Where is the official Core Storage documentation?
Do know the answers to any of these? Leave a comment!