Ban unarmored GPG keys

[rpdelaney]

In #329 we added a ban for ASCII-armored GPG private keys. But as far as I can tell we don't have a method for detecting un-armored private keys, since they resemble unstructured binary data.

$ gpg --export-secret-keys > secrets
$ file -I secrets
secrets: application/octet-stream; charset=binary

However, that doesn't mean they can't be identified (evidently):

$ file secrets 
secrets: PGP    Secret Key - 4096b created on Sun Nov 17 19:37:04 2013 - RSA (Encrypt or Sign) e=65537 hashed AES with 128-bit key Salted&Iterated S2K SHA-11

It looks like the mime-type for PGP keys is defined here: https://tools.ietf.org/html/rfc3156

---

This leads to a fork in the road for implementation:

We could make a subprocess call to file.

I don't like this for portability reasons. We don't know that file exists and behaves consistently everywhere pre-commit will be deployed.

We could use the magic library.

This works:

$ python3
Python 3.7.1 (default, Nov  6 2018, 18:45:35) 
[Clang 10.0.0 (clang-1000.11.45.5)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import magic
>>> mime = magic.Magic()
>>> mime.from_file('secrets')
'PGP\\011Secret Key - 4096b created on Sun Nov 17 19:37:04 2013 - RSA (Encrypt or Sign) e=65537 hashed AES with 128-bit key Salted&Iterated S2K SHA-1'

But it adds a dependency on a non-standard library: python-magic.

We could try to emulate the mimetype matching ourselves.

Reinvents a wheel and the consequences of getting it wrong when users trust us to get it right could be bad.

---

What do you think?

[asottile]

Neither file nor python-magic are portable enough for pre-commit-hook's sake (they both depend on libmagic).

There's an additional issue in that the hook currently only lints types: [text] (which would already exclude binary files).

That said, there's probably a way to identify the particular binary (from some reading the format seems to be defined in RFC 4880?)

[rpdelaney]

TIL python-magic shares a dependency with file.

There's an additional issue in that the hook currently only lints types: [text] (which would already exclude binary files).

Perhaps performance would be a concern if we started linting binary files as well. The thing should be able to give up and report a negative without reading the entire file.

That said, there's probably a way to identify the particular binary (from some reading the format seems to be defined in RFC 4880?)

I might send a message to the gnupg-users mailing list soliciting advice before we move forward with any plan like this. I expect that RFC to be stable (looks like it was last updated in 2009) but we should be more confident than that, in my opinion -- or at least we should be aware of the maintenance overhead involved with keeping track of future changes. The consequences of false negatives could be significant. ("You had one job...")

[asottile]

Yeah the need to avoid a dependence on libmagic was a lot of the rationale for the creation of identify (which was basically purpose-built for pre-commit!).

[rpdelaney]

Based on the README I'd be concerned identify isn't appropriate for this:

3. Do we recognize the file extension? If so, add the appropriate tags, stop here. These tags would include binary/text.

gpg is an over-determined extension since it is applied to messages and keys (and others I'm not thinking of). I don't think anybody needs a pre-commit hook to prohibit gpg encrypted messages. Maybe I'm wrong about that, though.

Also, it looks like files with unrecognized extensions can't be recognized by identify unless they are some flavor of plain text. So we'd have to rearrange that logic a lot to get this working with unarmored PGP keys.

With all that said, do you think identify should be refactored to make this possible, or would this just be a separate and free-standing hook in pre-commit-hooks?

[asottile]

The easiest thing (for now) is to probably make a free-standing hook. It may make sense in the future to have identify be able to identify binary things based on magic number headers -- though there hasn't been much of a use case for it until now.

[rpdelaney]

Alright. I'll draft something to gnupg-users at some point and report back. One last question before I go, since it will affect what I ask them:

• Do you think we should be able to prohibit gpg keys with a passphrase, or are we only concerned about those without?

I tend to think that if possible, we should be able to ban them entirely, password protected or not. I don't have an opinion on what the default behavior should be -- just that if we can it would be nice to let the user toggle it with a switch.

[asottile]

ban entirely seems fine to me

[rpdelaney]

Libmagic has python bindings with examples: https://github.com/file/file/tree/master/python

We could port code if necessary. Not sure yet how complex that would be.

[asottile]

Neither file nor python-magic are portable enough for pre-commit-hook's sake (they both depend on libmagic).

libmagic is off the table

[rpdelaney]

Yeah, you're too fast. I think you missed my ninja edit?

[asottile]

I saw your edit, I assumed you meant port the bindings which doesn't help because the gpg detection parts are inside the libmagic.so itself (C)

[rpdelaney]

Yeah I didn't put that clearly, sorry. It's not that useful that they have examples of how to call libmagic from python. I meant more like porting C code into python. I don't have time to dig deeper right now but I wanted to leave the comment as a reminder for when I look at this again.

[asottile]

if anything I think we can find the magic bytes in this file and detect those: https://github.com/file/file/blob/master/magic/Magdir/gnu

[rpdelaney]

From that file:

# GnuPG
# The format is very similar to pgp
0	string          \001gpg                 GPG key trust database
>4	byte            x                       version %d
# Note: magic.mime had 0x8501 for the next line instead of 0x8502
0	beshort		0x8502			GPG encrypted data
!:mime text/PGP # encoding: data

I don't think we need to reject symmetrically encrypted files, nor keybox files.

PGP definitions here: https://github.com/file/file/blob/master/magic/Magdir/pgp

This stuff jumped out at me:

0       beshort         0x9501                  PGP key security ring
!:mime	application/x-pgp-keyring
0       beshort         0x9500                  PGP key security ring
!:mime	application/x-pgp-keyring
0	beshort		0xa600			PGP encrypted data
#!:mime application/pgp-encrypted
#0 string -----BEGIN\040PGP text/PGP armored data

>0 byte 0x6c Secret-Key (old)
>0 byte 0x6e Secret-Subkey (old)
>>1 byte&0xc0 0x40 Secret-Key
>>1 byte&0xc0 0xc0 Secret-Subkey

# PGP RSA (e=65537) secret (sub-)key header

0	byte	0x95			PGP Secret Key -
>1	use	pgpkey
0	byte	0x97			PGP Secret Sub-key -
>1	use	pgpkey
0	byte	0x9d
# Update: Joerg Jenderek
# secret subkey packet (tag 7) with same structure as secret key packet (tag 5)
# skip Fetus.Sys16 CALIBUS.MAIN OrbFix.Sys16.Ex by looking for positive len
>1	ubeshort	>0
#>1	ubeshort	x		\b, body length 0x%x
# next packet type often 88h,89h~(tag 2)~Signature Packet
#>>(1.S+3)	ubyte	x		\b, next packet type 0x%x
# skip Dragon.SHR DEMO.INIT by looking for positive version
>>3	ubyte		>0
# skip BUISSON.13 GUITAR1 by looking for low version number
>>>3	ubyte		<5		PGP Secret Sub-key
# sub-key are normally part of secret key. So it does not occur as standalone file
#!:ext	bin
# version 2,3~old 4~new . Comment following line for version 5.28 look
>>>>3	ubyte		x		(v%d)
>>>>3	ubyte		x		-
# old versions 2 or 3 but no real example found
>>>>3	ubyte		<4
# 2 byte for key bits in version 5.28 look
>>>>>11		ubeshort	x	%db
>>>>>4		beldate		x	created on %s -
# old versions use 2 additional bytes after time stamp
#>>>>>8		ubeshort	x	0x%x
# display key algorithm 1~RSA Encrypt|Sign - 21~Diffie-Hellman
>>>>>10	  	use		key_algo
>>>>>(11.S/8)	ubequad		x
# look after first key
>>>>>>&5	use		keyend
# new version
>>>>3	ubyte		>3
>>>>>9		ubeshort	x	%db
>>>>>4		beldate		x	created on %s -
# display key algorithm
>>>>>8		use		key_algo
>>>>>(9.S/8)	ubequad		x
# look after first key for something like s2k
>>>>>>&3 use keyend

I really have no idea about session keys. That's a whole 'nother thing.

[asottile]

ooh neat! I saw 0 string \001gpg GPG key trust database in the file I linked and assumed (incorrectly) I had found the right one