Crypto++

[Ykkrosh]

I want to compute the MD5 of a stream of bytes (for simulation state hashing), feeding in many arbitrary-sized (typically small) chunks of bytes. Currently I'm using Crypto++ which makes it trivial (create an MD5 object, call md5.Update(data, len)). The problem is that Crypto++ seems to be a pain on Windows: it can compile as both dynamic and static, except the dynamic library only includes FIPS Approved algorithms (which doesn't include MD5), and the static library is 45MB (and we want to provide pre-built libraries for Windows in our SVN, and that's unreasonably big). Also its .h files trigger a load of warnings in MSVC.

(There are some future cases where other Crypto++ algorithms might be useful - zlib for stream compression, and some kind of proper encryption for networking. Currently it's just being used in the one place, for MD5.)

Some options I can think of:

* Strip out all the bits of Crypto++ that we don't use, before compiling it as a static library, to save space. (But that's a pain with dependencies, upgrades, testing, etc, and recompiling when we upgrade the standard compiler.)

* Use a lighter-weight crypto library which provides similar functionality but doesn't have zillions of templates. (Which ones are good?)

* Copy the MD5 parts from an existing library and hack it to remove unnecessary dependencies, then import it into our code. (That solves all the bother with external libraries, and MD5 is pretty straightforward so there's not much code involved. Got to be very careful not to break padding and endianness etc, though. Doesn't help us with non-MD5 algorithms, but zlib is probably easy enough to use as a raw API or a custom wrapper, and secure multiplayer is a complete unknown at the moment so we can't plan for it anyway.)

Any thoughts on the best approach?

[janwas]

hm, my first thought/question is why you're focused on MD5. For example, SHA-1 hopefully isn't that much slower (I've written a fast implementation myself, the point being that it's possible) and is included in the DLL (reference: http://www.cryptopp.com/docs/ref/). Come to think about it, we could use my implementation, but it is written in assembly (then again, we already have the build system infrastructure to handle that).

About other algorithms in crypto++: I agree we can live with zlib and don't need to turn to crypto++ for a slightly nicer interface. crypto++ would be useful for encryption, though.

It seems it'd be good to stick with the FIPS-approved (and smaller) DLL. I don't have experience with other libraries and it's been awhile since I looked at alternatives.

As to the warnings: not a problem We can wrap cryptopp's header in one of ours (see lib/external_libraries/*) - that allows setting warning level to W3 or temporarily disabling specific warnings.

[Ykkrosh]

Hmm, hadn't thought of that, and I suppose SHA-1 could work - apparently it's only almost twice as slow as MD5, and the bottleneck is likely to be elsewhere. But if we used the DLL for SHA-1, we still couldn't use Crypto++'s zlib, and it seems a bit silly to have a 1.2MB DLL for a single hash function... Seems like the quickest solution for now though, so I guess I'll go with that

[janwas]

Only 2x slower sounds decent (now at home, I can check my old implementation's quoted performance: apparently it required 11 cycles per byte on an Athlon XP).

Don't worry, the code size police won't come after you for adding another DLL - quick definitely has its virtues, especially for a non-critical component like this

[kaiserfro]

OpenSSL's libcrypto should provide md5. I've never tried it on windows though, but is should be doable, i would think.

[janwas]

Thanks for the suggestion!

I see Philip went with the FIPS DLL version of cryptopp - unfortunately it looks like that fails (throws OS_RNG_ERR) on WinXP SP2 for reasons unknown. This happens before main is entered, so it's hard to debug.

Maybe it is easier to go with OpenSSL after all. I gave building it a go: it requires Perl and NASM (which we use anyway), and results in an 800 KB libeay32.dll that contains MD5 and SHA1 functions. Want to use that instead?

(If so: I installed the latest ActivePerl, grabbed openssl-0.9.7-stable-SNAP-20100105.tar.gz, added NASM to the path, copied nasm.exe to nasmw.exe, and followed the instructions in INSTALL.W32)

[Ykkrosh]

an 800 KB libeay32.dll that contains MD5 and SHA1 functions. Want to use that instead?

Hmm... Not really . Seems like a lot of bloat and complexity and potential for unexpected breakages (like this OS_RNG_ERR thing), plus the cost of reading the documentation and working out how to use it. I'd expect it should only take half an hour to copy-and-paste an MD5 implementation and write a wrapper to feed it 16-byte chunks and test it, so I'd be happier to simply do that. (I could do that later today, depending on how wildly wrong my estimate is.)

[janwas]

Fair enough, but please keep in mind future encryption or authentication requirements of the network code - we won't want to roll that ourselves

[Ykkrosh]

Yeah, that would definitely be complex enough to justify an external library . I'm just thinking that since we don't yet know what those requirements will be, we can't make any correct decisions about what library we'll need, and it would be wasteful to set one up now and then find we have to throw it away later, so I'd prefer to wait before picking one (and avoid relying on any in the meantime).

[janwas]

Makes sense (After all, MD5sum is a pretty minor snippet of code)

[Ykkrosh]

(This is implemented now here (tests) - hopefully not many bugs...)

(I've left the Crypto++ include/lib/DLL files in SVN in case we do decide to use it in the future; if we decide to do something different for networking, we can clean it up then.)