Android, Windows

Text encodings: A horror story

October 26th, 2010 | Comment?

Things have been rather quite here at Cryptnos lately, primarily because we’ve been very busy. Not just with Cryptnos itself, unfortunately, but it’s been keeping us busy as well. I wanted to make a quick post (or as quick as I seem to be able to write) to let you know what’s been going on.

The bad news is that we’ve had two outstanding but separate issues vying for our attention. One affects only Cryptnos for Windows, while the other was reported with Cryptnos for Android but affects the Windows client as well. The second issue also has been delaying work on the Java client, still in progress; we wanted to get that ironed out before we ventured in the newer waters. The good news is that we think we have both issues either fixed or mitigated (i.e. postponed), and we hope to release 1.2.x versions of both the Windows and Android clients soon. The even better news is that Cryptnos Online is pretty much unaffected; well, it’s affected, but the other clients can be configured to conform to it for compatibility.

The worst news of all, however, is that the fix for the biggest issue, the one that affects all clients, may potentially break passwords. Read on, if ye dare….

Let’s get the easy one over with first: There is a known issue with the new update checker added to Cryptnos for Windows 1.1.x. You can read all the gory details in this Google Code issue item, as well as the version 1.1.1 release announcement. Long story short: we added code to check the Cryptnos site periodically to see if a new version had been released. We tested this code on as many machines as we could and encountered no problems, but a few users reported a critical error that prevented Cryptnos from even starting. We tracked the issue down to the new update checker code. We added additional layers of error checking and reporting, but that didn’t seem to help. Since we can’t reproduce the problem and we don’t have enough information to track it down even further, we’ve decided to cut our losses and remove the update checker code from the project, at least for now. This will at least postpone the issue and let folks actually use the program, which is what everyone really wants.

Unfortunately, this means that people will have to manually check for updates for now, but that’s the price we’ll have to pay. Of course, there are ways to get news about new releases; you can always subscribe to one or more of our RSS feeds. The main feed will give you the fire hose (well, it’s more like an anemic, cheap water gun with our frequency of updates), while those only interested in release announcements can subscribe directly to the Releases category feed. This will also help you with updates for the other platform releases as well, although folks installing the Android client from the Android Market should get releases pushed out automatically.

Issue #2: This issue was initially reported for Cryptnos for Android, but we later discovered that it seems to affect all the clients. A user upgraded her phone and moved from a custom, unofficial port or “ROM” of Android to a different phone with a standard “vanilla” version of Android. In the process, her passwords seemed to “break”; the data was retained just fine, but entering all the correct parameter information resulted in different generated passwords on new phone from those generated on the old phone.

At this point, communication with the user seemed to break off and we haven’t heard back from her. However, we pursued this as best as we could and looked at the most likely culprit: character or text encoding. Without getting too technical, encodings are the means of converting human readable text into a binary, numeric representation that computers can understand, and vice versa. The problem with text encoding is that for various historical, cultural, and technical reasons, there are dozens of encodings in use today and many of them are incompatible with each other. A given string of text may be encoded in many different ways, resulting in many different binary representations. And since the cryptographic hashes Cryptnos uses for password generation work only on binary data, the text encoding used to convert the text of your site tokens and master password becomes a critical element.

The bottom line is that I made a number of incorrect assumptions about how Java and .NET handle character encoding. Both use a globally accepted standard, Unicode, internally. The problem is, there are several different flavors of Unicode, each resulting in different binary representations, and once you start writing parameters to databases, registry keys, and export files, other defaults come into play, some of which may be incompatible. If different character encodings are used on different platforms or even different versions of the same platform, the resulting generated passwords will not be the same.

The good news is that the fix is essentially an easy one: make sure to specify a safe, globally acceptable encoding for all platforms. In our case, we decided to go with UTF-8, which is convenient because it seems from casual observation to be the default for Android and it’s easy to specify in .NET as well. In practice, however, applying this fix becomes a minefield of compatibility. Once we start changing the encoding, we potentially break user’s passwords. There’s no way around this issue, and it’s impossible to know exactly how many users would be impacted. (None of our tracking metrics record this kind of information, unfortunately.)

It’s been a harrowing experience trying to find the best way to approach this thorny issue. However, here are some bullet points that we came away with from this experience:

  • Starting with Cryptnos 1.2.x, all new installations of Cryptnos on every platform will try to default to using UTF-8 for text encoding, regardless of any system defaults. This offers the greatest compatibility and brings it directly in line with Cryptnos Online, which already uses UTF-8 (and which is not user configurable).
  • Existing installations of Cryptnos will continue to use the system default encoding, i.e. whatever encoding the underlying operating system or environment uses. This may or may not be UTF-8; it depends on how the platform is configured. Sticking with the existing setting will keep old passwords from “breaking”, but may make them incompatible with other, newer installations.
  • All versions of Cryptnos 1.2.x and higher, except for Cryptnos Online, will include a new Advanced Settings section that will allow the user to modify the text encoding Cryptnos will use during password generation. You’ll get a warning dialog the first time you go into this section during a given session, just to let you know you could break things fiddling with these settings. However, this will put the option to change the encoding in your hands, giving you the ability to change your encoding on a given platform so it matches another and keeps your passwords in sync.
  • Import/export files generated by Cryptnos 1.1.x and higher always have and always will use UTF-8 internally and will not be changed by the above setting. Import/export files from versions of Cryptnos even older than that will continue to use their own, platform specific formats, complete with their original encodings. Thus, all your export backups should be safe.
  • As stated above, our casual testing has shown that most if not all “vanilla” or standard installations of Android already use UTF-8 as the system default encoding. It might be a good idea to go into the new Advanced Settings activity to verify this; there’s a little label in there that will display the system default regardless of the current application setting. We preface this with “casual testing” here because we’ve tested every device we can get our hands on, as well as the emulators provided with Google’s Android development kit, and all of these show UTF-8 as the default. However, our call for reports from the field have been met only with silence, so there’s no way to know for sure if this is true for every case.
  • As a corollary to the above, we’re making an official policy that non-standard versions of Android (“mods”, “ROMs”, custom-built images, i.e. any version of the OS that does not come directly from Google or the device manufacturer) will be considered unsupported. You’re welcome to use Cryptnos on these systems, of course, provided there is an understanding that we will not provide support for these platforms. Google’s standard text encoding may be UTF-8, but it’s too easy for non-standard versions of the platform to go in different directions. We think that the new Advanced Settings activity will make this moot in the long run, but we’d rather be safe than sorry.
  • Many users will hopefully be unaffected by this issue, especially those in English speaking locations. Many character sets include US ASCII as a subset, including the various flavors of Unicode and especially UTF-8. Sadly, users of other languages may have different standard character sets, especially on Windows, meaning the encodings and thus the generated passwords will be different. There is no easy or user-friendly fix for this issue. Our only suggestion is to back up your parameter data and give it a try. With version 1.2.x, you will have the ability to modify the encoding used by all platforms (exception Online) and thus bring them into sync. Pick one platform to be your master and adjust the settings in the other clients to match the master client. In a worst-case scenario, if you continue to have incompatibility issues you can always “downgrade” to an earlier version of Cryptnos and continue to use it. You can find earlier versions in the Download sections of the various Google Code sites, although you may to search for “All Downloads” to see deprecated packages.

We’re still in the testing phase for version 1.2.0 for both Windows and Android. Users daring enough to help us out can find beta versions on the Google Code sites (Windows and Android). Once testing is complete, we’ll announce here when the 1.2.0 versions have been officially released.

Tags: , , ,


You can skip to the end and leave a response. Pinging is currently not allowed.

Be nice. Keep it clean. Stay on topic. No spam.

You can use these tags:
<a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>

You must be logged in to post a comment.