HexEdit 5.0

I haven't released a new version of HexEdit for 5 years!! But I still keep getting emails about it, even feature requests.It's time for a new version - 5.0!

In the interim, between other things, I have found time to add a few new things (mainly for my own use) such as:

• made the latest version open-source - now hosted on GitHub
• finally implemented a more advanced file compare facility
• added ability to preview graphics file from memory (easy using FreeImage lib)
• lots of support for Windows 7 taskbar (see below)
• started using excellent Crypto++ library for digests/encryption
• fixed calculator annoyances (+ moved to MPIR inf. precision library)
• improved performance of some operations using SIMD instructions (SSE2)
• lots of other minor fixes

Open-source

I first released HexEdit as open-source software on 1/1/1999 and it was immensely popular within a year (downloaded more than 1 million times just from download.com by June 2000). It was ranked highest in Google for "Windows hex editor" and "binary file editor" and third for "hex editor" at that time.

In 2001 I released a shareware version (later called HexEdit Pro). I only did this because many users wanted to pay for it and I thought it might encourage me to continue work on it.

The shareware version was quite popular over a decade ago - I was selling about one copy per day for a while.  However, over the years sales dwindled.  Worse was that the Google ranking dropped away rapidly.  It seems that Google rankings, for whatever reason, favor free things - even though I always provided free open-source version which was the same as the shareware version from a few years earlier (in an SVN repo on SourceForge).

“... requires
Visual Studio
2010”

I retired the shareware version in 2012. The complete latest source of the latest version is now in a Git repo on GitHub. Note that the latest code requires Visual Studio 2010 or later since it makes use of SSE2 intrinsics, which were only added to the C++ compiler in VS2010.

File Compare

The original version of HexEdit had a basic file compare feature which allowed you to compare 2 files (or different parts of the same file) simply by opening 2 windows and pressing Alt+C. Additionally, if the "highlighter" was on this would highlight the differences between the files while you continued to press Alt+C.

However, this was facility was limited. For example, you could not get an overview or even a count of the differences. Most importantly it could not handle insertions/deletions.

“[old system] couldn't handle
  insertions/deletions”

Over the years I tried out different ways to do file compares and was never happy with the results. I even tried other hex editors to see how they did it. (I usually avoid trying other hex editors since they tend to taint my thinking.) I have yet to see a completely satisfactory file compare facility in a hex editor but have finally settled on an implementation for HexEdit 5.0.

Noteable features include:

• allows control of how searching continues once an insertion/deletion is found
• very fast (at least without insertions/deletions enabled) by using SSE2
• separate compare list "tool" window listng all areas the same/different/ins/del
• self-compare feature that allows you to monitor changes to a file over time

Preview view

A few years ago a user asked for the ability to preview bitmap files in HexEdit. Using the FreeImage.DLL (see FreeImage Library) I found this feature very easy to add to HexEdit. (It is displayed in another view window like templates or aerial view - either in a split window or a full-size tabbed window.)

The "preview" view shows the graphics file data as it is in memory, so you can play around with file data and immediately see the changes you have made. This is enormously powerful when combined with a template for the file format you are using.

Moreover the HexEdit preview window (just like the Windows photo previewer and other photo viewers) provides:

• zoom
• drag to scroll
• full support for transparency
• different background options for transparency (eg checkboard)
• supports many graphics file formats (see FreeImage Features)

Actually it is better than the Windows photo previewer as it supports many more image file formats (not just bitmap formats) and has more background options.  (Though it does not yet have rotate/flip options.)

Windows 7 TaskBar 

There are lots of useful things that appeared in Windows with the introduction of the TaskBar in Windows 7:

• preview of open "documents" by holding the mouse over the TaskBar button
• 3 different list (most-common, etc) seen when right-clicking the TaskBar button
• ability to pin documents to the TaskBar
• ability to add items to the TaskBar context (right-click) menu
• progress for long operations is displayed visually in the taskbar button

HexEdit takes advantage of the new TaskBar including

1. showing progress for operations (handy when HexEdit is in the background)
2. Recently opened "document" list
3. Most commonly opened "document" list
4. Favorite "documents"  list
5. Quickly add to the favorites list in HexEdit using Favorites toolbar button
• Favorites button adds the current file to Favorites category
• all files in Favorites category are added to TaskBar Favorites list
6. Preview of all open documents

There was one large difficulty with adding to the TaskBar "document" lists - the type of the document (based on the file extension) must be registered as openable with HexEdit.  This is OK for normal applications - eg MSWORD would be registered to open .DOC and .DOCX files - but for a hex editor you need to be able to open any type of file.  Moreover, to register a file type as openable by some program you need administrator privileges.  All this makes it difficult for HexEdit to maintain the TaskBar "document" lists.

To get around these problems HexEdit needs to fire up a separate program in admin mode to make the registry changes, which requires user interaction - ie the user has to click the UAC "Allow" button.

Ultimate CRC Tool

I started using the Boost (C++) library is HexEdit many years ago for many things but one really cool things is its support for calculating CRCs. You can adjust all the CRC parameters to effectively do any CRC calculation that has, or ever will be invented. Moreover, you can bake the parameters in a compile time (using templates) for best performance or you can dynamically adjust the CRC parameters at run-time.

HexEdit makes use of both of these abilities of the Boost CRC code:

1. Fast calculations of common CRCs (CRC16, CRC32, CRC-CCITT etc)

• my tests shows these have better performance than any other C/C++ CRC functions
• like all commands on Operations menu works with any size selection (yes > 4GBytes)

2. General CRC calculator

• adjust all CRC parameters at run-time
• slower than "built-in" CRC commands
• save/recall settings
• or recall standard CRC32 etc

Crypto++

HexEdit has always had strong support for encryption - eg you can encrypt/decrypt with any algorithm installed in the Windows Crypto API. We also added support for the MD5 digest (cryptographic checksum) in 2006.

HexEdit Pro 4.0 also supported SHA (now sometimes called SHA-1) using a beta version of the Boost library. Unfortunately, the Boost code was removed for legal reasons so I have converted HexEdit to use the excellent Crypto++ library.

Currently Crypto++ is used for MD5, SHA-1, SHA-2 and SHA-3 digests, base 32 and base 64 encoding and decoding. The source code is also set up to easily add support for encryption algorithms of Crypto++ (such as the new ECC encryption). Hopefully, many or all of the algorithms will be supported in HexEdit 5.1.

PIT file template

My Phone

I bought a Samsung Galaxy Note Tab (GT-N7000) about 4 or 5 years ago and I love it. I especially like the screen size (and resolution) as it makes it easy to read PDF files on the train. Some people think it's too big for a pocket but on the contrary I have always had problems with narrower phones falling out of my pocket when I run for the train. The N7000 is my dream phone.

However, it has been getting a bit slow so I thought I should do a factory reset. "Why not upgrade to a later version of Android at the same time" I thought. (Samsung never upgraded it past Jellybean but there are unofficial ROMs for Lollipop and even Marshmallow.) So I installed a version of Lollipop (from CyanogenMod). This worked fine except that afterwards I did not seem to have any space on my "internal SD card" and I quickly ran out of space on my /data partition. (Samsung stupidly only make the Android /data partition 2Gb!!?)  This basically was wasting about 11Gb of the 16Gb in the phone.

I tried lots of things, including buying a new (64Gb) SD card but I could not manage to free enough space on my /data partition to install all the apps I need (let alone music and photos), or move them to the SD card. I looked at modifying the Linux partition table myself but then I found that Samsung have a "leaked" utility to change (among other things) the partition table.  This utility is called Odin - but there is also a superior open-source alternative to Odin called Heimdall.

PIT Files

To use Odin/Hemdall to modify partitions you need a PIT (partition information table) file. I managed to generate my current PIT file but had trouble finding out how to modify if to enlarge the /data partition (called "DATAFS" in the PIT file). Now someone has even written a utility called "PIT Magic" to modify PIT files. The trouble is that I could not work out how to use it and there were some obvious mistakes (such as the fact for each table entry it calls the partition starting block the "block size" field whereas it is obviously the number of the first block of the partition not the size.

Anyway, to get to the point I created a template for HexEdit that I found useful for viewing and editing PIT files. I used the source code for Heimdall to work out what some of the fields in PIT files are for. Though I am not sure of the purpose of all the fields, the important ones are the start_block and num_blocks fields which you can modify to adjust the size of the partitions.  (I would not try adding, deleting or renaming partitions using this template as I am not sure how dependent Android is on these things.)

HexEdit PIT File Template

The advantages of the HexEdit template over the PIT magic program are:

1. Quick to create and modify. (It only took me < 2 hours to create it.)
2. Much easier to understand what's happening.
3. Easier to see and modify the partiton fields in the template.
4. Easy to see what bytes are modified in the HexEdit main window.
5. Warns if any of the partitions overlap. (This can avoid a very nasty situation.)

Here is the code for the template:

<?xml version="1.0"?>
<!DOCTYPE binary_file_format SYSTEM "BinaryFileFormat.dtd">
<binary_file_format name="PIT" type_name="struct PIT" comment="Partition Info Table"
                    expr="({this.NumEntries} Entries)">
   <data type="int" name="MagicNumber" format="unsigned" len="4" domain="this == 0x12349876" display="hex"/>
   <data type="int" name="NumEntries" format="unsigned" len="4"/>
   <data type="none" len="20"/>
   <eval expr="curr_entry = 0" display_error="false" display_result="false" comment="init loop var"/>
   <for name="Entry" count="NumEntries">
      <struct type_name="struct Entry" comment="one partition in the table"
              expr="{this.name} {this.num_blocks; dec} blocks" name="">
        <data type="int" format="unsigned" len="4" comment="Binary Type?"/>
        <data type="int" name="device_type" format="unsigned" len="4" domain="{FAT=1,MMC,}" bits="2"
              direction="up"/>
        <data type="int" name="id" format="unsigned" len="4" display=""/>
        <data type="int" name="attr" format="unsigned" len="4" domain="{RO, RW, }" bits="1" direction="up"/>
        <data type="int" name="update_attr" format="unsigned" len="4" display="hex"/>
        <data type="int" name="start_block" format="unsigned" len="4" display="dec"/>
        <data type="int" name="num_blocks" format="unsigned" len="4" display="dec"/>
        <data type="int" format="unsigned" len="4" comment="File Offset?"/>
        <data type="int" format="unsigned" len="4" comment="File Size?"/>
        <data type="string" name="name" len="32"/>
        <data type="string" name="flash_file_name" len="32"/>
        <data type="string" name="fota_file_name" len="32"/>
        <if test="curr_entry &gt; 0 &amp;&amp; num_blocks &gt; 0">
           <eval expr="start_block <= Entry[curr_entry - 1].start_block + Entry[curr_entry - 1].num_blocks"
                 display_error="true" comment="check that this partn starts after end of previous"/>
        </if>
        <eval expr="++curr_entry" comment="inc loop var"/>
      </struct>
   </for>
</binary_file_format>

Paste this text into an XML file (eg _PIT.XML) and put it in your HexEdit app-data folder (eg C:\Users\andrew\AppData\Roaming\ECSoftware\HexEdit). Then open a PIT file in HexEdit and then open the template (Template -> File Type -> PIT file). In the Template view on the left there is a
grid-tree, so right-click the column header and turn on the Data column. (You might like to turn on others columns like Hex Address, Size, Comments, etc.)

Now expand the tree view (PIT) then expand the table (Entries). The 11th table entry (DATAFS) is for the /data partition, so expand that and inspect the start_block and num_blocks fields. If you want to increase the size of the partition then double-click the Data column for num_blocks and enter a larger number. Of course, you must adjust the starting block and number of blocks of the following partition (UMS) correspondingly.

I used this to modify the partition table on my phone and now I have a 8GB /data partition rather than the pathetic (default) 2GB. (I could have made increased it to 12GB but did not want to be greedy.) I have installed 41 extra apps on the phone and still have 5.21GB free.

VS2012 Quick Review

I recently had quite a few donations for HexEdit so I could afford to splash out and buy Visual Studio 2012 (aka VS2012 or VS11). I fully expected indispensable features to be removed or poorly reimplemented as I had experienced this time and time again when upgrading VS (see the list below of some major things). However, I had heard good reviews and was living in hope.


Good Things

I will start with the good things. First, VS2012 does not crash or run really slowly -- problems I have had after upgrading in the past. (But note that I am running Update 2.) It does seem to have a plethora of useful features, of which the ones I have tried worked well.

Many people have commented about the horrible colour scheme but, though I prefer the colours from any of the previous versions of VS, it's not a big deal. However, the new toolbar buttons (almost colourless) do make it hard to find the right command among the many toolbars/menus.

Now to the bad things...


Customization

One thing I noticed straight away is the painful toolbar/menu customization. Admittedly, this is the same as VS2010, but it is an absolute pain and I was hoping it had been improved.

The first thing I always do when I install a new version of VS, or install on a different system (and I use multiple versions of VS at different sites and on many different systems/VMWare images) is to create a toolbar of about 30 buttons that I commonly use then copy+drag them to the menu bar. (I create the toolbar first, as a backup, in case the menubar config stuffs up as it is prone to.) Then I can close all toolbars and have more room for code (and the multitude of VS tool windows). I then move all the tool windows around to where I want, as the default docking positions are awful and do not make use of multiple monitors.

This process takes 5 to 10 minutes in earlier versions (back as far as VC++4.X) by simply dragging (or copy+dragging) buttons from the Customize dialog or other toolbars/menus. In VS2012 it can take an hour or more to get things set up correctly. So it now takes at least 10 times longer than it took in 1995 - that is not progress!!

Possibly to compensate for this problem, VS now allows you to save your customizations (see Tools/Import and Export Settings). That way if your customizations get lost (as they do) or you want to install VS on a different system then you don't have to go through the same tedious process all over again. However, the settings export facility has some nasty surprises too (see below).

Last week, I bit the bullet and used the VS 2012 Customize dialog to put about 30 items on the menu bar. I then positioned all major tool windows (both in build and debug modes). Then I saved my settings (Menu and Command Bar Customizations + Windows Layouts). I probably spent about 2 hours on this, getting everything just right.

A few days later I somehow managed to open two copies of VS2012, which lost all my customizations! Well at least I had my settings backed up. But when I tried to restore my saved settings I got this message:

Your settings were imported, but there are some warnings. 

Warning 1: Category 'Window Layouts' ({eb4ba109-a9db-4445-bd09-e7604bcdce8A}) could not be migrated because the author of the category did not provide support for migration.

None of my customizations were restored!!! I have now gone back to using VS2008, as I don't feel inclined to repeat the above dispiriting procedure.

In summary the customization problems, when compared to earlier versions of VS2012, are:

• menu/toolbar customization is slow and tedious compared to simply dragging
• are given no information on the commands (apart from their name)
• the lists in the Customization dialog suffer from the same horrible UI design bug as the Bookmarks window of not having an "append" slot (see the VS Bookmarks section in my blog post: Asymmetric Bounds)
• it's generally confusing and counter-intuitive compared to old system
• no way to backup menus/toolbars by dragging them onto a different toolbar

Keystroke Macros

Every editor I have used in almost 30 years has provided some sort of facility to record and playback keystrokes. This included PMate from Phoenix Technologies (1984), Brief (1985 - 1988), Emacs (1988 - 1991), Epsilon (1991 - 1996), VS IDE editor (1996 - present). So it was with great surprise that I discovered that VS2012 did not have this ability.

I normally map Record and Playback to F7 and F8, as I use them so often. When I tried to do this in VS2012 I spent ages looking for the commands to map. To me it was unfathomable that they had left this out so I just kept looking. Eventually a Google search revealed that many others had discovered the same horrible truth.

One justification for the omission (from someone at Microsoft) was that less than 1% of people use it. The problem with this logic is:

• to the 1% who do use it, it's essential
• for those who don't, ways should be found to encourage them to make use of such a time-saving facility
• I could find dozens of features that are used by less than 0.1% - why not drop those instead?
• I actually find it very unlikely that 1% is anywhere near correct

Previous Upgrade Disasters

Visual Studio has a bad history of removing or castrating good features when moving to a new release. (MS seems to be obsessed with rewriting it every few years.) Here are a few examples.

1. In VC++5 the instant help (ie, the context help you get when you press F1 on a keyword or function name) was upgraded to use HTML help files. In 4.X you could get instantaneous help on a C standard library function, etc. In VC++5 this now took a very noticeable and distracting amount of time (up to 30 seconds from memory) and sometimes failed altogether.
    
2. In VS.Net (aka VS2002) MS completely rewrote the IDE which meant that a number of features simply disappeared. One example is the named bookmarks that I found absolutely invaluable in keeping track of different areas of code in large legacy C projects. In the new VS, bookmarks could not even be given a name and were only remembered if they were in files that were open when you closed the project. Since then bookmarks have come back even better but why does VS have to take 2 steps backward before taking 3 (very slow) steps forward.
    
3. I do a lot of searching (I press Ctrl+F3 on average once per minute!) and like everyone I often have a search fail for no apparent reason. This was why i really loved the search toolbar buttons that allowed you to toggle different search options (whole word, case-insensitive, regular expressions). In VC++6 and earlier I always added these buttons to the menu bar so that I could immediately see if they were on or off. This makes it so easy to check your search options and work out why your search failed.
   
   These toolbar buttons failed to make it into VS.Net. Though they have made a re-appearance in VS2012 in the little search tool that appears in the top right corner of the IDE editor window that appears when you press Ctrl+F.
    
4. The very useful XML editor disppeared in VS2008.

Conclusion

I have gone back to using VS2008 which is a shame as there are a lot of things to like about VS2012. I still sometimes use VS2012 for things like the analysers or for testing C++11 language features (like lambda functions).

I would prefer to do all my development in one environment but until toolbar/menu customization is made easier and/or settings can be easily and reliably saved and restored I am not confident using the VS2012 IDE. I also sometimes need keystroke macros so would need to use a different editor for that.

VS2012 also needs to provide an alternative set of toolbar icons with a bit more colour so that different commands are more easily distinguished and recalled.

Passwords

I received an email some time ago asking whether I could add a facility to the Hex Edit Password dialog (see Operations -> Encryption -> Password...) to check the quality of the password entered.

I have seen web sites that check for various rules and I don't believe they are useful (except for a minimum length check as already provided in HexEdit).  I will explain this and how to create good passwords.  If you haven't the time to read the whole post, at least check out the summary at the end.

Password Validation Rules

First, a good password is not just one that is hard to break; it is important that it is easy to remember and type.  Second, whatever rules the password checker imposes affects the passwords that are chosen, which can itself create an avenue of attack.  Third, most of the time these rules don't ensure good passwords anyway.

I used to use a system for creating passwords and it was always really annoying when I tried a very good password but the web site prohibited it because of some rule, like it does not contain an uppercase letter or a digit.

Good passwords can fail these tests.  Moreover bad passwords can pass them.  You may have heard the story of one O'Reilly who worked in a high security (DOD) environment and for many years just used these passwords: "O'Reilly1", "O'Reilly2", ...  These passed all the rules (mixed case, punctuation, digits, etc) but were very definitely not good passwords.

The best approach is, as always, education.  The password creator should be aware of how passwords can be broken and what are good and bad password practices.

What Are Good Passwords?

There are several things to consider.  First and foremost the password must be hard to crack.  By far the best way to ensure this is to make sure the password is long enough.  I personally use passwords of at least 30 characters.  This may seem like too much typing, but it is often easier to type several words than to type a short jumble of random characters.

You probably know the commonly cited guidelines for passwords: that they should not be a word or name (in any language), involve your personal details such as car registration number, dog's name, SSN, etc.

What you are not told is that passwords should be easy to remember.  Passwords that are not easy to remember get written down and left in convenient places.

Another good practice is to avoid typing your password while anyone is watching, especially if they are holding a camera (or phone).  If you suspect that someone saw you type it and was trying to memorize it then you should change it immediately.

Difficult Passwords

I am currently doing some work contracting for a large company.  They have a lab of dozens of test systems and someone keeps creating passwords for these systems using an English word then mixing things up by making some letters uppercase and using Leet codes (eg E => 3, I => !, etc).  Here is an example:

  PaTh3t!c

There are so many things wrong with this approach.  When I am given such a password it can often take me many times to type it in correctly even when reading it.  I have seen other people just give up thinking they have been given the wrong characters.

Not only is it hard to type but it is also hard to remember.  After a while you remember how to physically type the password but forget the actual characters.  This can be a problem if you need to write the password down for another person (and you have to write these types of passwords down to have any chance of getting them right).

That's the other problem with this sort of password - it's so hard to remember that people have to write it down.  I am always finding little bits of paper with passwords written on them.  That's not good security at all.

Enforcement

I was a UNIX system administrator for many years and the one thing I learnt about passwords is that the worst approach is to try to force people to use good ones.

One policy that most operating systems provide is to force users to change their password regularly.  I really don't see the point in changing passwords more often than every year or so, as long as the password is a good one and you do not suspect that it has been compromised (in which case you should change it straight away).

Forcing people to think of a new password every few months, or less, means they will not try to think of a very good one.  Many Windows servers (ie, domain controllers) force monthly changes - I suspect this is some sort of default that someone at Microsoft thought was a good idea.  In this situation people invariably use the same password with slight variations.  An informal survey I conducted in an office with this policy showed that 100% of people just used the same password but added an incrementing number to the end of it.  Using this strategy some had reached large numbers (ie well over one hundred) meaning they effectively had been using the same password for more than a decade.

What I Do

I have used the same method of creating passwords since I was a UNIX system administrator and became aware of their importance (and how poor most people's passwords are).  I take a line from a song.  For example:

  Money, so they say, is the root of all evil today

Of course, now that this information is public it makes it easier for you to crack my passwords!  Even so it would take an enormous amount of time to try to crack my passwords by taking different consecutive groups of words from books and songs, even if you knew what books and songs I like.

For even more security, I also try to add some changes such as replacing "to" or "too" with 2, etc.  For example:

  $, so they say, = the root of all evil 2day

An added benefit is that the password is more likely to pass the password checking rules that require digits, punctuation characters, etc.

Summary

Here are the main points:

• don't use passwords of one (or even a few) words in any language
• don't use passwords related to public knowledge about yourself
• use long passwords, if possible
• don't use passwords that are hard to remember
• don't force passwords to be changed too often (eg more than every 3 months)
• don't write passwords down where they can be seen by unauthorized persons
• don't type your password where visible to a person, camera or phone
• change your password immediately if you think it may have been compromised

Adding Windows 7 TaskBar Support

The new MFC (10) has built-in support for the Windows 7 TaskBar.  A lot of people are using Windows 7 now so I thought I better update HexEdit to take advantage of the new features.  In particular I wanted to add support for:

1. thumbnails that appear when you hold the mouse over a TaskBar button
2. for progress display when HexEdit is performing a long task
3. for the task list that is shown when you right-click the taskbar button

I describe my success with these below but first look at some general issues I first had to solve.

MFC 10 and Windows 7

The first thing you need to use the Windows 7 TaskBar is an App Id.  I just added a resource string with ID of AFX_IDS_APP_ID and MFC found it automatically.  You can actually let MFC create the App ID from other resource strings but this way I could set it to exactly what I wanted (ECSoftware.HexEdit.Free.40).

The other things you need to do to get access to all the new features of MFC is to set WINVER to 0x0601.  (It was previously 0x0501.)  By itself this will allow MFC to take advantage of Windows 7.  (I have no idea why internally Windows 7 is actually 6.1)

However, setting WINVER to a value greater than 0x501 (version number for XP) may cause some things to not work if the software is to run under Windows XP.  And I still want it to run on XP!  For example, the Windows headers make the structure NONCLIENTMETRICS 4 bytes larger for Vista and later (WINVER >= 0x0600) so I check at run-time what version of Windows is running and adjust the size of the structure before passing it to SystemParametersInfo().

Furthermore, I still want the code to build under Visual Studio 2008 (MFC 9) since not everyone has VS2010.  So some of the code needs to be conditionally-compiled using the _MFC_VER macro.  (Of course if you build HexEdit with VS2008 then you will not get the new MFC 10 features.)

Thumbnails

Some sample MFC 10 applications support thumbnails so I thought it would be easy to add.  I eventually got it working (see screen shot) but it took me a while to work out how to add this to an existing MFC application.

Eventually, I worked out that I only had to call EnableMDITabbedGroups() at startup, magically even showing multiple thumbnails if there were multiple file open in HexEdit.  Moreover, there were extra features such as options for a close button on tabs, coloured tabs, etc.  Even better the new tab system added the ability to create vertical and horizontal tab groups and easily move windows between them (like MS first added to Visual Studio 2005).

There were some problems that I needed to work out though such as getting my tab context menus to work again.  This was actually quite easy and even continued to allow the context menu to be customizable (which is actually not a feature of any of the sample applications).  Also the tip window (which shows the associated file name when you hold the mouse over a tab) worked simply by setting an option.

It was not till much later that I found a big problem.  The Next Window command no longer worked.  This command has been in HexEdit from the start and allows you to cycle through all the open windows.  It is particularly useful in keystroke macros.  Internally there was now a tab "window" at the same level as the MDI windows meaning that my Windows API call to "get next window" code failed.  This was not too hard to fix once I understood the problem.

I also noticed that the Cascade/Tile etc options on the Window menu now do not do anything at all, so I disable them if the new MDI tabs are on.  (There is an option to use the old style tabs.)

Progress

Windows 7 has a cool feature where if a program is performing a long task it can show a green progress band as a background to the taskbar button.  This allows you to go and do something else when your program is busy but keep an eye on the progress by simply checking the taskbar button.

When working with large files HexEdit can take a long time to perform some operations, so this is the perfect application for the new Windows 7 TaskBar progress feature.  To add this change was trivial since HexEdit already supports a progress bar in the status bar.  It was a simple matter to add 4 lines to the function that already handles the progress display to get the Windows 7 progress to work.

Jump List

Encouraged by how easy the previous addition was I jumped straight into adding a custom jump list.  However, I soon found that this was no easy task at all.

First I should explain that the jump list is a list of options you see when you right-click on a TaskBar button - usually this is just a list of recently opened files.  The list is retained and available even when the software is not running.  By default MFC will give you a list of recently opened files, but I wanted three sections:

• Recently Open Files
• Commonly Opened Files
• Favorites (indicated in HexEdit by adding to the Favorites category)

After a lot of reading and experimenting I could not work out how to get it to work.  There are some articles on using the new CJumpList class on MS various MS websites but they all contain mistakes or are incomplete.

None of the included MFC samples demonstrated the the CJumpList class and a search of the Internet reveal no complete samples except for the one at http://neuron.tuke.sk/~maria.vircik/apw/VS2010TrainingKit-pre%20tych%20co%20chcu%20vediet%20viac/Labs/Taskbar%20-%20MFC/Source/.  But there was no way to download the complete project for this apart from tediously extracting one file at a time.  I tried various FireFox and IE add-ons such as PimpFish but all were useless for just downloading the project files.

Eventually I managed to download all the files for the sample using the Expression Web "Import Site Wizard".

With the help of this project I believe I now know more about how jump lists work and the problems I previously encountered.  One of the problems is that for a file to be added to the jump list the software must be registered to be able to open that type of file.

Conclusion

As usual MFC makes some things really easy and some things really hard.  (Most of the time things are easy once you work out how to do them which itself can be hard.)

I have added the code for the thumbnails and progress and committed the changes to the SourceForge repository.  Unfortunately these changes were added after the release of HexEdit 4.0 but you can get them try them if you want to build HexEdit yourself.  Otherwise you will have to wait till I release HexEdit 4.1.

Clipboard

After a few people have asked for it, you can now set the default clipboard format to optionally be "hex text".  Actually there are 4 options now as explained below.  There are some advantages to the way HexEdit has always worked with the clipboard but the new options allow users to configure the program to work the way they expect or want it to work.

Moreover, copying a selection of any size to the clipboard is now supported (previously there was a limit of 16 MBytes).

History

When I first started work on HexEdit in 1997 I tried another Windows hex editor (I think it was called Hex Workshop).  When I used it to copy and paste data it had the very poor behaviour that all bytes with the value zero (ie nul bytes) were lost.  On investigation I discovered that it was copying data to the clipboard simply as text, but the Windows clipboard text format is not designed to handle binary data, which meant that some byte values were lost.

Then I discovered that you could copy and paste binary data using Visual Studio (actually it was called Developer Studio in those days).  The Visual Studio hex editor uses a special binary format that allows all byte values to be copied and pasted.  This was the obvious clipboard format to use with a binary file editor.  However, the disadvantage with the way Visual Studio worked is that you could not copy text data onto the clipboard to paste into another program such as a text editor.

HexEdit 1.0 Clipboard Formats

If you didn't know, the Windows clipboard allows you to copy the same "thing" onto the clipboard in several different formats at once.  For example, you could copy a picture as separate bitmap and vector graphics formats and even perhaps as some sort of text that describes the picture.  This allows great flexibility for programs to interact.

The obvious solution for HexEdit was that when copying data to the clipboard to copy it in two separate formats:
1. as binary data (in the format used by Visual Studio)
2. in standard text format to allow pasting into other programs

When pasting, HexEdit would use the binary data format if present, so there was no chance of any loss of data when copying and pasting within HexEdit itself.  This also meant that sharing data with the Visual Studio hex editor was simple.

If the binary data format was not present, and the text format was present, it would just paste the text data.

Text Formats

However, the above system only provided simple text support (but still much better than Visual Studio's support, which was none).

The HexEdit character area has always supported display in different character sets, so when copying to the clipboard as text it was obvious that the text should be converted from the current character set into the Windows clipboard text format (which was ASCII for Windows 95 and descendants, or Unicode for Windows NT/2K/XP/7).  For example, this allowed you to easily extract data from an EBCDIC text file and paste it into a Windows text editor or word processor which worked with ASCII or Unicode.

So HexEdit allowed you to convert when pasting into a different program but this ability was lost when copying and pasting within HexEdit itself.  That is, you could not simply copy and paste within HexEdit to convert from one character set to another since HexEdit first looks for a binary format and pastes it verbatim (ie, with no conversion).

So I added the Paste As ASCII/EBCDIC/Unicode commands to allow pasting and conversion of text.  Many users, especially those working with EBCDIC text found this very useful.

Hex Text

Another deficiency was that sometimes you want to copy the binary data as "hex text".  For example, say you have data of containing an ASCII 'A' character and an ASCII 'B' character and a nul byte.  When copied to the clipboard in Hex Edit this would have been copied as:

binary data: 0x41, 0x42, 0x00  (3 bytes)
text data:   "AB"              (2 bytes)

But sometimes you want the equivalent "hex text":

text data:  "41 42 00"         (8 bytes including spaces)

One use of this is to copy the hex values into a text editor - which some people like to do for some reason.  Another use is to copy an address from the binary data in a file as "hex text" and paste it into the Hex Jump Tool to jump to the address.

This was why I added the "Copy as Hex Text" command to HexEdit.

Paste From Hex Text

Now you would think that the above facilities would cover any possible contingency.  However, some users wanted more.  Some were used to other hex editors which took the simplistic approach of simply copying and pasting all data as "hex text".  Someone complained that since there was a "Copy as Hex Text" command there should be a corresponding "Paste From Hex Text" command.  (There is an "Import from Hex Text" command, which reads hex text from a file but no equivalent to read hex text from the clipboard.)

Apparently several other hex editors do all their clipboard interaction using "hex text".  This allows binary data to be handled without loss and also allowed pasting "hex text" into a text editor.  However, it does have a few problems:

a. It is much slower for large amounts of data due to the conversion time for both copying and pasting.
b. Uses more memory as there are at least 3 bytes stored on the clipboard for every byte of binary data.
c. You can't copy actual text data from the hex editor into a text editor, since it is always converted to "hex text".

However HexEdit Pro 4.0 provides this facility as an option.  (See Workspace/Editing page of the Options dialog.)  Actually, there are now four options:

1. As HexEdit has always done it (binary data and text data)
2. Always use "hex text"
3. If the data appears to be text then use option 1 above, else use option 2.  This allows copying of actual text data so it can be pasted into something else.
4. When the selection was made in the "character" area (to right of hex area) use option 1 above, else (if selection is in the hex area) use option 2.

I think the above options should cover just about anything, but if you have any feedback on improvements they are most welcome.

Large Selections

However, this is not the only improvement to clipboard use in HexEdit Pro 4.0.  One problem that a small number of users have encountered is when working with very large amounts of data.  The problem is that Windows limits the amount of data that can be placed on the clipboard.  (Under Windows 95 this was 16 MBytes, but under Windows 2000/XP/7 the maximum size varies depending on the amount of RAM in the system but is usually much less than 100 MBytes.)

When placing large amounts of data on the clipboard HexEdit Pro 4.0 uses a custom clipboard format, which uses a temporary file on disk to store the actual data.  All that is placed in memory on the clipboard is the name of the data file.

This facility allows copying and pasting blocks of unlimited size, limited only by available disk space in the user's temporary data area.  This has been tested with a selection of 100 GigaBytes, which is more than 1,000 times bigger than could previously be handled.  Of course, it takes a bit of time to copy this much data around on disk.

Bookmarks

If you have ever had to maintain a large, poorly designed piece of software then you know that even a trivial change can take a long time to make.  The problem is not actually making the changes to the code but finding what to do and often most of the time might be actually finding exactly where to make the change.  Personally, I have often spent hours, even days, trying to find something but once found the actual code change might only take a few minutes.  (Note that I am not actually talking about HexEdit here though I admit that the source for it can suffer from this problem as it is poorly designed in places.)

VC++ Named Bookmarks

Back when I first started using the Microsoft C++ compiler (VC++ 4.1, I think) I found the named bookmark system absolutely indispensable for keeping track of important places in the code.  I also used it to keep track of all bugs that I fixed by prefixing the bookmark name with a bug number - that way I could easily find the relevant code if I had a similar or related bug.  I liked it so much that I modelled HexEdit's bookmark system on the Visual Studio system.  (Actually I think the VC++ IDE was called Developer Studio, not Visual Studio, in those days.)

The things that made the bookmark system particularly useful were that bookmarks were persistent, project-wide and you could assign each bookmark a meaningful name.  You needed to know nothing but the name of the bookmark to be able to jump straight to the source code.  You could even jump to a bookmark in a file that was not even open and the IDE would kindly open it for you.

[The worst thing was that the bookmarks were stored in a binary format and you could easily lose all your bookmarks if the file became corrupted, which often happened.  I quickly learnt to backup the bookmarks file regularly but not being text it was not amenable to being placed under version control.]

As I said, I modelled the HexEdit bookmarks on the VC++ 6 (DevStudio) bookmarks system but there were also some improvements...

Improved Dialog

First I made the bookmarks dialog modeless so you could leave it open to always be able to see the list.  This also makes it simple to jump to a bookmark (just by double-clicking it) and it's also much easier to add a new one.

It is just stupid for a dialog that contains a list to not be resizeable, especialy as monitors are always becoming of higher resolution.  I always found the pokey little Dev Studio dialog to be annoying.

Finding Bookmarks

The HexEdit bookmarks dialog supports several columns of information which can be resized and even hidden.  Sorting on a particular column can make it easy find the bookmark you are looking for.

One thing I find annoying is when you can't remember the name of a bookmark.  HexEdit also keeps track of when each bookmark was last modified (created or moved) and when it was last accessed, or jumped to.  This information can be useful to find a bookmark you can't remember the name of.

Fixing Annoyances

It's annoying when you go to jump to a bookmark but the file it points to no longer exists on disk.  HexEdit provides a Validate button so you can check that all the bookmarks are valid.

I also got rid of the silly idea of having completely separate lists of named and unnamed bookmarks.  In HexEdit if you create an "unnamed" bookmark (eg by pressing Ctrl+F2) it adds a bookmark prefixed with "Unnamed" such as "Unnamed001".


One annoyance I encounter when using HexEdit is that bookmarks mysteriously seem to move.  (And no, it is not a bug in HexEdit).  I suspect the most common reason is that the file has been modified outside of HexEdit rendering the bookmark slightly out of place.  As a way around this I recently added a very useful feature -- if you see a bookmark is in the wrong place you can just click and drag it to the right place.

Bookmarks Tool

Before I forget I should mention the bookmarks tool.  This is a drop list on the standard toolbar which shows all the bookmarks in the active file.  As you move around in a file the tool updates to shows the closest bookmark above the cursor.  What I find really convenient is the ability to select a bookmark from the list to quickly jump to it.

Visual Studio Bookmarks

When VS.Net (aka Visual Studio 2002) was released Microsoft rewrote the IDE and accidentally or intentionally removed almost all support for bookmarks.  This was a major step backward from VC++6 so obviously I wrote an email of complaint.  I also made a list of suggestions in the same or another email including most of the facilities in HexEdit:

• modeless, resizeable, dockable bookmarks window
• sortable list with multiple columns
• Ctrl+F2 (unnamed bookmark) should just create a special name and add it to the list of named bookmarks
• bookmark categories

I don't know if anyone even looked at my emails but since then Visual Studio has added all these features and a few more.  Though the features are good the implementation sucks but discussion of that may have to wait for another post.