Sunday 27 November 2011

Clipboard

After a few people have asked for it, you can now set the default clipboard format to optionally be "hex text".  Actually there are 4 options now as explained below.  There are some advantages to the way HexEdit has always worked with the clipboard but the new options allow users to configure the program to work the way they expect or want it to work.

Moreover, copying a selection of any size to the clipboard is now supported (previously there was a limit of 16 MBytes).

History

When I first started work on HexEdit in 1997 I tried another Windows hex editor (I think it was called Hex Workshop).  When I used it to copy and paste data it had the very poor behaviour that all bytes with the value zero (ie nul bytes) were lost.  On investigation I discovered that it was copying data to the clipboard simply as text, but the Windows clipboard text format is not designed to handle binary data, which meant that some byte values were lost.

Then I discovered that you could copy and paste binary data using Visual Studio (actually it was called Developer Studio in those days).  The Visual Studio hex editor uses a special binary format that allows all byte values to be copied and pasted.  This was the obvious clipboard format to use with a binary file editor.  However, the disadvantage with the way Visual Studio worked is that you could not copy text data onto the clipboard to paste into another program such as a text editor.

HexEdit 1.0 Clipboard Formats

If you didn't know, the Windows clipboard allows you to copy the same "thing" onto the clipboard in several different formats at once.  For example, you could copy a picture as separate bitmap and vector graphics formats and even perhaps as some sort of text that describes the picture.  This allows great flexibility for programs to interact.

The obvious solution for HexEdit was that when copying data to the clipboard to copy it in two separate formats:
1. as binary data (in the format used by Visual Studio)
2. in standard text format to allow pasting into other programs

When pasting, HexEdit would use the binary data format if present, so there was no chance of any loss of data when copying and pasting within HexEdit itself.  This also meant that sharing data with the Visual Studio hex editor was simple.

If the binary data format was not present, and the text format was present, it would just paste the text data.

Text Formats

However, the above system only provided simple text support (but still much better than Visual Studio's support, which was none).

The HexEdit character area has always supported display in different character sets, so when copying to the clipboard as text it was obvious that the text should be converted from the current character set into the Windows clipboard text format (which was ASCII for Windows 95 and descendants, or Unicode for Windows NT/2K/XP/7).  For example, this allowed you to easily extract data from an EBCDIC text file and paste it into a Windows text editor or word processor which worked with ASCII or Unicode.

So HexEdit allowed you to convert when pasting into a different program but this ability was lost when copying and pasting within HexEdit itself.  That is, you could not simply copy and paste within HexEdit to convert from one character set to another since HexEdit first looks for a binary format and pastes it verbatim (ie, with no conversion).

So I added the Paste As ASCII/EBCDIC/Unicode commands to allow pasting and conversion of text.  Many users, especially those working with EBCDIC text found this very useful.

Hex Text

Another deficiency was that sometimes you want to copy the binary data as "hex text".  For example, say you have data of containing an ASCII 'A' character and an ASCII 'B' character and a nul byte.  When copied to the clipboard in Hex Edit this would have been copied as:

binary data: 0x41, 0x42, 0x00  (3 bytes)
text data:   "AB"              (2 bytes)

But sometimes you want the equivalent "hex text":

text data:  "41 42 00"         (8 bytes including spaces)

One use of this is to copy the hex values into a text editor - which some people like to do for some reason.  Another use is to copy an address from the binary data in a file as "hex text" and paste it into the Hex Jump Tool to jump to the address.

This was why I added the "Copy as Hex Text" command to HexEdit.

Paste From Hex Text

Now you would think that the above facilities would cover any possible contingency.  However, some users wanted more.  Some were used to other hex editors which took the simplistic approach of simply copying and pasting all data as "hex text".  Someone complained that since there was a "Copy as Hex Text" command there should be a corresponding "Paste From Hex Text" command.  (There is an "Import from Hex Text" command, which reads hex text from a file but no equivalent to read hex text from the clipboard.)

Apparently several other hex editors do all their clipboard interaction using "hex text".  This allows binary data to be handled without loss and also allowed pasting "hex text" into a text editor.  However, it does have a few problems:

a. It is much slower for large amounts of data due to the conversion time for both copying and pasting.
b. Uses more memory as there are at least 3 bytes stored on the clipboard for every byte of binary data.
c. You can't copy actual text data from the hex editor into a text editor, since it is always converted to "hex text".

However HexEdit Pro 4.0 provides this facility as an option.  (See Workspace/Editing page of the Options dialog.)  Actually, there are now four options:


  1. As HexEdit has always done it (binary data and text data)
  2. Always use "hex text"
  3. If the data appears to be text then use option 1 above, else use option 2.  This allows copying of actual text data so it can be pasted into something else.
  4. When the selection was made in the "character" area (to right of hex area) use option 1 above, else (if selection is in the hex area) use option 2.

I think the above options should cover just about anything, but if you have any feedback on improvements they are most welcome.

Large Selections

However, this is not the only improvement to clipboard use in HexEdit Pro 4.0.  One problem that a small number of users have encountered is when working with very large amounts of data.  The problem is that Windows limits the amount of data that can be placed on the clipboard.  (Under Windows 95 this was 16 MBytes, but under Windows 2000/XP/7 the maximum size varies depending on the amount of RAM in the system but is usually much less than 100 MBytes.)

When placing large amounts of data on the clipboard HexEdit Pro 4.0 uses a custom clipboard format, which uses a temporary file on disk to store the actual data.  All that is placed in memory on the clipboard is the name of the data file.

This facility allows copying and pasting blocks of unlimited size, limited only by available disk space in the user's temporary data area.  This has been tested with a selection of 100 GigaBytes, which is more than 1,000 times bigger than could previously be handled.  Of course, it takes a bit of time to copy this much data around on disk.