Tuesday 4 October 2016

HexEdit 5.0

I haven't released a new version of HexEdit for 5 years!! But I still keep getting emails about it, even feature requests.It's time for a new version - 5.0!

In the interim, between other things, I have found time to add a few new things (mainly for my own use) such as:
  • made the latest version open-source - now hosted on GitHub
  • finally implemented a more advanced file compare facility
  • added ability to preview graphics file from memory (easy using FreeImage lib)
  • lots of support for Windows 7 taskbar (see below)
  • started using excellent Crypto++ library for digests/encryption
  • fixed calculator annoyances (+ moved to MPIR inf. precision library)
  • improved performance of some operations using SIMD instructions (SSE2)
  • lots of other minor fixes
Open-source

I first released HexEdit as open-source software on 1/1/1999 and it was immensely popular within a year (downloaded more than 1 million times just from download.com by June 2000). It was ranked highest in Google for "Windows hex editor" and "binary file editor" and third for "hex editor" at that time.

In 2001 I released a shareware version (later called HexEdit Pro). I only did this because many users wanted to pay for it and I thought it might encourage me to continue work on it.

The shareware version was quite popular over a decade ago - I was selling about one copy per day for a while.  However, over the years sales dwindled.  Worse was that the Google ranking dropped away rapidly.  It seems that Google rankings, for whatever reason, favor free things - even though I always provided free open-source version which was the same as the shareware version from a few years earlier (in an SVN repo on SourceForge).

“... requires
Visual Studio
2010”
I retired the shareware version in 2012. The complete latest source of the latest version is now in a Git repo on GitHub. Note that the latest code requires Visual Studio 2010 or later since it makes use of SSE2 intrinsics, which were only added to the C++ compiler in VS2010.

File Compare

The original version of HexEdit had a basic file compare feature which allowed you to compare 2 files (or different parts of the same file) simply by opening 2 windows and pressing Alt+C. Additionally, if the "highlighter" was on this would highlight the differences between the files while you continued to press Alt+C.

However, this was facility was limited. For example, you could not get an overview or even a count of the differences. Most importantly it could not handle insertions/deletions.
“[old system] couldn't handle
  insertions/deletions”

Over the years I tried out different ways to do file compares and was never happy with the results. I even tried other hex editors to see how they did it. (I usually avoid trying other hex editors since they tend to taint my thinking.) I have yet to see a completely satisfactory file compare facility in a hex editor but have finally settled on an implementation for HexEdit 5.0.

Noteable features include:
  • allows control of how searching continues once an insertion/deletion is found
  • very fast (at least without insertions/deletions enabled) by using SSE2
  • separate compare list "tool" window listng all areas the same/different/ins/del
  • self-compare feature that allows you to monitor changes to a file over time

Preview view

A few years ago a user asked for the ability to preview bitmap files in HexEdit. Using the FreeImage.DLL (see FreeImage Library) I found this feature very easy to add to HexEdit. (It is displayed in another view window like templates or aerial view - either in a split window or a full-size tabbed window.)

The "preview" view shows the graphics file data as it is in memory, so you can play around with file data and immediately see the changes you have made. This is enormously powerful when combined with a template for the file format you are using.

Moreover the HexEdit preview window (just like the Windows photo previewer and other photo viewers) provides:
  • zoom
  • drag to scroll
  • full support for transparency
  • different background options for transparency (eg checkboard)
  • supports many graphics file formats (see FreeImage Features)

Actually it is better than the Windows photo previewer as it supports many more image file formats (not just bitmap formats) and has more background options.  (Though it does not yet have rotate/flip options.)

Windows 7 TaskBar 

There are lots of useful things that appeared in Windows with the introduction of the TaskBar in Windows 7:
  • preview of open "documents" by holding the mouse over the TaskBar button
  • 3 different list (most-common, etc) seen when right-clicking the TaskBar button
  • ability to pin documents to the TaskBar
  • ability to add items to the TaskBar context (right-click) menu
  • progress for long operations is displayed visually in the taskbar button
HexEdit takes advantage of the new TaskBar including
  1. showing progress for operations (handy when HexEdit is in the background)
  2. Recently opened "document" list
  3. Most commonly opened "document" list
  4. Favorite "documents"  list
  5. Quickly add to the favorites list in HexEdit using Favorites toolbar button
    • Favorites button adds the current file to Favorites category
    • all files in Favorites category are added to TaskBar Favorites list
  6. Preview of all open documents

There was one large difficulty with adding to the TaskBar "document" lists - the type of the document (based on the file extension) must be registered as openable with HexEdit.  This is OK for normal applications - eg MSWORD would be registered to open .DOC and .DOCX files - but for a hex editor you need to be able to open any type of file.  Moreover, to register a file type as openable by some program you need administrator privileges.  All this makes it difficult for HexEdit to maintain the TaskBar "document" lists.

To get around these problems HexEdit needs to fire up a separate program in admin mode to make the registry changes, which requires user interaction - ie the user has to click the UAC "Allow" button.

Ultimate CRC Tool

I started using the Boost (C++) library is HexEdit many years ago for many things but one really cool things is its support for calculating CRCs. You can adjust all the CRC parameters to effectively do any CRC calculation that has, or ever will be invented. Moreover, you can bake the parameters in a compile time (using templates) for best performance or you can dynamically adjust the CRC parameters at run-time.

HexEdit makes use of both of these abilities of the Boost CRC code:

1. Fast calculations of common CRCs (CRC16, CRC32, CRC-CCITT etc)
  • my tests shows these have better performance than any other C/C++ CRC functions
  • like all commands on Operations menu works with any size selection (yes > 4GBytes)
2. General CRC calculator
  • adjust all CRC parameters at run-time
  • slower than "built-in" CRC commands
  • save/recall settings
  • or recall standard CRC32 etc


Crypto++

HexEdit has always had strong support for encryption - eg you can encrypt/decrypt with any algorithm installed in the Windows Crypto API. We also added support for the MD5 digest (cryptographic checksum) in 2006.

HexEdit Pro 4.0 also supported SHA (now sometimes called SHA-1) using a beta version of the Boost library. Unfortunately, the Boost code was removed for legal reasons so I have converted HexEdit to use the excellent Crypto++ library.

Currently Crypto++ is used for MD5, SHA-1, SHA-2 and SHA-3 digests, base 32 and base 64 encoding and decoding. The source code is also set up to easily add support for encryption algorithms of Crypto++ (such as the new ECC encryption). Hopefully, many or all of the algorithms will be supported in HexEdit 5.1.




Wednesday 9 March 2016

PIT file template

My Phone

I bought a Samsung Galaxy Note Tab (GT-N7000) about 4 or 5 years ago and I love it. I especially like the screen size (and resolution) as it makes it easy to read PDF files on the train. Some people think it's too big for a pocket but on the contrary I have always had problems with narrower phones falling out of my pocket when I run for the train. The N7000 is my dream phone.

However, it has been getting a bit slow so I thought I should do a factory reset. "Why not upgrade to a later version of Android at the same time" I thought. (Samsung never upgraded it past Jellybean but there are unofficial ROMs for Lollipop and even Marshmallow.) So I installed a version of Lollipop (from CyanogenMod). This worked fine except that afterwards I did not seem to have any space on my "internal SD card" and I quickly ran out of space on my /data partition. (Samsung stupidly only make the Android /data partition 2Gb!!?)  This basically was wasting about 11Gb of the 16Gb in the phone.

I tried lots of things, including buying a new (64Gb) SD card but I could not manage to free enough space on my /data partition to install all the apps I need (let alone music and photos), or move them to the SD card. I looked at modifying the Linux partition table myself but then I found that Samsung have a "leaked" utility to change (among other things) the partition table.  This utility is called Odin - but there is also a superior open-source alternative to Odin called Heimdall.

PIT Files

To use Odin/Hemdall to modify partitions you need a PIT (partition information table) file. I managed to generate my current PIT file but had trouble finding out how to modify if to enlarge the /data partition (called "DATAFS" in the PIT file). Now someone has even written a utility called "PIT Magic" to modify PIT files. The trouble is that I could not work out how to use it and there were some obvious mistakes (such as the fact for each table entry it calls the partition starting block the "block size" field whereas it is obviously the number of the first block of the partition not the size.

Anyway, to get to the point I created a template for HexEdit that I found useful for viewing and editing PIT files. I used the source code for Heimdall to work out what some of the fields in PIT files are for. Though I am not sure of the purpose of all the fields, the important ones are the start_block and num_blocks fields which you can modify to adjust the size of the partitions.  (I would not try adding, deleting or renaming partitions using this template as I am not sure how dependent Android is on these things.)

HexEdit PIT File Template

The advantages of the HexEdit template over the PIT magic program are:

1. Quick to create and modify. (It only took me < 2 hours to create it.)
2. Much easier to understand what's happening.
3. Easier to see and modify the partiton fields in the template.
4. Easy to see what bytes are modified in the HexEdit main window.
5. Warns if any of the partitions overlap. (This can avoid a very nasty situation.)

Here is the code for the template:
<?xml version="1.0"?>
<!DOCTYPE binary_file_format SYSTEM "BinaryFileFormat.dtd">
<binary_file_format name="PIT" type_name="struct PIT" comment="Partition Info Table"
                    expr="({this.NumEntries} Entries)">
   <data type="int" name="MagicNumber" format="unsigned" len="4" domain="this == 0x12349876" display="hex"/>
   <data type="int" name="NumEntries" format="unsigned" len="4"/>
   <data type="none" len="20"/>
   <eval expr="curr_entry = 0" display_error="false" display_result="false" comment="init loop var"/>
   <for name="Entry" count="NumEntries">
      <struct type_name="struct Entry" comment="one partition in the table"
              expr="{this.name} {this.num_blocks; dec} blocks" name="">
        <data type="int" format="unsigned" len="4" comment="Binary Type?"/>
        <data type="int" name="device_type" format="unsigned" len="4" domain="{FAT=1,MMC,}" bits="2"
              direction="up"/>
        <data type="int" name="id" format="unsigned" len="4" display=""/>
        <data type="int" name="attr" format="unsigned" len="4" domain="{RO, RW, }" bits="1" direction="up"/>
        <data type="int" name="update_attr" format="unsigned" len="4" display="hex"/>
        <data type="int" name="start_block" format="unsigned" len="4" display="dec"/>
        <data type="int" name="num_blocks" format="unsigned" len="4" display="dec"/>
        <data type="int" format="unsigned" len="4" comment="File Offset?"/>
        <data type="int" format="unsigned" len="4" comment="File Size?"/>
        <data type="string" name="name" len="32"/>
        <data type="string" name="flash_file_name" len="32"/>
        <data type="string" name="fota_file_name" len="32"/>
        <if test="curr_entry &gt; 0 &amp;&amp; num_blocks &gt; 0">
           <eval expr="start_block <= Entry[curr_entry - 1].start_block + Entry[curr_entry - 1].num_blocks"
                 display_error="true" comment="check that this partn starts after end of previous"/>
        </if>
        <eval expr="++curr_entry" comment="inc loop var"/>
      </struct>
   </for>
</binary_file_format>

Paste this text into an XML file (eg _PIT.XML) and put it in your HexEdit app-data folder (eg C:\Users\andrew\AppData\Roaming\ECSoftware\HexEdit). Then open a PIT file in HexEdit and then open the template (Template -> File Type -> PIT file). In the Template view on the left there is a
grid-tree, so right-click the column header and turn on the Data column. (You might like to turn on others columns like Hex Address, Size, Comments, etc.)

Now expand the tree view (PIT) then expand the table (Entries). The 11th table entry (DATAFS) is for the /data partition, so expand that and inspect the start_block and num_blocks fields. If you want to increase the size of the partition then double-click the Data column for num_blocks and enter a larger number. Of course, you must adjust the starting block and number of blocks of the following partition (UMS) correspondingly.

I used this to modify the partition table on my phone and now I have a 8GB /data partition rather than the pathetic (default) 2GB. (I could have made increased it to 12GB but did not want to be greedy.) I have installed 41 extra apps on the phone and still have 5.21GB free.