Friday, June 29, 2012

Using hashdog

Now that you have successfully installed hashdog it is time to start generating some databases that we can start using in our forensic investigations. Hashdog uses the option ‘--input’ for specifying the file or the directory to recursively generate hashes for.  As I mentioned in my previous blog post, hashdog uses 7-Zip for extracting archives. The program will actually check all the files it processes if the file in question can be extracted. If that is the case, hashdog will extract the file and generate hashes for the content of the archive it just extracted as well. It will continue to do so until there are no more files to extract in the archive and then move on to the next file in the directory that is being processed. To illustrate how this is being done, I am going to take you through an example were we build our own hash database.


Generating your own hash database
In this example I have three file that I want to generate a database with MD5 hashes for. That is done by using the switch ‘--md5-file’ and specifying the path to the file were we want to store the results in, our hash database. As input to hashdog am I using a couple of files I have downloaded from the Internet, namely the installer for Firefox version 13.0.1 and Red Drive from JSCAPE. I also copied the ‘ls’ binary from my Debian system to the directory that I am going to process.


pmedina@forensic:~/hashdog$ ls -l /files/
total 17452
-rw-r--r-- 1 pmedina pmedina 16577248 Jun 26 07:47 Firefox Setup 13.0.1.exe
-rwxr-xr-x 1 root    root      108008 Jun 28 09:57 ls
-rw-r--r-- 1 pmedina pmedina  1148549 Oct 24  2011 RedDrive.zip
pmedina@forensic:~/hashdog$ ./hashdog.pl --input /files --md5sum-file /tmp/example.txt
pmedina@forensic:~/hashdog$ ./hashdog.pl --input /files --md5sum-file /tmp/example.txt
[*] hashdog.pl version: 0.72 written by Par Osterberg Medina
[-] minimum filesize to process: 1 bytes
[-] archive binary: 7-Zip [64] 9.20  Copyright (c) 1999-2010 Igor Pavlov  2010-11-18
[-] using tmp folder: /tmp/hd830002
[+] processing files recursivly from: /files
[+] RedDrive.zip
[-] extracting archive: zip
[+] RedDrive.zip/install.msi
[-] extracting archive: Compound
[+] RedDrive.zip/install.msi/!_Columns
[+] RedDrive.zip/install.msi/!EventMapping
[+] RedDrive.zip/install.msi/Binary.InstallUtil
[-] extracting archive: PE
[+] RedDrive.zip/install.msi/Binary.InstallUtil/.rsrc/VERSION/1
[+] RedDrive.zip/install.msi/Binary.InstallUtil/.data
[+] RedDrive.zip/install.msi/Binary.InstallUtil/.reloc
[+] RedDrive.zip/install.msi/Binary.InstallUtil/.text
[+] RedDrive.zip/install.msi/[5]SummaryInformation
[+] RedDrive.zip/install.msi/!Media
[+] RedDrive.zip/install.msi/Binary._70B6BD6470D90F593F71019EF5DC9D42
..
..


The complete output of the command above is too large to include here but I will start looking at the beginning of the command execution and explain what happens. The first file that we process and generate a checksum for is the ‘RedDrive.zip’ file itself. That file is identified by 7-Zip as a zip archive and is therefore extracted. The zip archive contains one file called ‘install.msi’ that is also identified as an archive. After calculating the MD5 checksum for the ‘install.msi' file the content of the .msi archive is also extracted. Within the .msi archive there are a bunch of files that hashdog generate checksums for and extract if possible. 


Archive types to skip
As you can see in the command output above 7-Zip will also try to extract the resources sections contained in a Portable Executable (PE) file, something we might not always want to do. The same behavior is also observed for the Linux binary ‘ls’ that I previously copied to the directory we are processing.


[+] ls
[-] extracting archive: ELF
[+] ls/4
[+] ls/6
[+] ls/5
[+] ls/0
[+] ls/2
[+] ls/3
[+] ls/1


In most cases we do not want to extract the content of ELF binaries and PE files and by using the switch ‘--archive-skip’ we can specify which archive types we want to exclude from extraction. The option ‘--archive-skip’ takes a case insensitive comma separated list of archives that should not be processed and in our case we want to specify the archives PE and ELF.


pmedina@forensic:~/hashdog$ wc -l /tmp/example.txt
2644 /tmp/example.txt
pmedina@forensic:~/hashdog$ ./hashdog.pl --input /files --md5sum-file /tmp/example.txt --archive-skip=PE,ELF
[*] hashdog.pl version: 0.72 written by Par Osterberg Medina
[-] minimum filesize to process: 1 bytes
[-] archive binary: 7-Zip [64] 9.20  Copyright (c) 1999-2010 Igor Pavlov  2010-11-18
[-] using tmp folder: /tmp/hd950156
[+] processing files recursivly from: /files
[+] RedDrive.zip
[-] extracting archive: zip
[+] RedDrive.zip/install.msi
[-] extracting archive: Compound
..
..
[-] extracting archive: PENsis
[+] Firefox Setup 13.0.1.exe/setup.exe/\x01_1
[+] Firefox Setup 13.0.1.exe/setup.exe/\x01_6
[+] Firefox Setup 13.0.1.exe/setup.exe/\x01_2
[+] Firefox Setup 13.0.1.exe/setup.exe/\x01\x01/\x01\x1a/InstallOptions.dll
[+] Firefox Setup 13.0.1.exe/setup.exe/\x01\x01/\x01\x1a/modern-header.bmp
[+] Firefox Setup 13.0.1.exe/setup.exe/\x01\x01/\x01\x1a/ServicesHelper.dll
[+] Firefox Setup 13.0.1.exe/setup.exe/\x01\x01/\x01\x1a/ioSpecial.ini
[+] Firefox Setup 13.0.1.exe/setup.exe/\x01\x01/\x01\x1a/modern-wizard.bmp
[+] Firefox Setup 13.0.1.exe/setup.exe/\x01_5
[+] Firefox Setup 13.0.1.exe/setup.exe/\x01\x15/\x01\x1a/nsExec.dll
[+] Firefox Setup 13.0.1.exe/setup.exe/\x01\x15/\x01\x1a/Banner.dll
[+] Firefox Setup 13.0.1.exe/setup.exe/\x01
[+] Firefox Setup 13.0.1.exe/setup.exe/\x01\x1a/System.dll
[+] Firefox Setup 13.0.1.exe/setup.exe/\x01_3
[+] Firefox Setup 13.0.1.exe/setup.exe/\x01_4
[+] done, finished in: 0 hours, 0 minutes and 16 seconds
[-] deleting the tmp directory
pmedina@forensic:~/hashdog$ wc -l /tmp/example.txt
2212 /tmp/example.txt


Our database has now 432 less entries than it did before we added the ‘--archive-skip’ option.

Full file path in the hashset
Opening the file that we created and looking at the result of the command that was run above we see that the file has two columns, the MD5 checksum and the name of the file.


pmedina@forensic:~/hashdog$ head /tmp/example.txt
2e54d3fb64ff68607c38ecb482f5fa25  RedDrive.zip
70e5118a1d0cff1a59b820a919b15949  install.msi
a6ae9f9e02477526bbac1e97357141be  !_Columns
a76026797c61d04c1c9990366e48208e  !EventMapping
4b6f4f52de80f1a7890c9bd0a7cac5e3  Binary.InstallUtil
45d11bc27761a502bce036adbcf64f7d  [5]SummaryInformation
d2dd55a6b2d6d768ab6254c169d41ce9  !Media
8806ebee0e08ab6338d0fdc87be83fc4  Binary._70B6BD6470D90F593F71019EF5DC9D42
d6b3635d8e144efae4ab1753695c19af  !Dialog
8e097d7e4ebf2f6cde863fbd7de296e6  !Feature


As you can see above, the file name in the hashset is not the full path to the file that we generated the checksum for. Having the full path to the file in the hash database will increase the size of the file a lot and as in the case with installers, the path to your file will almost never match the path to the file you have on disk. There are however some cases were you want to include to full path to the file in your database. This can be accomplished by specifying the ‘--md5sum-fullpath’ switch.


pmedina@forensic:~/hashdog$ wc -c /tmp/example.txt
113263 /tmp/example.txt
pmedina@forensic:~/hashdog$ ./hashdog.pl --input /files --md5sum-file /tmp/example.txt --archive-skip=PE,ELF --md5sum-fullpath
[*] hashdog.pl version: 0.72 written by Par Osterberg Medina
[-] minimum filesize to process: 1 bytes
[-] archive binary: 7-Zip [64] 9.20  Copyright (c) 1999-2010 Igor Pavlov  2010-11-18
[-] using tmp folder: /tmp/hd950156
[+] processing files recursivly from: /files
[+] RedDrive.zip
[-] extracting archive: zip
..
..
[+] Firefox Setup 13.0.1.exe/setup.exe/\x01\x15/\x01\x1a/Banner.dll
[+] Firefox Setup 13.0.1.exe/setup.exe/\x01
[+] Firefox Setup 13.0.1.exe/setup.exe/\x01\x1a/System.dll
[+] Firefox Setup 13.0.1.exe/setup.exe/\x01_3
[+] Firefox Setup 13.0.1.exe/setup.exe/\x01_4
[+] done, finished in: 0 hours, 0 minutes and 15 seconds
[-] deleting the tmp directory 
pmedina@forensic:~/hashdog$ wc -c /tmp/example.txt
271032 /tmp/example.txt
pmedina@forensic:~/hashdog$ head /tmp/example.txt
2e54d3fb64ff68607c38ecb482f5fa25  RedDrive.zip
70e5118a1d0cff1a59b820a919b15949  RedDrive.zip/install.msi
a6ae9f9e02477526bbac1e97357141be  RedDrive.zip/install.msi/!_Columns
a76026797c61d04c1c9990366e48208e  RedDrive.zip/install.msi/!EventMapping
4b6f4f52de80f1a7890c9bd0a7cac5e3  RedDrive.zip/install.msi/Binary.InstallUtil
45d11bc27761a502bce036adbcf64f7d  RedDrive.zip/install.msi/[5]SummaryInformation
d2dd55a6b2d6d768ab6254c169d41ce9  RedDrive.zip/install.msi/!Media
8806ebee0e08ab6338d0fdc87be83fc4  RedDrive.zip/install.msi/Binary._70B6BD6470D90F593F71019EF5DC9D42
d6b3635d8e144efae4ab1753695c19af  RedDrive.zip/install.msi/!Dialog
8e097d7e4ebf2f6cde863fbd7de296e6  RedDrive.zip/install.msi/!Feature


The size of our file increased quite a lot when we started to include the full path in our database. It got 157769 bytes larger, an increase in size by almost 42%. The question you have to ask yourself when you are generating our hash databases is - does adding the full path in the file justify the increase in file size?


Generating your own RDS file
Even though ‘hashdog’ can be used to generate hash databases in a lot of different formats, the format that will probably be of most use to us as forensic examiners and incident responders is the RDS format. This is the data format that NIST uses in their NSRL Reference Data Set (RDS) and can be imported in a lot of tools. For a deeper dive into the format, I suggest you read the PDF “Data Formats of the NSRL Reference Data Set (RDS) Distribution”. http://www.nsrl.nist.gov/Documents/Data-Formats-of-the-NSRL-Reference-Data-Set-16.pdf


As in the previous example above, all we need to if we want to generate a hash database using the RDS format, is to specify the switch ‘--rds-file’ followed by the path to the file we want to store the result in. If we want to, we can also choose to store the full path to the files in the database by using the switch ‘--rds-fullpath’. 


pmedina@forensic:~/hashdog$ ./hashdog.pl --input /files --md5sum-file /tmp/example.txt --archive-skip=PE,ELF --md5sum-fullpath --rds-file /tmp/example-rds.txt
[*] hashdog.pl version: 0.72 written by Par Osterberg Medina
[-] minimum filesize to process: 1 bytes
[-] archive binary: 7-Zip [64] 9.20  Copyright (c) 1999-2010 Igor Pavlov  2010-11-18
[-] using tmp folder: /tmp/hd950156
[+] processing files recursivly from: /files
[+] RedDrive.zip
[-] extracting archive: zip
..
..
[+] Firefox Setup 13.0.1.exe/setup.exe/\x01_3
[+] Firefox Setup 13.0.1.exe/setup.exe/\x01_4
[+] done, finished in: 0 hours, 0 minutes and 16 seconds
[-] deleting the tmp directory
pmedina@forensic:~/hashdog$ wc -l /tmp/example.txt /tmp/example-rds.txt
  2212 /tmp/example.txt
  2213 /tmp/example-rds.txt
  4425 total
pmedina@forensic:~/hashdog$ wc -c /tmp/example.txt /tmp/example-rds.txt
271032 /tmp/example.txt
274652 /tmp/example-rds.txt
545684 total
pmedina@forensic:~/hashdog$ head /tmp/example-rds.txt
"SHA-1","MD5","CRC32","FileName","FileSize","ProductCode","OpSystemCode","SpecialCode"
"d9c40dd2f1fb08927e773a0dc70d75fedd71549e","2e54d3fb64ff68607c38ecb482f5fa25","732c6df0","RedDrive.zip",1148549,0,"WIN",""
"d1fa60a19ca5095731eb78cd6f6c7e3eca2cf57c","70e5118a1d0cff1a59b820a919b15949","c224e555","install.msi",1413120,0,"WIN",""
"c9220e529ad1ca74d6d7b4a8a17529e326f617cf","a6ae9f9e02477526bbac1e97357141be","8b1ddae5","!_Columns",3328,0,"WIN",""
"86d90e616458eea7188605b9f601c6bb7b46aeaf","a76026797c61d04c1c9990366e48208e","3a0a30ce","!EventMapping",208,0,"WIN",""
"e45efe29240c68452730fc32327eb3048a162e2d","4b6f4f52de80f1a7890c9bd0a7cac5e3","fa0f9bcc","Binary.InstallUtil",55296,0,"WIN",""
"e397c2f9d0ec530a8453193aa5b15eb40c48e00f","45d11bc27761a502bce036adbcf64f7d","77ec3d28","[5]SummaryInformation",412,0,"WIN",""
"36da55c877afeb7af220c80b7021368985a1d24e","d2dd55a6b2d6d768ab6254c169d41ce9","0c1fb7c4","!Media",12,0,"WIN",""
"3690c3ca05b6ed8ef86589dc93495811b1fb49db","8806ebee0e08ab6338d0fdc87be83fc4","b9c0a320","Binary._70B6BD6470D90F593F71019EF5DC9D42",12192,0,"WIN",""
"3ccfc07833e6025ffdc0b110c45c2e2c97588efc","d6b3635d8e144efae4ab1753695c19af","abc7c53e","!Dialog",594,0,"WIN",""
pmedina@forensic:~/hashdog$


In the example above, we used the switches ‘--md5-file’ and ‘--rds-file’ to generate both a file with MD5 checksums as well as a file that is using the RDS format at the same time. Currently, hashdog can generate hashsets using the MD5 checksums, SHA-1 checksums and hashsets in the RDS format, but support for more formats might be added in the future.

Version 0.72 of hashdog released


I corrected a bug in the way archives of multiple types are handled and how file names with control characters in them are stored in the hash database. Please download and install the new version available here; https://github.com/parosterbergmedina/hashdog 

Wednesday, June 27, 2012

Installing hashdog


Yesterday I gave a presentation on the SANS DFIR 2012 conference regarding cryptographic checksums and how those can be used in computer forensic. In my talk I demonstrated a tool called ‘hashdog’ which can be used to generate databases with hashes of files. Files to either look for or discard in a forensic investigation. This blog post will take you to the steps needed to install ‘hashdog’ with its dependencies.

The first thing you need to do is to download ‘hashdog’ from its github repository which can be found here; https://github.com/parosterbergmedina/hashdog. Being written in perl, ‘hashdog’ has a few perl-modules that it depends on and before continuing we have to make sure that these modules are installed. My system is running Debian stable so all examples are from that distribution – your packages might be named differently. Verify that you have the following perl-modules installed on your system by running the ‘apt-get’ command listed below.

pmedina@forensic:~/hashdog$ sudo apt-get install perl-base perl-modules libdigest-md5-file-perl libdigest-sha-perl libstring-crc32-perl
Reading package lists... Done
Building dependency tree
Reading state information... Done
libdigest-md5-file-perl is already the newest version.
libdigest-sha-perl is already the newest version.
libstring-crc32-perl is already the newest version.
perl-base is already the newest version.
perl-modules is already the newest version.
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.

Digest::MD5::File

There is a bug in the perl module ‘Digest::MD5::File’ and the way it handles files that ends with a trailing space (0x20). Bug 76174 has been addressed with the release of version 0.08 but Debian stable still uses the old 0.07 version. If you are running Debian stable and you do not want to have any issues with these types of files, you have two options. You can either manually install version ‘0.08’ from testing or unstable, or just simply edit the ‘File.pm’ yourself and fix the problem by following the steps outlined below.

The first step is to verify that this bug affects you.

pmedina@forensic:~/hashdog$ echo "this is a test" > "/tmp/testfile "
pmedina@forensic:~/hashdog$ ls /tmp/ | grep testfile | xxd
0000000: 7465 7374 6669 6c65 200a                 testfile .
pmedina@forensic:~/hashdog$ md5sum "/tmp/testfile "
e19c1283c925b3206685ff522acfe3e6  /tmp/testfile
pmedina@forensic:~/hashdog$ perl -MDigest::MD5::File=file_md5_hex -e 'print file_md5_hex("/tmp/testfile ") ."\n"';

pmedina@forensic:~/hashdog$

If the perl command did not produce a MD5 hash, we will patch the ‘File.pm’ file and verify that our fix worked.

pmedina@forensic:~/hashdog$ cat /tmp/patch
53c53
<         open my ($fh), $file or return;
---
>       open (my $fh,"<", $file) or return;
pmedina@forensic:~/hashdog$ patch /usr/share/perl5/Digest/MD5/File.pm -i /tmp/patch -o /tmp/updated
patching file /usr/share/perl5/Digest/MD5/File.pm
pmedina@forensic:~/hashdog$ sudo mv /tmp/updated /usr/share/perl5/Digest/MD5/File.pm
pmedina@forensic:~/hashdog$ chmod +r /usr/share/perl5/Digest/MD5/File.pm
pmedina@forensic:~/hashdog$ perl -MDigest::MD5::File=file_md5_hex -e 'print file_md5_hex("/tmp/testfile ") ."\n"';
e19c1283c925b3206685ff522acfe3e6

7-Zip

To test if a file is an archive and can be unpacked, ‘hashdog’ uses the program 7-Zip. 7-Zip, which has been ported to Linux as p7zip, is currently at version 9.20 while Debian stable still uses version 9.04. Even though most of the functionality in ‘hashdog’ will still work using that old version of pkzip, support for more archives has been added in later versions so upgrading might be a good idea. To use the versions included in Debian stable, simply install the packages ‘p7zip-full’ and ‘p7zip-rar’ using ‘apt-get’. This will automatically solve any dependencies as well.

pmedina@forensic:~/hashdog$ sudo apt-get install p7zip-full p7zip-rar
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following NEW packages will be installed:
  p7zip-full p7zip-rar
0 upgraded, 2 newly installed, 0 to remove and 0 not upgraded.
Need to get 0 B/1,454 kB of archives.
After this operation, 3,867 kB of additional disk space will be used.
Selecting previously deselected package p7zip-full.
(Reading database ... 43741 files and directories currently installed.)
Unpacking p7zip-full (from .../p7zip-full_9.04~dfsg.1-1_amd64.deb) ...
Selecting previously deselected package p7zip-rar.
Unpacking p7zip-rar (from .../p7zip-rar_9.04~ds.1-1_amd64.deb) ...
Processing triggers for man-db ...
Setting up p7zip-full (9.04~dfsg.1-1) ...
Setting up p7zip-rar (9.04~ds.1-1) ...
pmedina@forensic:~/hashdog$ 7z | grep -i zip
7-Zip 9.04 beta  Copyright (c) 1999-2009 Igor Pavlov  2009-05-30
p7zip Version 9.04 (locale=en_US.UTF-8,Utf16=on,HugeFiles=on,1 CPU)
  -so: write data to stdout (eg: 7z a dummy -tgzip -so Doc.txt > archive.gz)

If you want to use version 9.20 of p7zip, the packages needs to be downloaded and installed manually. My forensic workstation is running Debian for the AMD64 platform so the packages I will be downloading and installing are compiled for that specific platform. Do a search for ‘p7zip’ and ‘p7zip-rar’ on http://packages.debian.org if you want to find packages for a different version.

pmedina@forensic:~/hashdog$ wget --no-verbose http://http.us.debian.org/debian/pool/non-free/p/p7zip-rar/p7zip-rar_9.20.1~ds.1-3_amd64.deb
2012-06-27 20:33:21 URL:http://http.us.debian.org/debian/pool/non-free/p/p7zip-rar/p7zip-rar_9.20.1~ds.1-3_amd64.deb [57870/57870] -> "p7zip-rar_9.20.1~ds.1-3_amd64.deb" [1]
pmedina@forensic:~/hashdog$ md5sum *.deb
deb71878cd75b0e7d70f1b62e08a77c7  p7zip-full_9.20.1~dfsg.1-4_amd64.deb
85ad60c37ebfc2bf8d7dc4cc7d26e6cc  p7zip-rar_9.20.1~ds.1-3_amd64.deb
pmedina@forensic:~/hashdog$ sudo dpkg --install p7zip-full_9.20.1~dfsg.1-4_amd64.deb p7zip-rar_9.20.1~ds.1-3_amd64.deb
(Reading database ... 43811 files and directories currently installed.)
Preparing to replace p7zip-full 9.04~dfsg.1-1 (using p7zip-full_9.20.1~dfsg.1-4_amd64.deb) ...
Unpacking replacement p7zip-full ...
Preparing to replace p7zip-rar 9.04~ds.1-1 (using p7zip-rar_9.20.1~ds.1-3_amd64.deb) ...
Unpacking replacement p7zip-rar ...
Setting up p7zip-full (9.20.1~dfsg.1-4) ...
Processing triggers for man-db ...
Setting up p7zip-rar (9.20.1~ds.1-3) ...
pmedina@forensic:~/hashdog$ 7z | grep -i zip
7-Zip [64] 9.20  Copyright (c) 1999-2010 Igor Pavlov  2010-11-18
p7zip Version 9.20 (locale=en_US.UTF-8,Utf16=on,HugeFiles=on,1 CPU)

Hashdog

Now all the dependencies of ‘hashdog’ should be taken care of and you should be able to start using the program without any issues. To verify this, simply execute the program.

pmedina@forensic:~/hashdog$ ./hashdog.pl
[*] hashdog.pl version: 0.71 written by Par Osterberg Medina
specify a file or directory to process with '--input'
usage: hashdog.pl [--input] [--md5sum-file|--sha1sum-file|--rds-file] {options}

        --input|-i {file/dir}    file or directory to process
        --md5sum-file {file}     generate a file with md5 checksums
        --md5sum-fullpath        use full file paths in the file with md5 checksums
        --sha1sum-file {file}    generate a file with sha-1 checksums
        --sha1sum-fullpath       use full file paths in the file with sha-1 checksums
        --rds-file {file}        generate a file with checksums using the RDS format
        --rds-fullpath           use full file paths in the RDS file
        --archive-bin {file}     path to the 7-Zip binary
        --archive-skip {list}    comma sperated list of archive types to not expand
        --min-filesize {number}  minimun filesize in bytes to process
        --tmp                    specify the the tmp folder to use
        --verbose|-v             verbose output
        --debug|-d               turn on debug output
pmedina@forensic:~/hashdog$