Using hashdog

Now that you have successfully installed hashdog it is time to start generating some databases that we can start using in our forensic investigations. Hashdog uses the option ‘--input’ for specifying the file or the directory to recursively generate hashes for.  As I mentioned in my previous blog post, hashdog uses 7-Zip for extracting archives. The program will actually check all the files it processes if the file in question can be extracted. If that is the case, hashdog will extract the file and generate hashes for the content of the archive it just extracted as well. It will continue to do so until there are no more files to extract in the archive and then move on to the next file in the directory that is being processed. To illustrate how this is being done, I am going to take you through an example were we build our own hash database.


Generating your own hash database
In this example I have three file that I want to generate a database with MD5 hashes for. That is done by using the switch ‘--md5-file’ and specifying the path to the file were we want to store the results in, our hash database. As input to hashdog am I using a couple of files I have downloaded from the Internet, namely the installer for Firefox version 13.0.1 and Red Drive from JSCAPE. I also copied the ‘ls’ binary from my Debian system to the directory that I am going to process.


pmedina@forensic:~/hashdog$ ls -l /files/
total 17452
-rw-r--r-- 1 pmedina pmedina 16577248 Jun 26 07:47 Firefox Setup 13.0.1.exe
-rwxr-xr-x 1 root    root      108008 Jun 28 09:57 ls
-rw-r--r-- 1 pmedina pmedina  1148549 Oct 24  2011 RedDrive.zip
pmedina@forensic:~/hashdog$ ./hashdog.pl --input /files --md5sum-file /tmp/example.txt
pmedina@forensic:~/hashdog$ ./hashdog.pl --input /files --md5sum-file /tmp/example.txt
[*] hashdog.pl version: 0.72 written by Par Osterberg Medina
[-] minimum filesize to process: 1 bytes
[-] archive binary: 7-Zip [64] 9.20  Copyright (c) 1999-2010 Igor Pavlov  2010-11-18
[-] using tmp folder: /tmp/hd830002
[+] processing files recursivly from: /files
[+] RedDrive.zip
[-] extracting archive: zip
[+] RedDrive.zip/install.msi
[-] extracting archive: Compound
[+] RedDrive.zip/install.msi/!_Columns
[+] RedDrive.zip/install.msi/!EventMapping
[+] RedDrive.zip/install.msi/Binary.InstallUtil
[-] extracting archive: PE
[+] RedDrive.zip/install.msi/Binary.InstallUtil/.rsrc/VERSION/1
[+] RedDrive.zip/install.msi/Binary.InstallUtil/.data
[+] RedDrive.zip/install.msi/Binary.InstallUtil/.reloc
[+] RedDrive.zip/install.msi/Binary.InstallUtil/.text
[+] RedDrive.zip/install.msi/[5]SummaryInformation
[+] RedDrive.zip/install.msi/!Media
[+] RedDrive.zip/install.msi/Binary._70B6BD6470D90F593F71019EF5DC9D42
..
..


The complete output of the command above is too large to include here but I will start looking at the beginning of the command execution and explain what happens. The first file that we process and generate a checksum for is the ‘RedDrive.zip’ file itself. That file is identified by 7-Zip as a zip archive and is therefore extracted. The zip archive contains one file called ‘install.msi’ that is also identified as an archive. After calculating the MD5 checksum for the ‘install.msi' file the content of the .msi archive is also extracted. Within the .msi archive there are a bunch of files that hashdog generate checksums for and extract if possible. 


Archive types to skip
As you can see in the command output above 7-Zip will also try to extract the resources sections contained in a Portable Executable (PE) file, something we might not always want to do. The same behavior is also observed for the Linux binary ‘ls’ that I previously copied to the directory we are processing.


[+] ls
[-] extracting archive: ELF
[+] ls/4
[+] ls/6
[+] ls/5
[+] ls/0
[+] ls/2
[+] ls/3
[+] ls/1


In most cases we do not want to extract the content of ELF binaries and PE files and by using the switch ‘--archive-skip’ we can specify which archive types we want to exclude from extraction. The option ‘--archive-skip’ takes a case insensitive comma separated list of archives that should not be processed and in our case we want to specify the archives PE and ELF.


pmedina@forensic:~/hashdog$ wc -l /tmp/example.txt
2644 /tmp/example.txt
pmedina@forensic:~/hashdog$ ./hashdog.pl --input /files --md5sum-file /tmp/example.txt --archive-skip=PE,ELF
[*] hashdog.pl version: 0.72 written by Par Osterberg Medina
[-] minimum filesize to process: 1 bytes
[-] archive binary: 7-Zip [64] 9.20  Copyright (c) 1999-2010 Igor Pavlov  2010-11-18
[-] using tmp folder: /tmp/hd950156
[+] processing files recursivly from: /files
[+] RedDrive.zip
[-] extracting archive: zip
[+] RedDrive.zip/install.msi
[-] extracting archive: Compound
..
..
[-] extracting archive: PENsis
[+] Firefox Setup 13.0.1.exe/setup.exe/\x01_1
[+] Firefox Setup 13.0.1.exe/setup.exe/\x01_6
[+] Firefox Setup 13.0.1.exe/setup.exe/\x01_2
[+] Firefox Setup 13.0.1.exe/setup.exe/\x01\x01/\x01\x1a/InstallOptions.dll
[+] Firefox Setup 13.0.1.exe/setup.exe/\x01\x01/\x01\x1a/modern-header.bmp
[+] Firefox Setup 13.0.1.exe/setup.exe/\x01\x01/\x01\x1a/ServicesHelper.dll
[+] Firefox Setup 13.0.1.exe/setup.exe/\x01\x01/\x01\x1a/ioSpecial.ini
[+] Firefox Setup 13.0.1.exe/setup.exe/\x01\x01/\x01\x1a/modern-wizard.bmp
[+] Firefox Setup 13.0.1.exe/setup.exe/\x01_5
[+] Firefox Setup 13.0.1.exe/setup.exe/\x01\x15/\x01\x1a/nsExec.dll
[+] Firefox Setup 13.0.1.exe/setup.exe/\x01\x15/\x01\x1a/Banner.dll
[+] Firefox Setup 13.0.1.exe/setup.exe/\x01
[+] Firefox Setup 13.0.1.exe/setup.exe/\x01\x1a/System.dll
[+] Firefox Setup 13.0.1.exe/setup.exe/\x01_3
[+] Firefox Setup 13.0.1.exe/setup.exe/\x01_4
[+] done, finished in: 0 hours, 0 minutes and 16 seconds
[-] deleting the tmp directory
pmedina@forensic:~/hashdog$ wc -l /tmp/example.txt
2212 /tmp/example.txt


Our database has now 432 less entries than it did before we added the ‘--archive-skip’ option.

Full file path in the hashset
Opening the file that we created and looking at the result of the command that was run above we see that the file has two columns, the MD5 checksum and the name of the file.


pmedina@forensic:~/hashdog$ head /tmp/example.txt
2e54d3fb64ff68607c38ecb482f5fa25  RedDrive.zip
70e5118a1d0cff1a59b820a919b15949  install.msi
a6ae9f9e02477526bbac1e97357141be  !_Columns
a76026797c61d04c1c9990366e48208e  !EventMapping
4b6f4f52de80f1a7890c9bd0a7cac5e3  Binary.InstallUtil
45d11bc27761a502bce036adbcf64f7d  [5]SummaryInformation
d2dd55a6b2d6d768ab6254c169d41ce9  !Media
8806ebee0e08ab6338d0fdc87be83fc4  Binary._70B6BD6470D90F593F71019EF5DC9D42
d6b3635d8e144efae4ab1753695c19af  !Dialog
8e097d7e4ebf2f6cde863fbd7de296e6  !Feature


As you can see above, the file name in the hashset is not the full path to the file that we generated the checksum for. Having the full path to the file in the hash database will increase the size of the file a lot and as in the case with installers, the path to your file will almost never match the path to the file you have on disk. There are however some cases were you want to include to full path to the file in your database. This can be accomplished by specifying the ‘--md5sum-fullpath’ switch.


pmedina@forensic:~/hashdog$ wc -c /tmp/example.txt
113263 /tmp/example.txt
pmedina@forensic:~/hashdog$ ./hashdog.pl --input /files --md5sum-file /tmp/example.txt --archive-skip=PE,ELF --md5sum-fullpath
[*] hashdog.pl version: 0.72 written by Par Osterberg Medina
[-] minimum filesize to process: 1 bytes
[-] archive binary: 7-Zip [64] 9.20  Copyright (c) 1999-2010 Igor Pavlov  2010-11-18
[-] using tmp folder: /tmp/hd950156
[+] processing files recursivly from: /files
[+] RedDrive.zip
[-] extracting archive: zip
..
..
[+] Firefox Setup 13.0.1.exe/setup.exe/\x01\x15/\x01\x1a/Banner.dll
[+] Firefox Setup 13.0.1.exe/setup.exe/\x01
[+] Firefox Setup 13.0.1.exe/setup.exe/\x01\x1a/System.dll
[+] Firefox Setup 13.0.1.exe/setup.exe/\x01_3
[+] Firefox Setup 13.0.1.exe/setup.exe/\x01_4
[+] done, finished in: 0 hours, 0 minutes and 15 seconds
[-] deleting the tmp directory 
pmedina@forensic:~/hashdog$ wc -c /tmp/example.txt
271032 /tmp/example.txt
pmedina@forensic:~/hashdog$ head /tmp/example.txt
2e54d3fb64ff68607c38ecb482f5fa25  RedDrive.zip
70e5118a1d0cff1a59b820a919b15949  RedDrive.zip/install.msi
a6ae9f9e02477526bbac1e97357141be  RedDrive.zip/install.msi/!_Columns
a76026797c61d04c1c9990366e48208e  RedDrive.zip/install.msi/!EventMapping
4b6f4f52de80f1a7890c9bd0a7cac5e3  RedDrive.zip/install.msi/Binary.InstallUtil
45d11bc27761a502bce036adbcf64f7d  RedDrive.zip/install.msi/[5]SummaryInformation
d2dd55a6b2d6d768ab6254c169d41ce9  RedDrive.zip/install.msi/!Media
8806ebee0e08ab6338d0fdc87be83fc4  RedDrive.zip/install.msi/Binary._70B6BD6470D90F593F71019EF5DC9D42
d6b3635d8e144efae4ab1753695c19af  RedDrive.zip/install.msi/!Dialog
8e097d7e4ebf2f6cde863fbd7de296e6  RedDrive.zip/install.msi/!Feature


The size of our file increased quite a lot when we started to include the full path in our database. It got 157769 bytes larger, an increase in size by almost 42%. The question you have to ask yourself when you are generating our hash databases is - does adding the full path in the file justify the increase in file size?


Generating your own RDS file
Even though ‘hashdog’ can be used to generate hash databases in a lot of different formats, the format that will probably be of most use to us as forensic examiners and incident responders is the RDS format. This is the data format that NIST uses in their NSRL Reference Data Set (RDS) and can be imported in a lot of tools. For a deeper dive into the format, I suggest you read the PDF “Data Formats of the NSRL Reference Data Set (RDS) Distribution”. http://www.nsrl.nist.gov/Documents/Data-Formats-of-the-NSRL-Reference-Data-Set-16.pdf


As in the previous example above, all we need to if we want to generate a hash database using the RDS format, is to specify the switch ‘--rds-file’ followed by the path to the file we want to store the result in. If we want to, we can also choose to store the full path to the files in the database by using the switch ‘--rds-fullpath’. 


pmedina@forensic:~/hashdog$ ./hashdog.pl --input /files --md5sum-file /tmp/example.txt --archive-skip=PE,ELF --md5sum-fullpath --rds-file /tmp/example-rds.txt
[*] hashdog.pl version: 0.72 written by Par Osterberg Medina
[-] minimum filesize to process: 1 bytes
[-] archive binary: 7-Zip [64] 9.20  Copyright (c) 1999-2010 Igor Pavlov  2010-11-18
[-] using tmp folder: /tmp/hd950156
[+] processing files recursivly from: /files
[+] RedDrive.zip
[-] extracting archive: zip
..
..
[+] Firefox Setup 13.0.1.exe/setup.exe/\x01_3
[+] Firefox Setup 13.0.1.exe/setup.exe/\x01_4
[+] done, finished in: 0 hours, 0 minutes and 16 seconds
[-] deleting the tmp directory
pmedina@forensic:~/hashdog$ wc -l /tmp/example.txt /tmp/example-rds.txt
  2212 /tmp/example.txt
  2213 /tmp/example-rds.txt
  4425 total
pmedina@forensic:~/hashdog$ wc -c /tmp/example.txt /tmp/example-rds.txt
271032 /tmp/example.txt
274652 /tmp/example-rds.txt
545684 total
pmedina@forensic:~/hashdog$ head /tmp/example-rds.txt
"SHA-1","MD5","CRC32","FileName","FileSize","ProductCode","OpSystemCode","SpecialCode"
"d9c40dd2f1fb08927e773a0dc70d75fedd71549e","2e54d3fb64ff68607c38ecb482f5fa25","732c6df0","RedDrive.zip",1148549,0,"WIN",""
"d1fa60a19ca5095731eb78cd6f6c7e3eca2cf57c","70e5118a1d0cff1a59b820a919b15949","c224e555","install.msi",1413120,0,"WIN",""
"c9220e529ad1ca74d6d7b4a8a17529e326f617cf","a6ae9f9e02477526bbac1e97357141be","8b1ddae5","!_Columns",3328,0,"WIN",""
"86d90e616458eea7188605b9f601c6bb7b46aeaf","a76026797c61d04c1c9990366e48208e","3a0a30ce","!EventMapping",208,0,"WIN",""
"e45efe29240c68452730fc32327eb3048a162e2d","4b6f4f52de80f1a7890c9bd0a7cac5e3","fa0f9bcc","Binary.InstallUtil",55296,0,"WIN",""
"e397c2f9d0ec530a8453193aa5b15eb40c48e00f","45d11bc27761a502bce036adbcf64f7d","77ec3d28","[5]SummaryInformation",412,0,"WIN",""
"36da55c877afeb7af220c80b7021368985a1d24e","d2dd55a6b2d6d768ab6254c169d41ce9","0c1fb7c4","!Media",12,0,"WIN",""
"3690c3ca05b6ed8ef86589dc93495811b1fb49db","8806ebee0e08ab6338d0fdc87be83fc4","b9c0a320","Binary._70B6BD6470D90F593F71019EF5DC9D42",12192,0,"WIN",""
"3ccfc07833e6025ffdc0b110c45c2e2c97588efc","d6b3635d8e144efae4ab1753695c19af","abc7c53e","!Dialog",594,0,"WIN",""
pmedina@forensic:~/hashdog$


In the example above, we used the switches ‘--md5-file’ and ‘--rds-file’ to generate both a file with MD5 checksums as well as a file that is using the RDS format at the same time. Currently, hashdog can generate hashsets using the MD5 checksums, SHA-1 checksums and hashsets in the RDS format, but support for more formats might be added in the future.