Skip to content

Added fiwalk disk image demo #40

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
ajnelson-nist merged 4 commits into from
Nov 17, 2022
Merged

Added fiwalk disk image demo #40

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
0449a39
added demo that shows how to use fiwalk on a disk image and report re…
bitsgalore Nov 9, 2022
1a9c2ea
Merge branch 'main' into add_fiwalk_disk_image_demo
ajnelson-nist Nov 9, 2022
2367347
Add type review to demo, and start type annotation in fiwalk.py
ajnelson-nist Nov 9, 2022
088853a
added explanation and warning
bitsgalore Nov 17, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
34 changes: 34 additions & 0 deletions demos/demo_fiwalk_diskimage.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
#! /usr/bin/env python3
"""
This demo shows how to invoke Fiwalk as a subprocess, taking a disk image as
input. Fiwalk's dfxml XML output is sent to an in-memory buffer, which is then
written to an output file. Note that this may fail for very large disk images
if the required buffer size exceeds available RAM!
"""

import sys
import io
from dfxml import fiwalk

def writeDfxml(imageFile: str, outFile: str) -> None:
"""Generate filesystem metadata for disk image and and write resulting
dfxml to file"""
# Analyse image file
with open(imageFile, "rb") as ifs:
fwOutBuffer = fiwalk.fiwalk_xml_stream(imagefile=ifs)
fwOut = fwOutBuffer.read()

# Write dfxml to output file
with io.open(outFile, "wb") as fOut:
fOut.write(fwOut)
Comment on lines +17 to +23

@simsong simsong Nov 9, 2022

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the contribution. My concern about this is that it requires the entire XML stream to be buffered in RAM at once (in fwOut) but there may be no realistic way to do this. It will fail for large images, however, unless run on systems with a huge amount of available memory. (The XML format is not very efficient.)

@bitsgalore bitsgalore Nov 9, 2022

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If there's a more efficient way to do this I'm all for it, because yes I can see how this could easily go wrong for large images. I've never worked with io.BufferedReader objects before though, and I'm not quite sure how to do this.

@ajnelson-nist ajnelson-nist Nov 9, 2022

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think this changes the status quo for how DFXML has been implemented to date. The read-write library dfxml/objects.py does some full-in-memory string construction as well.

My litmus test (not to include in CI!) would probably be seeing if this can run against the NPS 2TB image.

@simsong simsong Nov 9, 2022

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It’s complicated. I believe there is an iterator in the dfxml package that runs fiwalk as a sub process , parses it with Expat, and calls a provided call back with an object as the argument. This is the interface that all of my code used. But that was 10 years ago.


def main() -> None:
if len(sys.argv) < 3:
print("Usage: {} <imageFile> <outFile>".format(sys.argv[0]))
exit(1)
imageFile = sys.argv[1]
outFile = sys.argv[2]
writeDfxml(imageFile, outFile)

if __name__ == "__main__":
main()
11 changes: 9 additions & 2 deletions dfxml/fiwalk.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,7 @@

import os
import sys
import typing

sys.path.append( os.path.join(os.path.dirname(__file__), ".."))

Expand Down Expand Up @@ -99,7 +100,7 @@ def fiwalk_xml_version(filename=None):
return p.get_version(filename)

################################################################
def E01_glob(fn):
def E01_glob(fn: str) -> typing.List[str]:
import os.path
"""If the filename ends .E01, then glob it. Currently only handles E01 through EZZ"""
ret = [fn]
Expand All @@ -124,7 +125,12 @@ def E01_glob(fn):
return ret


def fiwalk_xml_stream(imagefile=None,flags=0,fiwalk="fiwalk",fiwalk_args=""):
def fiwalk_xml_stream(
imagefile: typing.BinaryIO,
flags=0,
fiwalk: str = "fiwalk",
fiwalk_args: str = ""
) -> typing.BinaryIO:
""" Returns an fiwalk XML stream given a disk image by running fiwalk."""
if flags & ALLOC_ONLY: fiwalk_args += "-O"
from subprocess import call,Popen,PIPE
Expand All @@ -136,6 +142,7 @@ def fiwalk_xml_stream(imagefile=None,flags=0,fiwalk="fiwalk",fiwalk_args=""):
cmd = [fiwalk,'-x']
if fiwalk_args: cmd += fiwalk_args.split()
p = Popen(cmd + E01_glob(imagefile.name),stdout=PIPE)
assert isinstance(p.stdout, typing.BinaryIO), "Failed to open pipe to subprocess stdout."
return p.stdout

def fiwalk_using_sax(imagefile=None,xmlfile=None,fiwalk="fiwalk",flags=0,callback=None,fiwalk_args=""):
Expand Down
2 changes: 2 additions & 0 deletions tests/Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -81,6 +81,7 @@ check-mypy: \
../dfxml/bin/idifference.py \
../dfxml/bin/summarize_differential_dfxml.py \
../dfxml/__init__.py \
../dfxml/fiwalk.py \
../dfxml/objects.py \
misc_bin_tests \
misc_object_tests
Expand All @@ -92,6 +93,7 @@ check-mypy-strict: \
source venv/bin/activate \
&& mypy \
--strict \
../demos/demo_fiwalk_diskimage.py \
../dfxml/bin/idifference2.py \
../dfxml/bin/make_differential_dfxml.py \
../dfxml/bin/walk_to_dfxml.py \
Expand Down