Working with large files
October 5th, 2005 by alpriest
I’ve been working on a 30gb Xml file today, importing it into one of our systems. There are always problems working with files this big - how do you get just a small segment to test on? how can you view small regions of it, or search it?
Most tools available for Windows aren’t up to the job, but i’ve finally found 3 which will help you should you have similar requirements:
1) Baretail. This will show you any part of a text file regardless of file length. It only loads what you’re looking at into memory and thus has a constant memory footprint. It also provides multiple highlights and will follow (tail) the file for updates. You can copy sections of the file from here into a new file (copy/paste).
2) Baregrep. This will search any size file for a given pattern.
3) frhed. By using the ‘partial open’ feature, you can get it to open just a fragment of any file in hex mode thus letting you see what’s actually there.
