Find & Replace It!

version 0.6.0

User Manual

Last update: August 2009

1About Find & Replace It!

1.1Summary

Find & Replace It! is a powerful, cross-platform text file processor. It allows you to perform very complex batch replacements inside text files of any size. It supports regular expression syntax and dozens of encodings. It can batch processing the encoding of files, as well as style of end-of-line.

1.2Main features

The list below describes some of the most important characteristics of Find & Replace It!:

• Can find fixed string expressions as well as wildcard and regular expressions

• Allow multi-line matching

• Handles text encoding while displaying, reading and writing file contents, including multi bytes encodings like Unicode

• Preserves line endings while processing files

• Preserves BOM while processing Unicode files

• Allows you to perform dynamic replacements based on found expression captures

• Provides built-in processing function for dynamic replacements (e.g. convert captured expressions to lower case, Base64 encoding, Hex encoding, UTF-8 encoding, etc.)

• Provides a JavaScript like interface to customize replacements on the fly by script processing

• Displays matched expressions reports for file search/replace operations

• Full featured dynamic preview of matched expressions and replacements

• Provides tools for converting text encoding

• Provides tools for converting line endings (Windows, Unix, Macintosh, Unicode)

• Detects text encoding and line endings of files

• Provides advanced filtering options for selecting files that need to be processed, including file name filters and file path exclusion filters

• Allows you to load and save expressions to find, replacement definitions and file filters

• Handles huge files (> 10 GB)

• Regular expression editor

• Fully multithreaded for fast processing and responsiveness

• Allows you to cancel long operations

• GUI is totally modular

• Cross-platform: Windows, Mac OS X and Linux

1.3Supported platforms

Here is a list of supported platforms:

• MS Windows:

• Windows 98 [not tested]

• Windows NT 4.0 [not tested]

• Windows 2000 [not tested]

• Windows XP

• Windows Vista

• Mac OS X Intel:

• Mac OS X 10.3.9 and up [not tested]

• Mac OS X 10.4.x

• Mac OS X 10.5.x

• Mac OS X PPC:

• Mac OS X 10.3.9 and up [not tested]

• Mac OS X 10.4.x [not tested]

• Mac OS X 10.5.x

• Linux:

• Ubuntu 8.04 (32 bits)

• Ubuntu 9.04 (32 bits)

• Fedora 11 (32 bits)

• openSUSE 11.0 (32 bits)

Important notes about platform support:

• The platforms noted as [not tested] in the list, are platforms on which the software should work but with no warranty. All development tools used in this software are certified on these non-tested platforms but we didn't check by ourself. Most of these platforms are pretty old, that's why there are not actively supported.

• The Linux installer requires the following commands: bzip2, lzma, xdg-mime, xdg-desktop-menu, xdg-icon-resource. If one of them is missing, the installation will stop and you might have to manually remove the installation folder.

1.4Support and services

For general information, please visit our website at: http://www.dprog.ch

If you have any questions about pricing and/or license terms, don't hesitate to write to us at: order@dprog.ch

All support requests regarding software usage as well as general questions about demo version must be addressed to: support@dprog.ch

Please note that support is only available to registered customers who have a valid license for their software copies. Moreover we kindly request our customers to use the online form for posting support requests. This form is accessible through: http://www.dprog.ch/home/index.php/support

Finally, you might read the online Terms of Use statement for details about services provided by dProg – Philippe Docourt.

1.5Known issues and limitations

The text-encoding detection uses a simple heuristic that does not always provide accurate results except for all Unicode encodings (i.e.: UTF-8, UTF-16, UTF-32).

For performance reasons the Find and Replace Preview has a content limit of 5*1024*1024 characters.

For performance reasons it's not possible to search for an expression longer than 5*1024*1024 characters.

The HTML Viewer component might crash under Mac OS X when unloading flash content, if the Enable plugins option is marked. This problem does not occur under Windows and Linux.

The product activation requires an Internet connection.

2Product activation

Every time you start the application a dialog window will ask you to activate your copy of Find & Replace It!. This activation is necessary to access all features of the software and requires an Internet connection.

The activation window shows you an expiry date for activating the software. After this date, the software will not start any more without being activated first. Until this date, you can simply refuse the activation by pressing Cancel and use the software in demo mode. Once the product has been activated, the activation dialog will not any more show up at startup time.

When buying your copy of Find & Replace It! from http://www.dprog.ch/home/index.php/buy?product_id=1 you should receive an activation key by e-mail. To activate the product, follow these steps:

1. Enter your login for www.dprog.ch website;

2. Enter your e-mail (the one you used to register on www.dprog.ch and where you received the activation key);

3. Type or paste your activation key in the appropriate field;

4. Press OK and wait for the answer. In case of success, the activation window will be automatically closed. Otherwise, you might try again later. In case of problem contact our support.

Note: You cannot close the activation window with OK unless you successfully activate your copy of the software. To close the window without activation, press Cancel.

You can activate or check your activation key at any time through the Help/Activate product... menu as shown below:

3Getting started

3.1Selecting files to search/replace in

There are two sections that work together for searching and selecting files. They are Search scope and Found files.

1. Select a folder to search in. This is the root path where you want to scan for files. You can type a path directly in the Search in text field or use the button on the right.

2. Choose to search files recursively into sub folders or not by toggling the Search in sub-directories check mark.

3. Enter zero, one or many file name filters within the File name filter field. These filters interpret wildcards characters like '*'. They must be comma separated.

4. Optionally add one or more expressions to exclude some file paths when searching for files. This can be achieve with the button. These filters can use wildcard or regular expression syntax. Note that all file paths are described with '/' separator whatever the platform or system locale is.

5. Select files that you want to process in the found files list. Unmarked files are not going to be read or touched. The content of this list is updated whenever you change search options. You can filter the content of this list through the file path filter above the list view. Click the column header to sort that column.

3.2Doing a backup of files before modification

The File backup section allows you to create a backup for each modified files.

1. 
   Mark the backup check box as shown above;

2. Type a suffix for your backup files.

3.3Detecting and selecting files encoding

The Found files section shows the found files according to your current options for searching files. The column Encoding is the only one that is editable by the user. You can choose by hand the appropriate codec for each file with a double-click on the appropriate line. This will show up a drop down list of available codecs:

The “best codec” for processing text encoding of each file is detected using a simple heuristic and it is selected by default. Please note that this heuristic is only reliable for detecting Unicode encoding. For other encodings it will only give you some suggestions. There might are many “acceptable codecs” and they are all marked with a blue light on the left side of the drop down list. This is shown below:

If the codec name is set to unknown at a given row, that means that no codec seems to match the associated file. When one or more codecs are detected as acceptable, the preferred text encoding is selected by default when it is available. The acceptable codec list is automatically determined whenever a new file is displayed in the list but not when files are changed (not yet). For refreshing the encoding detection, click on . This will detect the preferred codecs for all files.

If you need to look at the decoded content of a file, right-click on a file entry, then click on Open file in test preview. This will allow you to play with the codec used to decode the file.

Note: There is a special codec named System. This encoding varies with the current locale of your system. When not working with Unicode, this codec often appears to be a good default value for unknown encodings.

3.4Converting files encoding

For converting the text encoding of a given set of files, follow these steps:

1. Select the files you want to convert with a check mark in the File path column, within the Found files section;

2. Select the current text encoding of these files if the auto-detected encodings are not accurate;

3. Select the target encoding for your set of files;

4. Optionally you can select the Generate Byte Order Mark (BOM) check box. This will insert the BOM (Byte Order Mark) at the beginning of the file when it is written. This option only apply to Unicode text encoding: UTF-8, UTF-16 and UTF-32. Note that this option may interfere with the target encoding. For instance, if you choose an Unicode encoding that does not allow the BOM, it will turn your target encoding to the closest Unicode encoding that allows it.

5. Optionally you might schedule a backup of modified files;

6. Click on the button to start the encoding conversion. If you want to stop the conversion process, click on the button .

3.5Searching for an expression

The Expression to find section allows you to setup an expression to search for:

1. 
   Enter the expression to search into the Find text field. Alternatively you might use the button to edit your expression with the Regular Expression Editor.

2. You can choose the way your expression must be interpreted through the Syntax drop down list:

• Simple text or fixed string: means that the pattern to be matched is interpreted as a plain string

• Wildcard: is similar to the functionality found in command shells

  
  Regular expression: is a pattern for matching substrings in a text

3. Select the options that apply when matching against your expression:

The 'Minimal match' match option is only available when 'Regular expression' syntax is set. This turns the quantifiers in non-greedy mode.

4. Select the files you want to scan with a check mark in the File path column;

5. Select the current text encoding of these files if the auto-detected encodings are not accurate;

6. Optionally you can test your expression with the Find and replace preview ;

7. Click on the button to start searching your expression into the selected files. If you need to stop the search, click again on the same button which has been morphed into once the search has been started.

3.6Replacing found expressions with fixed string

1. Setup the expression you want to search;

2. 
   Type your replacement pattern for matched occurrences of your expression in the Replace with text field:

3. Optionally you might schedule a backup of modified files;

4. Click on the button to start searching your expression into the selected files. If you need to stop the replacement, click again on the same button which has been morphed into once the replacement has been started.

3.7Using the 'Find and Replace Preview'

When editing an expression to search for, it is convenient to match it against real data. In order to achieve this, let's take a tour of the Find and Replace Preview window. This is a multi-document editor with powerful features that allows you to:

• check the impact of a given text encoding when applied to a file content;

• edit a text sample against which you want to match an expression to find;

• preview matched occurrences of an expression to find inside a given text sample;

• preview resulting content of a text sample after the replacement of all occurrences of your expression with your replacement pattern;

• preview both found expressions and processed replacements inside a text sample;

• navigate through found occurrences of a specific expression or replacement pattern;

• preview HTML documents.

The following screenshots illustrate some of the capabilities described before:

Highlight of replaced matched expressions within a file. The tool-tip shows information about occurrence location.

Highlight of replaced expressions within a file.

Highlight of both found and replaced expressions within a file. The tool-tip shows information about replacement location.

Note that all line breaks in the preview are internally represented by a Line Feed character (LF, U+00A). This is always true, whatever the original end-of-line used in the displayed file. If you want to search for a multi-line expression with another style of line break, we strongly advise you to use a regular expression with appropriate \s+ sequences in order to match any kind of end-of-line.

In addition to the plain text preview, you can activate the HTML Viewer through the button. That will enable you to preview HTML documents, with either their original or altered content, without having to save them.

The HTML preview in action with our own website.

The viewer is directly synchronized with the current content of Find and Replace Preview. That means the Display mode will also apply on the HTML content. Because the viewer cannot resolve relative links in the HTML document from the content of the preview, you might need to enter an appropriate URL. This URL is used to process all resources referenced by relative links within the document (i.e.: CSS, images, scripts, etc.). This is not required when all resources are given with absolute path.

The HTML Viewer provides rendering of HyperText Markup Language (HTML), Extensible HyperText Markup Language (XHTML) and Scalable Vector Graphics (SVG) documents, styled using Cascading Style Sheets (CSS) and scripted with JavaScript. Some common plugins are also supported through the Netscape Plugin API, provided you have appropriate binary files for those plugins installed on your computer. The following locations are searched for plugins:

3.8Using the 'Regular Expression Editor'

When you need to write a multi-line expression to search, the Regular Expression Editor is your best friend. The multi-line edition is shown bellow with a simple text expression:

Note that all line break are internally represented by a Line Feed character (LF, U+00A). If you want to search for a multi-line expression with another style of line break, we strongly advise you to use a regular expression with appropriate '\s+' sequences in order to match any kind of end-of-line.

In addition, this editor simplifies the setup of regular expression. It provides tools to manage regular expression entities. On the right side of the text editor, there is a list of available regular expression entities (e.g.: special characters, grouping expression, etc.). If you leave the mouse over an item of this list, a explanatory tooltip will appear. This is show in the figure bellow:

The Regular Expression Editor has some nice features like syntax highlighting and scope matching (e.g.: matching scope for '()', '[]', '{}').

The Regular Expression Editor offers an automatic syntax check for wildcard and regular expressions:

As soon as your wildcard or regular expression pattern becomes invalid, it is underlined. A tooltip provides a brief description of the syntax error detected.

3.9Advanced replacements

Advanced replacement covers three main features that make Find & Replace It! really powerful:

• Injecting a fragment of the matched expression into the replacement text;

• Transforming a fragment of the matched expression before injecting it into the replacement text;

• Interact through a JavaScript interface with the replacement text.

Each of these points is described in the following chapters.

3.9.1Using captured texts within replacement string

This feature requires regular expression syntax for the expression to find; furthermore, you should be familiar with captures within regular expressions. To learn more about these notions we recommend to read the Regular expressions chapter. If you are familiar with regexp, read on the following example.

Whenever you capture some text fragments with an expression to find,

you can inject these captured fragments into your replacement pattern. This is done with a %1, %2, …, %9 pattern. Where the number that follows the percent sign is the capture index. %0 is a special, implicit capture that includes the full matched expression.

Let's imagine we have a CSV file containing contacts like in the following snippet:

First Name: John; Family Name: Smith; Phone: ...

First Name: Mike; Family Name: Dupont; Phone: ...

We want to swap the first two columns. Here we have to capture two variable expressions (first name and family name) and move them around. Here is an easy way to do it.

Find:

(First Name: [^;]+); ( Family Name: [^;]+)

The parentheses in expression above will capture two fragments of every matched occurrences in the CSV file.

Replace with:

%2; %1

The replacement pattern above is a dynamic text that varies for every matched occurrences. In fact %1 will be replaced by the content matched by the first parentheses scope. Idem with %2 and the second parentheses scope.

3.9.2Processing captured texts

This feature requires regular expression syntax for the expression to find; furthermore, you should be familiar with captures within regular expressions. To learn more about these notions we recommend to read the Regular expressions chapter. If you are familiar with regexp, read on the following example.

As shown in the previous chapter, it is possible to inject captured texts into your final replacement string, through a special syntax. It is also possible to apply an additional processing to captured strings before injecting them as a replacement expression. This can be handle with the Capture processing section:

Capture #1 and #2 are available but not capture #3. A distinct process has been attached to each capture, however it is not compulsory.

The left column is not editable. The check mark is toggled depending on the presence of captures and placeholders respectively within the expression to find and within the replacement pattern. The right column let you choose a transformation to apply to the capture, before injecting it as the replacement text at its placeholder location.

Let's imagine that we want to upper case the first letter that follows a ':' sign inside a file. A tedious solution might be to replace all : a with : A, : b with : B and so on. This will take some time. And then, what happens with accentuated letters or oriental characters ? What happens if a tab sometimes replaces the whitespace after the ':' sign? What if there no whitespace at all or many whitespaces due to a typing mistake? This solution is definitively inappropriate.

Here is a better way to handle this task:

Find:

:\s*(\w)

The expression above will match all ':' followed by any number of whitespace characters (including tabs and line breaks) and at at least a word character.

In the Capture processing section, select To upper case as process for the first capture. This will transform to upper case the content matched by the first pair of parentheses, before injecting it as %1 in the replacement pattern.

Finally we replace with:

: %1

If the built-in capture processes are not sufficient, you might try the scripting interface.

3.9.3Processing replacement pattern on the fly by script

Find & Replace It! provides a JavaScript like interface to customize replacements on the fly by script. This is especially useful when you need some logic to interpret an expression and generate a replacement pattern based on it (e.g. find all numbers in a text and divide them by a given factor to convert units). To achieve this you only have to type your JavaScript in the script editor.

our script will be called at 3 different occasions. All of them can be accumulated as desired:

• Once with the original replacement pattern and the full matched expression;

• Once for each captured text in your expression if the capture process has been set to Apply script;

• Once at the very end of the process, after all capture processing, with the resulting replacement pattern.

A dummy script outputting invariable properties from the scripting context.

The scripting interface provides a simple way to access the current matched context as well as the replacement pattern through the global replaceCommand object. The table below summarizes the context information made available to the script:

Let's imagine we have a text file containing numerous numerical values. All of these values represent a length given in millimeters. We would like to convert all these distances from millimeters to inches. Sounds tricky to you? Script processing enables you to handle that task like any other replacement tasks. First we setup an expression that matches all numbers (integers and floating point):

(-?(\d+)\.?(\d*))

As a replacement we simply inject the full captured number:

%1

In order to apply some script on the replacement pattern activate capture processing (i. e. check mark), set process to Apply script, then activate the script processing editor option as shown below:

Finally copy and paste the following script within the script editor:

// Function to convert millimeters to inches

function convertMillimetersToInches(valueInMilimeters) {

return valueInMilimeters/25.4;

}

// We want to output a custom replacement for capture at index 1

if(replaceCommand.captureIndex==1) {

// Convert the numerical value captured at current

// index (i.e 1) from millimeters to inches

var convertedValue =

convertMillimetersToInches(replaceCommand.captureText);

// Get number of digits to format output

var digitCount =

replaceCommand.capturedTexts[2].toString().length;

// Take in account that the converted value is about ten times

// smaller, therefore we subtract one digit to represent it

// with same accuracy

digitCount -= 1;

// Get the number of captured decimals (original decimal count)

digitCount += replaceCommand.capturedTexts[3].toString().length;

// Ouput the formatted value with an equivalent number of digits

convertedValue.toPrecision(digitCount);

} else {

// For the other indexes (i.e. 0, 2, 3, 4) we simply output

// the current replacement text

replaceCommand.replacementText;

}

That's it! As an alternative we might remove the outer parentheses, then use %0 as replacement pattern instead of %1 and adjust the script accordingly.

3.10Using the output

The Output window is a multi-document preview for find and replace reports. Every time you search for an expression within files, a report will be outputted to the active console tab. The report includes links to files that contains searched/replaced expressions. A simple click on this link will open the file within Find and Replace Preview. A double-click on this link will open the file with a suitable application.

The console showing a report for found expressions. The report gives you some statistics.

4Tips and tricks

4.1Saving your work

All sections containing this button support persistent serialization. Therefore, you can save an expression alongside with its replacement pattern and script, as well as all components for searching and filtering files. Of course the twin button enables you to load a file previously saved.

The File menu provides a way of saving and loading the full content of the interface. When saving, this is almost equivalent to concatenating all files generated by all buttons of all sub-windows.

Finally, if you open a fri file from a sub-window button (e.g.: Find and Replace Control Panel), only content related to this sub-window will be loaded. That means you can easily load a part of a file saved through the File/Save menu.

All saved files are automatically suffixed with fri. They are called UI files (i.e.: User Interface File) for Find & Replace It!.

4.2Multi-document tabs

Some components use tabs to display multiple documents. To manually open a new document/tab, use this button . To close a document/tab, click on the button located on the corresponding tab.

4.3Working with text areas

4.3.1Editing text

This is the list of key bindings which are implemented for editing:

Keypresses	Action
Backspace	Deletes the character to the left of the cursor.
Delete	Deletes the character to the right of the cursor.
Ctrl+C	Copy the selected text to the clipboard.
Ctrl+Insert	Copy the selected text to the clipboard.
Ctrl+K	Deletes to the end of the line.
Ctrl+V	Pastes the clipboard text into text edit.
Shift+Insert	Pastes the clipboard text into text edit.
Ctrl+X	Deletes the selected text and copies it to the clipboard.
Shift+Delete	Deletes the selected text and copies it to the clipboard.
Ctrl+Z	Undoes the last operation.
Ctrl+Y	Redoes the last operation.
Left	Moves the cursor one character to the left.
Ctrl+Left	Moves the cursor one word to the left.
Right	Moves the cursor one character to the right.
Ctrl+Right	Moves the cursor one word to the right.
Up	Moves the cursor one line up.
Down	Moves the cursor one line down.
PageUp	Moves the cursor one page up.
PageDown	Moves the cursor one page down.
Home	Moves the cursor to the beginning of the line.
Ctrl+Home	Moves the cursor to the beginning of the text.
End	Moves the cursor to the end of the line.
Ctrl+End	Moves the cursor to the end of the text.
Alt+Wheel	Scrolls the page horizontally (the Wheel is the mouse wheel).

To select (mark) text hold down the Shift key whilst pressing one of the movement keystrokes, for example,?Shift+Right?will select the character to the right, and?Shift+Ctrl+Right?will select the word to the right, etc.

4.3.2Undo/redo changes

It is possible to undo and redo any change made in a text area when this area is editable. On the Edit tool bar or in the Edit menu, simply click on:

• Undo (Ctrl+Z)

• Redo (Ctrl+Y)

4.3.3Changing display

The commands located in the Text display menu let you change the appearance of text zone content:

• Zoom out (Ctrl++)

• Zoom in (Ctrl+-)

• Select a font

All these commands are available for all text area int the software but they only apply to the last area that has been activated. Therefore you might have to click somewhere inside a text area to get a result.

4.3.4Searching for text

It is possible to search for text within any text area of the graphical interface. Simply click on in the Edit tool bar or in the Edit menu. This will show up the Find Text window:

The search will occur in the last area that has been activated. The background is colorized in green when there is a match, in red when there is no match.

4.4Multi file selection in the found files list

To operate on many files at once it is possible to use the context menu on the Found files list. This menu let you act on the the current selection. As shown below, possible actions on selected files are: toggling check marks, selecting encoding, loading files in the preview. Applying an action on a selection of files. Note that 'Open file in test preview' will open all selected files as distinct documents within Find and Replace Preview.

4.5Get examples

There are some sample files shipped with Find Replace It!. Under Windows and Linux these files are located within the following directory:

/res/examples

On Mac OS X, these files are located in:

/FindReplaceIt.app/Contents/Resources/examples

The files suffixed with fri are UI files (i.e.: User Interface File) for Find & Replace It!. Such files contain stored user interface data and can be opened with buttons.

The files suffixed with .txt are sample data provided for convenience, in order to easily test the capabilities of Find & Replace It!.

4.6Debugging script

When willing to script some replacement texts it is convenient to debug the script. Find & Replace It! comes with an integrated debugger. To start the debugger, click on the Execute in debugger button located under the script editor. The debugger will show up:

Script debugger in action. On the right we can see the 'Locals' dock window which displays the current context provided by the 'replaceCommand' object.

A user manual for script debugger is available at http://doc.trolltech.com/4.5/qtscriptdebugger-manual.html

5Regular expressions¹

A regular expression, or "regexp", is a pattern for matching substrings in a text. This is useful in many contexts, e.g.:

The Find & Replace It! regexp is modeled on Perl's regexp language. It fully supports Unicode. The regexp can also be used in a simpler, Wildcard mode that is similar to the functionality found in command shells. The syntax rules used by regexp can be changed through the Syntax combo box. In particular, the pattern syntax can be set to Simple text, which means the pattern to be matched is interpreted as a plain string, i.e., special characters (e.g., backslash) are not escaped.

A good text on regexps is Mastering Regular Expressions (Third Edition) by Jeffrey E. F. Friedl, ISBN 0-596-52812-4.

5.1Introduction

Regexps are built up from expressions, quantifiers, and assertions. The simplest expression is a character, e.g. x or 5. An expression can also be a set of characters enclosed in square brackets. [ABCD] will match an A or a B or a C or a D. We can write this same expression as [A-D], and an experession to match any captital letter in the English alphabet is written as [A-Z].

A quantifier specifies the number of occurrences of an expression that must be matched. x{1,1} means match one and only one x. x{1,5} means match a sequence of x characters that contains at least one x but no more than five.

Note that in general regexps cannot be used to check for balanced brackets or tags. For example, a regexp can be written to match an opening html and its closing , if the tags are not nested, but if the tags are nested, that same regexp will match an opening tag with the wrong closing . For the fragment bold bolder, the first would be matched with the first , which is not correct. However, it is possible to write a regexp that will match nested brackets or tags correctly, but only if the number of nesting levels is fixed and known. If the number of nesting levels is not fixed and known, it is impossible to write a regexp that will not fail.