2. Words/characters Recognition Problems

The search method may be unable to identify materials when any of the following is true: (1) illegible handwriting (even hard for human), (2) writing text has irregular spaces, (3) keywords in text corresponding to are misspelled, (4) relevant text contains invisible characters such as control characters, special characters, and format characters, and (5) search keys and critical works in text use different encoding schemes. For this class of documents, the design of the search algorithm and the quality of the optical character recognizer (“OCR”) can directly affect search result. The impact can be unpredictable: it may ruin one kinds of documents, all documents created by one author, all documents processed by a vendor. 

For other graphic files, the accuracy of a search would depend upon file formats, how files are created, and how texts are searched.

4. Lack of Antecedents and Contextual Deficiency

A very common problem is passing information by context or prior understanding. When critical information is passed as contextual information, the author does not need to use any of the common words to express it. A statement: let go, OK, you get it (and virtually anything)..... can be used as proving a transaction that the author has been asked before. When contextual information does not appear in the current email or document, there is no good method for capturing this type of documents.

Unspecified antecedents are very common problem in most documents. This problem has a lot to do with the purpose of the corporate documents. Documents are created to get business done, by a group of employees who know everything concerning the business. The writing is informal most times. There is no need to state background, purposes, current standing, related activities, involved staff, external facts, and relevant business, company history, and legal environment, unless it is necessary. They write documents to address the issues they want to address. A vast number of documents standalone is out of context. The email may be one sentence “Have we filed appeal with the court?” “we visited their headquarter to see the facilities.” “The board has approved the deal.” It is common even for long documents. For example, a very long table contains several columns of numbers without any explanations; long reports have all kinds of analysis data but do not indicate their uses; a long email does not reveal why they discuss the subject, whom they represent, and how the subject is related to other subjects (all information might be in other separate email). There are all kinds of scenarios for documents to become undecipherable due to lack of a large number of antecedents. Document reviewers can recognize all words and phrases, but cannot understand the precise meaning in some aspects important to their discovery decisions.

Documents often reveal their true meanings only when they are read in light of other related documents. Most of search words cannot capture relevant documents by using search methods. 

5. Character Encoding Problems

Documents may be created by using text editors and various word processors. Two common practices are copying text from a document to document, and using a document from any source as a start template to save time. Each of the letters or characters in the files is represented by a binary value. Text files also contain special characters, control characters and sometimes format characters. When those files are saved, read, copied, and transmitted, those binary values should be preserved. However, some programs may change their binary values. If a change affects formats, they may affect search if a search algorithm uses format in its match method. 

Serious problems can be seen in documents in foreign languages. Many foreign languages may be expressed by different encoding schemes. A letter or character may be expressed in different length with different binary values. When a program reads, opens, displays, and saves a file, it must recognize the encoding scheme in the file before the program can process it correctly. Not every program is capable of identifying all used encoding schemes in the file and processing it correctly. If the text in a file is created according to one encoding scheme, but a program processes the file using a different encoding scheme, it would alter the file. If the altered file is saved, the program permanently damages the file. That is why many files contain wrong, strange, or garbed characters.

Writing several languages in one file and using mixed encoding schemes in a file in the same language can cause additional problems. A reading, writing, and rendering program may be unable to recognize several encoding schemes for one or multiple language texts within the same document. After such a file is opened and saved, some foreign languages may become garbed. They may be altered, even though they appear to be the same.

When a document contains text in several languages, the text may be  created by copying text from other sources or old documents. Thus, the text or some part of it may use different encoding schemes. The copied text may be not in compliance with the unicode encoding standard, or the used encoding scheme, and the document thus contains mixed encoding schemes. One possible way of altering encoded text is when a program, which does not support multiple encoding schemes, is used to process the document. The program may alter the portion of text encoded by a scheme that the program dose not support. When the document is saved, this part of text is forever altered. Other programs will not be able to process the altered text after the document is saved. This explains the common phenomenon that some documents contain garbage in part of the text, text one or more languages, or in some text within the document.

Although the unicode scheme is intended to eliminate these problems, many old text files that do not comply with the unicode standard still exist and they may be used as sources for copying. In addition, a large number of old document templates and model documents are still used as start documents to save time. A large number of legacy programs including word processors, email clients, text edits, and web browsers are still in use. They are unable to correctly determine Unicode and other encoding schemes. The differences among different coding schemes can be very small. Sometimes, a change to an encoding scheme may affect only certain characters. It can be very strange. In some cases, only one in every ten characters is altered. In other cases, only certain letters or characters are altered without any apparent sign. In the worst case, the program may alter only invisible format characters.  

A search algorithm will fail to find intended words and phrases if (1) the text encoding scheme is different from the encoding scheme of the search key, (2) the encoding scheme of the text has been changed, (3) the search algorithm supports only certain encoding schemes while the text contains an encoding scheme which is not in compliance with unicode standard, (4) certain binary values have been altered, making match impossible. For those reasons, various obvious words in text cannot be “found.” 

The search algorithm of any search method may affect how well the search method can handle all character-matching problems. At this point, the search technologies are unsatisfactory. Too little effort has been done to understand the encoding problems that happen in a large number of documents. Little effort has been made to restore correct encoding in the cases where it is possible. In the worst cases, the text in a majority of documents are garbled. In all kinds of situations, it is hard to argue that search can achieve a valid result.

6. Documents Intended to Keep Secret

Any intended meaning can be expressed by implication, connotation, analog, and uncommon expressions, while transaction may be referenced by project name, project code, transaction nickname, and all kinds of other unfamiliar words. 

When a discussion subject is sensitive or is highly private or confidential, the author may express it in a vague way so that it is hard for others to understand. Any attempt to keep subject secret can defeat the search method because it would be impossible to know what kinds of search keys can be used to capture such expressions.

When an investigation is made to look at intentional wrong, search methods should not be used. 

7. Partial Communication

Documents may be written as an incomplete or partial communication. Such documents are only part of the “documents” for a two- or more-way communication cycle. The first part communication is normally a phone call, a personal interview, a personal or group meeting, and other separate communication (e.g., video conference, faxed document, and separate email). The second part of the communication is a response to the first part of communication. Since the first part of communication is unknown, the second part of communication is hard to understand. It may be a write-up requested in a prior interview, a response to a question posed in a meeting, or a report that has been scheduled long time ago. 

Partial communication is out of context to document reviewers. Sometimes, a few things can be inferred from such a document; sometimes, it is impossible to make any inference. In all cases, it is hard to get all required information for making reliable discovery decisions. Search method may totally screw up the discovery.

8. Complex and Sophisticated Schemes

A scenario is a perpetual money-transporting cycle. An Indian company retained a consulting firm in India for bridging it with government officials in order to get government contracts. The company never directly gave money to the consulting firm for improper purposes. The company and the consulting firm also had a joint venture for doing other business unrelated to the company's business. In that joint venture, the company always contributed more property and got less cash for its contribution. This actually resulted in a net cash flow from the company to the consulting firm. All transactions between the company and the consulting firm were lawful, and all commission payments were reasonable. However, a careful review of this transaction shows that the company gave a large amount of cash to the consulting firm by the abnormal distribution of cash in the shared venture. It is also found that the consulting firm spent 10 millions to bribe government officials. Now, a U. S. company conducts a due diligence review in connection with a proposed acquisition of the Indian company.  In this sophisticated cash transporting system: the company gave net cash by the joint venture to the consulting firm, the consulting firm bribed the government officials, the government officials got bribe, and official gave business to the company. It is highly unlikely that the search method with any search keys could capture all required documents unless the key designers know this perfect money-transporting scheme and are familiar with all of those documents. 

When an improper objective is achieved by using a sophisticated arrangement, the search method cannot be used to reliably capture relevant documents. In another example involving payment of money, a company may be rewarded with certain stock for donating money to other organizations in other countries. The company may not regard such donated stock as part of its business asset and may not even enter the stock values into the company book. However, the employees of the company can “sell” the stock at a nominal price to a middleman who then uses his own money to bribe government officials. It is hard to design keys to get those documents.

There are still other file formats that prevent a search method from finding responsive documents. They include encrypted files, executable applications reflecting relevant issues, audio files, and video files. 

Specific Examples Showing Search Method's Limitations

The following examples show why certain search keys cannot retrieve relevant and critical documents and thus cast doubt on the validity of the search method. When the search method is used, document reviewers cannot see excluded documents. Thus, it is impossible to determine specific misses. The only way to assess the true effects of the search method is to conduct a comparative study of (1) reviewing all documents from all potential customers and identify all responsive documents (using an improved review model), and (2) determining if all of the true relevant documents are within the captured documents. However, it is impractical to do such a study due to increased costs. Therefore, indirect evidence is used to show potential problems. The following problems can be observed at review sites.

Risks from Using Search Methods

All search methods and all prioritized methods do not make a final coding decisions like computer algorithms, and thus cannot directly create risks.  

Both of the methods pose some risks through disrupting review context and affecting review performance of licensed attorneys. This is how the methods increase risks. Under the current review model, licensed attorneys are not enabled to deliver potentially highest performance, but still have some chance to catch some critical documents if they are definite on face. Some risks of using search methods include that (1) failure to capture a document revealing an attorney's affiliation may prevent reviewers from correctly identifying some privileged documents, (2) failure to capture some documents disclosing prior litigation work product may prevent reviewers from identifying some work product, (3) failure to capture certain documents in a complex transaction may prevent reviewers from detecting unrelated liabilities; (4) failure to capture critical documents for a complex scheme may prevent reviewers from detecting the whole scheme; and (5) disrupting document context may prevent reviewers from identifying other risks, thereby the law firm in a bad position to protect the client's interest. 

If document review context is seriously disrupted due to prioritization of documents  or removal of helpful documents from the review pool, the reviewers will be in a worse situation to detect many kinds of issues critical to litigation. The law firm may be unaware of the client's scandals, similar conduct, poor credibility, malice and intention, business risks, and litigation risks. The adversary party, who might know those issues from other sources or prior dealing, could surprise the court when the law firm has neither resource nor time to address it. It is unrealistic to tackle millions of documents in a short notice. In litigation, a big surprise can change the parties' litigation postures instantly. 

Risks may be divided into four types: (1) risks that have some impact on the current case only, (2) risks that can decisively defeat the client's case, (3) risks that can negatively affect the client's competitive position, and (4) risks that can increase the probability of exposing additional lawsuits against the client. Some risks are small to moderate, but others are equivalent to the hull-loss risk in the aviation industry. If a materialized risk can instantly defeat the client's case, cripple its business or significantly diminish its competitive position (e.g., help several new competitors unseat the client's dominant position), and assist other persons in initiating a chain of lawsuits (e.g., private actions, regulatory actions, consumer class actions, and shareholders actions), this risk is equivalent to the hull-loss risk in aviation because they threaten the client's existence. 

Whether risks are of a hull-loss type or of limited nature would depend on the nature of the case and the risk itself. In the above example concerning complex and sophisticated schemes, using a search method may cause document reviewers to miss the money-transporting scheme. This risk is considered as a “hull loss” type to a corporate client because the company would “buy” a criminal prosecution with a chance of exposing additional civil and criminal liabilities from its own documents. It cannot be assumed that a mass document production will not implicate export violation, antitrust, patent infringement, trade secret misappropriation, securities fraud.... By disrupting review context, the search methods can interfere with reviewer's ability to identify other critical issues, business secrets, and unrelated risks. Some exposed risks may be materialized many years later, and some may hurt the client for years without the client's awareness. Only those who take advantage of the information know it. One must assume that a massive document production contains everything that could be used against the client.

Search methods and many other technologies can indirectly hurt the client through operation of what I call a “skilled person in the subject.”  Attorneys and document reviewers who is not a skilled person in the subject need full context information to conduct full analysis in order to fully understand the subject and risks. However, other persons skilled in the subject need only pieces of information to be combined with other information to fully understand the subject. This is very similar to the patent law's “skilled person in the art” concept which has been used to determine invention's merit for centuries. Both search methods can affect risk control in two different ways. Disrupting review context by search methods can cause document reviewers to fail to identify risk subjects, but will not stop them from producing risk-carrying materials. This is primarily due to the current non-aggressive discovery practice by which all non-responsive materials in non-privileged responsive documents are produced to the adversary. This practice is in sharp contrast to the conventional discovery standard of producing only responsive materials. 

From a huge production containing a millions of pieces of non-responsive information units, persons skilled in the subject can pick up partial information to be combined with other information they know to be used against the client. Competitors might have context information from other sources to assist them in figuring out the client's business plans, pipeline products, and trade secrets. The leaked information  may place competitors in a position to design around the client's business plans and easily take advantage of the client. There is no way for the client to effectively compete with competitors. By using a similar approach, a potential adversary can use pieces of information to be combined with other information they have as support of new lawsuit. It is a mistaken belief that a disrupted review context and/or incompetent reviewers will favor withholding their business information and risky materials. They do, but they do it by putting far more in the responsive pile than in the non-responsive pile, especially, when they have to code documents as responsive whenever they could not understand. 

In this highly competitive world, competitors and the adversary actively seek information form all sources. Considering the frequent leakage of government's information, intelligence information, and military information, and how businesses collect intelligence information, it must be presumed that producing a huge amount of non-responsive information that no body even pay attention to is a high risk. It cannot be assumed that the massive amount of non-responsive information in hundreds of thousands of documents is harmless. Coding documents by guessing should be viewed as a dangerous game if the case is viewed as a game against a whole world.

If a collaborative review model is used to improve review accuracy, one critical step is conducting a reconciling review for all documents that have been coded tentatively (where at least one critical fact is assumed and the significance can change). The validity of that step depends upon the effectiveness of the search method. However, as demonstrated above, search will not able to find troublesome documents. This analysis underscores the need for mapping out all unsearchable documents that appear to be critical or important. This can be done by entering a record in a database table for tracking each of suspected documents. The record must contain critical key, relevancy information and/or risk reason, and document location. In conducting reconciling review, a series of searches will be conducted against the tracking table in addition to  searches conducted against documents. If all critical documents are available for review, the limitation of the search method will not affect review product in a predictable way. However, if the search method has failed to capture some critical documents, human review cannot undo this effect. Search method could change case disposition.

Conclusion

Search methods and prioritized search methods together with proper search keys are useful in cases where litigants will not suffer from hull loss risks. The methods generally are incapable of reliably capturing all responsive documents important to litigation outcome. They are incapable of reliably capturing short documents, troublesome documents, and the documents intended to keep secret. Both methods can negatively affect performance of document reviewers primarily through disrupting documents context. They may indirectly increase risk of exposure through over inclusive production of non-responsive documents, which could be used by skilled person in the subject.