Importing large CSV files via Batch Apex

Overview

This article explains how to use a batch apex process toread a CSV file. You might need to use this asynchronous process to avoid the”Regex too complicated” error when importing a large amount of datafrom a CSV file. Suggestions from the developer community include splitting theCSV file, but if you need to read the data in one go, there is no option toprocess more than one file.

To resolve this issue, you can create a batch apexprocess to import a large amount of data in one file and process it.

ExampleClasses

In this article we are using two classes written by Marty Y. Chang.

To reada CSV file, these classes must be provided with the first row of the data andthe delimiter used in the file.

Creatinga Batch Apex Process

As thecommunity suggests, you must split the file to avoid exceeding the heap size.You can create a batch process to read the file in chunks and define the numberof lines to read for each chunk.

To dothis, create a batch apex process where the scope size defines the number ofrecords to read for each chunk. In the sample code that follows, lines from aCSV file are to be read rather than records. The start method returns anIterable<String> that contains the lines to be processed in the executemethod. Afterwards, the process reads the list of lines using the CSVReader inthe same way as an online process.

Here is an example: [code_inline id=2319]

Creating the Iterable<String>

Youmust also create a class that implements Iterator and Iterable. The purpose ofthe class is to create a string for each line in the file.

Here is an example: [code_inline id=2320]

UseCSVReader.cls to Read the Lines or Create a Custom Method?

Some lines in CSVReader.cls perform the same function as CSVIterator.cls mentionedearlier. You can do one of the following:

  • Use the CSVReader.cls in the execute method. Thecode runs the same lines twice, but you can use the method withoutmodification.
  • Createa custom method. You can create a custom method that avoids executing the samelines twice.

Here is the example: [code_inline id=2321]

IMPORTANT USAGE NOTICE PLEASE READ 

Copyright © 2011, FinancialForce.com, inc

All rights reserved.

Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:

  • Redistributions of source code must retain the above copyright notice, ttttthis list of conditions and the following disclaimer.
  • Redistributions in binary form must reproduce the above copyright notice, ttttthis list of conditions and the following disclaimer in the documentation ttttand/or other materials provided with the distribution.
  • Neither the name of the FinancialForce.com, inc nor the names of its contributors ttttmay be used to endorse or promote products derived from this software without ttttspecific prior written permission.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS “AS IS” AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

Let’s talk: 1-866-743-2220