Read more at jamessimone.net, or return to the homepage





Batchable & Queueable Apex

Batchable and Queueable Apex are both powerful processing frameworks within Salesforce. Unlock the power of both Batchable and Queueable Apex with the easily extendable DataProcessor pattern, which I'll detail in this post. This has been a long time coming. I've been dreaming of writing this post for years — after getting burned with a few Batch Apex classes.

Reading the Apex docs concerning Batch Apex, it sounds like a dream come true:

If you use a QueryLocator object, the governor limit for the total number of records retrieved by SOQL queries is bypassed. For example, a batch Apex job for the Account object can return a QueryLocator for all account records (up to 50 million records) in an org.

What the @%^!? 50 million records ?? God, we can all go home, the work's pretty much done for us (just kidding, we're all at home anyway). In reality, the use of Batch Apex is frequently a painful experience with:

Being the chief offenders. Let's not even get onto the subject of maintaining the order of batches once they're in the Apex Flex Queue. Have you ever tried to hit a moving target? If you're one of the poor sods who's tried to juggle which Batch Jobs were executing at any given time, yes — yes, you have tried. I've also already spoken about the Batchable boilerplate in the Enum post, so I won't beat on that horse.

With the introduction of Queueable Apex, it seemed that most people's prayers had been answered; queued jobs ran fast, and it's possible to write recursive Queueable Apex that (when set up correctly) gets around the DML row limit with carefully crafted queries that run in small batches ... essentially, fast Batchable Apex.

But after a few years of using Queueables instead of Batch Apex, I found myself really pushing the limits (no pun intended) of what was possible in a single transaction with only 10,000 rows that could be modified. The truth is, without good sentinel values on the records you're querying (some kind of IsUpdated flag), you can quickly run into trouble with Queueables running forever — head back to the Apex Jobs setup page to try to stop the job before it restarts itself!

Plus, if you need to modify the sentinel value on 1,000 records, but those records could in turn be responsible for creating more than 9 records each (which is certainly possible with one-to-many relationships), you've just run into the kind of territory I was recently exploring for a client. Query too little, and the jobs will run inefficiently; query too much and you run into a Limit exception. Classic Salesforce.

linkIntroducing the DataProcessor

Let's take it back to the end of the Repository post. It's not quite required reading for this post, but I do recommend it. The essentials are the use of the Query and Repository objects to encapsulate SOQL requests, and there'll be a necessary rehash below. That encapsulation is important because passing around Strings is a dangerous game ... and Batchable Apex sadly requires the use of Strings. To begin with, we'll need to revamp the Repository to safely encapsulate the Database.QueryLocator object that Batchables require in their start method. Unfortunately, there's no public constructor for this class, so we won't be able to mock a return value, but that's a story for another day. Here's what I'd like to test first:

classes/Repository_Tests.cls
1link@isTest

2linkstatic void it_should_return_count_properly() {

3link insert new Account(Name = 'Test');

4link Query nameEqualsTest = new Query(Account.Name, Query.Operator.EQUALS, 'Test');

5link IRepository repo = new Repository(Account.SObjectType, new List<SObjectField>());

6link

7link QueryWrapper wrapper = repo.getWrapper(nameEqualsTest);

8link

9link System.assertEquals(1, wrapper.ResultSize);

10link}

Pretty self-explanatory. The QueryWrapper object will encapsulate not only the Database.QueryLocator — it will also receive an aggregated count for underlying requests. This will allow DataProcessor consumers to judge based on how many results are returned if it will be necessary to batch the request or enqueue it. Salesforce does include the Database.countQuery method in the standard Apex library, but we'll need to tweak how the underlying query string is formed in order to properly conform to the format that countQuery expects. Let's review the implementation:

1link//your garden-variety POJO...

2linkpublic class QueryWrapper {

3link public QueryWrapper(Database.QueryLocator locator, Integer resultSize) {

4link this.Locator = locator;

5link this.ResultSize = resultSize;

6link }

7link

8link public Database.QueryLocator Locator { get; private set; }

9link public Integer ResultSize { get; private set; }

10link}

11link

12link//And then the full repository...

13linkpublic class Repository extends Crud implements IRepository {

14link private final Schema.SObjectType repoType;

15link private final List<Schema.SObjectField> queryFields;

16link

17link private Boolean shortCircuit = false;

18link

19link public Repository(Schema.SObjectType repoType, List<Schema.SObjectField> queryFields) {

20link this.repoType = repoType;

21link this.queryFields = queryFields;

22link }

23link

24link public QueryWrapper getWrapper(Query query) {

25link return this.getWrapper(new List<Query>{ query });

26link }

27link

28link public QueryWrapper getWrapper(List<Query> queries) {

29link String queryString = this.getQueryString(queries);

30link Integer resultSize = this.getAggregateResultSize(queries);

31link

32link Database.QueryLocator locator = Database.getQueryLocator(queryString);

33link return new QueryWrapper(locator, resultSize);

34link }

35link

36link public List<SObject> get(Query query) {

37link return this.get(new List<Query>{ query });

38link }

39link

40link public List<SObject> get(List<Query> queries) {

41link String finalQuery = this.getQueryString(queries);

42link System.debug('Query: \n' + finalQuery);

43link List<SObject> results = this.getFromQuery(finalQuery);

44link System.debug('Results: \n' + results);

45link return results;

46link }

47link

48link private String getQueryString(List<Query> queries) {

49link String selectClause = 'SELECT ' + this.addSelectFields();

50link String fromClause = this.getFrom();

51link String whereClause = this.addWheres(queries);

52link return selectClause + fromClause + whereClause;

53link }

54link

55link private String addSelectFields() {

56link Set<String> fieldStrings = new Set<String>{ 'Id' };

57link for(SObjectField field : this.queryFields) {

58link fieldStrings.add(field.getDescribe().getName());

59link }

60link return String.join(new List<String>(fieldStrings), ', ');

61link }

62link

63link private String getFrom() { return '\nFROM ' + this.repoType; }

64link

65link private String addWheres(List<Query> queries) {

66link List<String> wheres = new List<String>();

67link for(Query query : queries) {

68link if(query.isEmpty()) { this.shortCircuit = true; }

69link wheres.add(query.toString());

70link }

71link return '\nWHERE ' + String.join(wheres, '\nAND');

72link }

73link

74link private List<SObject> getFromQuery(String queryString) {

75link return shortCircuit ? new List<SObject>() : Database.query(queryString);

76link }

77link

78link private Integer getAggregateResultSize(List<Query> queries) {

79link String selectClause = 'SELECT Count()';

80link String fromClause = this.getFrom();

81link String whereClause = this.addWheres(queries);

82link

83link return Database.countQuery(selectClause + fromClause + whereClause);

84link }

85link}

Apologies — I normally paste snippets, but because this work builds on the class that was built in the Repository post, it was a little hard to avoid. That's the only rehash necessary, as the rest of the code is all new. The QueryWrapper object ends up with the info it needs, and we only burn one SOQL call in the meantime.

Initially, I was hoping to do something like the following with the DataProcessor:

classes/DataProcessor.cls
1linkpublic abstract class DataProcessor {

2link protected final Integer resultSize;

3link //if you needed to, you could abort

4link //at any time using the jobId

5link protected Id jobId;

6link

7link public DataProcessor(Factory factory) {

8link //implementers will use this constructor

9link //to install dependencies

10link }

11link

12link protected DataProcessor(QueryWrapper wrapper) {

13link this.resultSize = wrapper.ResultSize;

14link }

15link

16link //these end up being the only four methods

17link //you would care about in your implementing classes

18link protected virtual QueryWrapper getWrapper() {

19link throw new DataProcessorException('Not Implemented');

20link }

21link protected virtual void execute(List<SObject> records) { }

22link protected virtual void finish() { }

23link protected virtual Boolean isBatchable() {

24link return this.resultSize > Limits.getLimitDmlRows() / 3;

25link }

26link

27link public void process() {

28link QueryWrapper wrapper = this.getWrapper();

29link //or some other sentinel value you override

30link if(this.isBatchable()) {

31link Database.executeBatch(new DataProcessorBatchable(wrapper));

32link } else {

33link System.enqueueJob(new DataProcessorQueueable(wrapper));

34link }

35link }

36link //I would never do these linebreaks normally

37link //but I know it helps with reading on mobile

38link private virtual class DataProcessorBatchable

39link extends DataProcessor

40link implements Database.Batchable<SObject>, Database.Stateful {

41link private final String queryLocatorString;

42link protected DataProcessorBatchable(QueryWrapper wrapper) {

43link super(wrapper);

44link //trying to store Database.QueryLocator in an instance

45link //variable leads to the dreaded

46link //System.SerializationException:

47link //Not Serializable: Database.QueryLocator error

48link //cache the cleaned query string for re-use instead

49link this.queryLocatorString = wrapper.Locator.getQuery();

50link }

51link

52link public Database.QueryLocator start(Database.BatchableContext context) {

53link this.jobId = context.getJobId();

54link return Database.getQueryLocator(queryLocatorString);

55link }

56link public void execute(Database.BatchableContext context, List<SObject> records) {

57link //if you were doing something really zany here

58link //and the jobId had changed, you could re-save it here

59link this.execute(records);

60link }

61link public void finish(Database.BatchableContext context) {

62link //same

63link this.finish();

64link }

65link }

66link

67link private virtual class DataProcessorQueueable

68link extends DataProcessor

69link implements System.Queueable {

70link private final String query;

71link

72link protected DataProcessorQueueable(QueryWrapper wrapper) {

73link super(wrapper);

74link this.query = wrapper.Locator.getQuery();

75link }

76link

77link public void execute(QueueableContext context) {

78link this.jobId = context.getJobId();

79link this.execute(Database.query(query));

80link this.finish();

81link }

82link }

83link

84link private class DataProcessorException extends Exception{}

85link}

Unfortunately, I quickly ran into two errors that prevented the DataProcessor class from encapsulating the inner classes:

1link#First I had this problem in saving the above:

2linkError: Only top-level classes can implement Database.Batchable<SObject>

3link#Unlucky. Later, in testing the inner Queueable class:

4linkSystem.AsyncException: Queueable cannot be implemented with other system interfaces.

This was a bit of a bummer — I was really hoping to safely encapsulate the all of the processing logic within a single Apex class, hiding the implementation details.

Regardless, we'll still end up with a simple list of methods to override:

Since the second error I printed above was found while testing, I'll show the tests first:

classes/DataProcessor_Tests.cls
1link@isTest

2linkprivate class DataProcessorTests {

3link @TestSetup

4link static void setup() {

5link insert new Account(Name = ACCOUNT_NAME);

6link }

7link

8link @isTest

9link static void it_should_run_as_queueable_for_small_record_sizes() {

10link runTest();

11link System.assertEquals(

12link 'Completed',

13link [SELECT Status FROM AsyncApexJob WHERE JobType = 'Queueable'].Status

14link );

15link //ensure batch didn't also run

16link System.assertEquals(

17link 0,

18link [SELECT Id FROM AsyncApexJob WHERE JobType = 'BatchApexWorker'].size()

19link );

20link }

21link

22link @isTest

23link static void it_should_run_as_batchable_when_instructed_to() {

24link batchable = true;

25link runTest();

26link System.assertEquals(

27link 'Completed',

28link [SELECT Status FROM AsyncApexJob WHERE JobType = 'BatchApexWorker'].Status

29link );

30link //ensure queueable didn't also run

31link System.assertEquals(

32link 0,

33link [SELECT Id FROM AsyncApexJob WHERE JobType = 'Queueable'].size()

34link );

35link }

36link

37link static void runTest() {

38link Test.startTest();

39link //for actual implementers, you would actually

40link //be calling Factory.getFactory().getTheImplementer.process();

41link new TestAccountProcessor(Factory.getFactory()).process();

42link Test.stopTest();

43link

44link Account updatedAccount = [SELECT Name FROM Account];

45link System.assertEquals(ACCOUNT_NAME + ' TestAccountProcessor', updatedAccount.Name);

46link System.assertEquals(true, finished);

47link }

48link

49link static Boolean batchable = false;

50link static Boolean finished = false;

51link static String ACCOUNT_NAME = 'Hi';

52link

53link private class TestAccountProcessor extends DataProcessor {

54link private final IRepository accountRepo;

55link public TestAccountProcessor(Factory factory) {

56link super(factory);

57link //check out the Factory post if

58link //you're scratching your head looking at this!

59link this.accountRepo = factory.RepoFactory.getAccountRepo();

60link }

61link

62link //a simple implementation that fetches all accounts

63link //with the hard-coded name ... but in reality you could be

64link //querying up to 50 million rows!

65link protected override QueryWrapper getWrapper() {

66link return this.accountRepo.getWrapper(

67link new Query(Account.Name, Query.Operator.EQUALS, ACCOUNT_NAME)

68link );

69link }

70link

71link //a really terrible example implementation

72link //but the sky's the limit, really

73link //once this thing's kicked off, it can safely run

74link //with however many records were queried for

75link //in your getWrapper method

76link protected override void execute(List<SObject> records) {

77link List<Account> accounts = (List<Account>) records;

78link for(Account acc : accounts) {

79link acc.Name = acc.Name + ' TestAccountProcessor';

80link }

81link this.accountRepo.doUpdate(accounts);

82link }

83link

84link protected override void finish() {

85link finished = true;

86link }

87link

88link protected override Boolean isBatchable() {

89link return !batchable ? super.isBatchable() : batchable;

90link }

91link }

92link}

linkFixing the DataProcessor

I had to break the inner classes out into their own classes to get things working properly due to the limitations in inner classes shown above. Here's how things ended up:

classes/DataProcessor.cls
1linkpublic abstract class DataProcessor {

2link protected final Integer resultSize;

3link protected final DataProcessor processor;

4link

5link protected Id jobId;

6link

7link public DataProcessor(Factory factory) { }

8link

9link //when you see a constructor like this

10link //you know you're in for a good time

11link protected DataProcessor(QueryWrapper wrapper, DataProcessor processor) {

12link this.resultSize = wrapper.ResultSize;

13link this.processor = processor;

14link }

15link

16link protected virtual QueryWrapper getWrapper() {

17link throw new DataProcessorException('Not Implemented');

18link }

19link protected virtual void execute(List<SObject> records) { }

20link protected virtual void finish() { }

21link protected virtual Boolean isBatchable() {

22link return this.resultSize > Limits.getLimitDmlRows() / 3;

23link }

24link

25link public void process() {

26link QueryWrapper wrapper = this.getWrapper();

27link if(this.isBatchable()) {

28link //pass the current instance now that dependencies

29link //have been setup

30link Database.executeBatch(new DataProcessorBatchable(wrapper, this));

31link } else {

32link System.enqueueJob(new DataProcessorQueueable(wrapper, this));

33link }

34link }

35link

36link private class DataProcessorException extends Exception{}

37link}

38link

39linkpublic virtual class DataProcessorBatchable

40link extends DataProcessor

41link //etc, in either of these

42link //you might need Database.Callout as well

43link implements Database.Batchable<SObject>, Database.Stateful {

44link private final String queryLocatorString;

45link public DataProcessorBatchable(QueryWrapper wrapper, DataProcessor processor) {

46link super(wrapper, processor);

47link this.queryLocatorString = wrapper.Locator.getQuery();

48link }

49link

50link public Database.QueryLocator start(Database.BatchableContext context) {

51link this.jobId = context.getJobId();

52link return Database.getQueryLocator(queryLocatorString);

53link }

54link public void execute(Database.BatchableContext context, List<SObject> records) {

55link this.processor.execute(records);

56link }

57link public void finish(Database.BatchableContext context) {

58link this.processor.finish();

59link }

60link}

61link

62linkpublic virtual class DataProcessorQueueable

63link extends DataProcessor

64link implements System.Queueable {

65link private final String query;

66link

67link public DataProcessorQueueable(QueryWrapper wrapper, DataProcessor processor) {

68link super(wrapper, processor);

69link this.query = wrapper.Locator.getQuery();

70link }

71link

72link public void execute(QueueableContext context) {

73link this.jobId = context.getJobId();

74link this.processor.execute(Database.query(query));

75link this.processor.finish();

76link }

77link}

linkBatchable & Queueable Summary

And the DataProcessorTests pass (in a third of a second, no less). It's important to note that passing the current processor instance using this in DataProcessor.process is the key to success here. This allows the Queueable / Batchable implementation to only care about fulfilling the Salesforce interface requirements, while delegating the processing methodology to actual consumers. This also allows you to only test the things you need to test; the business logic written into the execute methods.

Though it requires three classes to properly setup the DataProcessor, I'm pleased with the results of this particular experiment. No more worrying about whether or not your Queueable is going to accidentally end up trying to process too many things. That's a great safety net to have! Plus, if your org rarely ventures into Batchable territory, if batches do end getting enqueued as query results grow in size, they'll have the chance to run at reasonable speeds (it's only when you're already heavily reliant on batches that slowdowns occur).

Another thing to note — did anybody catch the missed opportunity on Salesforce's side in not having the Context objects inherit in a sane way? It's a shame that there isn't some base class with something like a createdFromId method; I'm fine with there being QueueableContext, and SchedulableContext and BatchableContext interfaces, etc ... but it's not very object-oriented, considering that in the end, regardless of what the context is, you're getting an Id related to the object's initialization.

So what do you think? Is the DataProcessor pattern something you're interested in implementing in your own org(s)? You can browse the full source code for this example on my Github. I hope that this entry in the Joys Of Apex has proven enjoyable; stick around and check out the other posts if you're arriving here for the first time, and thanks for reading!

The original version of Batchable & Queueable Apex can be read on my blog.

Introducing the DataProcessorFixing the DataProcessorBatchable & Queueable Summary

Home Apex Logging Service Apex Object-Oriented Basics Batchable And Queueable Apex Building A Better Singleton Continuous Integration With SFDX Dependency Injection & Factory Pattern Enum Apex Class Gotchas Extendable Apis Future Methods, Callouts & Callbacks Idiomatic Salesforce Apex Introduction & Testing Philosophy Lazy Iterators Lightweight Trigger Handler LWC Composable Modal LWC Composable Pagination LWC Custom Lead Path Mocking DML React Versus Lightning Web Components Refactoring Tips & Tricks Repository Pattern setTimeout & Implementing Delays Sorting And Performance In Apex Test Driven Development Example Testing Custom Permissions Writing Performant Apex Tests



Read more tech articles