Lazy Iterators

Welcome back to the Joys Of Apex! You may remember from the footnotes of Sorting & Performance In Apex that I mentioned an article on iterators, which I first read in February while browsing the Salesforce subreddit. One thing that I dwelt on for some time after reading the article was how my clients might be able to use the power of lazy iteration - which differs from the eager iteration performed by the traditional "for" loops we use quite often in SFDC Apex development - to speed up trigger handlers. Were there performance gains to be had by delaying the evaluation of records in a Trigger context? I would also highly recommend the YouTube video that is posted in the article: this talk on lazy evaluations from GOTO 2018 is fantastic

I was impressed at the time with the object-oriented approach that Aidan Harding and the folks over at Nebula Consulting had undertaken when implementing the Lazy Iterator framework. The idea for this article has been brewing since then. Their LazyIterator object uses the Decorator pattern to add functionality to the underlying iterators found on all Salesforce List objects — reading through their codebase got me re-excited about working with collections in Apex. You may recall from the Writing Performant Apex Tests post that using iterators to page through collections is much faster than any of the "for" loop implementations!

I'd also like to thank Aidan for generously proof-reading the beta version of this, which led to a couple of great edits. This article is assuredly better for his input.

linkA Short History Of Lazy Evaluation

All problems in computer science can be solved using one more level of indirection. — David Wheeler

So-called "Lazy" evaluated functions have their actual execution delayed until a "terminator" function is called. It's common for lazy functions to be chained together using fluent interfaces, culminating with actions being performed when the terminator function is called. What can Salesforce developers writing Apex code stand to gain by learning more about lazy functions?

Fluent interfaces — or objects that return themselves during function calls — also tend to satisfy one of the prerequisites for Object-Oriented Programming; namely, encapsulation. While fluency as a property of functions/objects has become associated more with functional programming than with OOP, many developers are being unwittingly exposed to pseudo-fluent interfaces (and, as a result, functional paradigms) through the JavaScript collection API; it's not uncommon to see various filter, map, and reduce calls being chained together when iterating through lists in JavaScript. It's also not uncommon for people to underestimate the performance implications that come with using these functions — in JavaScript's vanilla functions' case, our code's readability increases at the cost of performance:

linkEager Evaluation in JavaScript & Apex

1linkconst list = [1, 2, 3, 4, 5, 6, 7];

2linkconst doubleOfEvens = list.filter((x) => x % 2 === 0).map((num) => num * 2);

3link

4linkconsole.log(doubleOfEvens);

5link//output: [ 4, 8, 12 ]

Well-named (and encapsulated) functions are appreciated by developers because they are easy to understand. Even if you don't know JavaScript, you can look at the above code snippet and divine its meaning: given a starting list of numbers, take the ones that don't have a remainder when divided by two, and multiply those values by two. However, when you look at what that code is effectively doing, you might begin to feel differently about it:

1linkvar list = [1, 2, 3, 4, 5, 6, 7];

2linkvar evens = [];

3linkfor (var i = 0; i < list.length; i++) {

4link  var num = list[i];

5link  if (num % 2 === 0) {

6link    evens.push(num);

7link  }

8link}

9link

10linkvar doubleOfEvens = [];

11linkfor (var index = 0; index < evens.length; index++) {

12link  var even = evens[index];

13link  doubleOfEvens.push(even * 2);

14link}

Yikes. Turning back to Apex, you might immediately see the parallels between this example and how a typical Apex trigger handler is set up:

classes/AccountHandler.cls
1linkpublic class AccountHandler extends TriggerHandler {

2link

3link  public override void beforeInsert(List<SObject> newRecords) {

4link    List<Account> accounts = (List<Account>)newRecords;

5link

6link    this.trimAccountName(accounts);

7link    this.formatAccountPhone(accounts);

8link    //etc ...

9link  }

10link

11link  private void trimAccountName(List<Account> accounts) {

12link    for(Account acc : accounts) {

13link      acc.Name = acc.Name.normalizeSpace();

14link    }

15link  }

16link

17link  private void formatAccountPhone(List<Account> accounts) {

18link    for(Account acc : accounts) {

19link      // do stuff with the phone

20link    }

21link  }

22link}

This pattern is pretty typical. It's not uncommon in larger organizations for the big objects — Leads and Opportunities in particular — to have dozens (if not hundreds) of trigger handler methods, each of which likely involves sifting through the entirety of the old/new records prior to performing business logic. As trigger handlers grow in size, related functionality is frequently broken out into other classes; this tends to obscure just how much processing is occurring as a handler churns through records.

Once records are being updated, many developers have utility methods designed to compare the old and new objects handed to SFDC developers in the form of Trigger.old and Trigger.new lists, but even these utility methods frequently take the form of iterating over the entirety of the old and new records to isolate matches. (This becomes a problem when you need to isolate many different groups of changed records to do further processing; e.g. Accounts that need their Opportunities updated, Accounts that need their Contacts updated, if a particular Opportunity Line Item is added, go back to the Account, etc ...) So what can we do? Should we do something to address this "problem" — or is it really not a problem at all, but one of the results of a growing system? As usual, to answer that question, we're going to have to write some tests.

linkMeasuring SObject Trigger Performance

These particular tests will make use of Queueable Apex. Why use Queueables over our typical Apex tests? There's an argument to be made (by some) that the overhead introduced by the test classes themselves actually obscure the production-level performance results. I haven't found that to be the case, but with long-running operations, it's possible for running tests to time out, so we'll be avoiding that issue altogether. In order to avoid hitting any kind of Anonymous Apex timeouts, we'll choose the easiest of the async Apex implementations to rig up a timer:

classes/QueueableTimer.cls
1linkpublic class QueueableTimer implements System.Queueable {

2link  private Datetime lastTime;

3link

4link  public QueueableTimer() {

5link    this.lastTime = System.now();

6link  }

7link

8link  public void execute(QueueableContext context) {

9link    Savepoint sp = Database.setSavepoint();

10link    List<Account> accounts = this.getExampleAccountsToInsert();

11link    this.log('Starting execute after gathering sample records');

12link

13link    insert accounts;

14link

15link    this.log('Ending');

16link    Database.rollback(sp);

17link  }

18link

19link  private List<Account> getExampleAccountsToInsert() {

20link    List<Account> accounts = new List<Account>();

21link    //savepoint usage consumes a DML row ...

22link    for(Integer index = 0; index < 9998; index++) {

23link      accounts.add(

24link        new Account(

25link          Name = ' Testing' + index.format() + ' ',

26link          Phone = '8438816989'

27link        )

28link      );

29link    }

30link    return accounts;

31link  }

32link

33link  private void log(String startingString) {

34link    System.debug(

35link      startingString + ', time passed: '

36link      + this.getSecondsPassed().format()

37link      + ' seconds'

38link    );

39link    this.lastTime = System.now();

40link  }

41link

42link  private Decimal getSecondsPassed() {

43link    return ((Decimal)(System.now().getTime()

44link      - this.lastTime.getTime()))

45link      .divide(1000, 4);

46link  }

47link}

This code is going to interact with our extremely plain AccountHandler object, called by the trigger on Accounts:

1linkpublic class AccountHandler extends TriggerHandler {

2link  public override void beforeInsert(List<SObject> insertedRecords) {

3link    //no methods for now

4link  }

5link}

Right now, with the beforeInsert method empty, kicking off the QueueableTimer prints the following:

1linkStarting execute after gathering sample records, time passed: 0.788 seconds

2linkEnding, time passed: 32.035 seconds

It's important to note that the baseline is being measured prior to the Savepoint being rolled back. Something that should be immediately obvious in looking at these results? It's not the setting up of the savepoint or the Account list that is leading to any kind of slowdown; this is just how long it takes for Salesforce triggers to process large record sets in 200 increment batches. I tested various other org structures, including those with Duplicate Rules enabled, workflows, validation rules, etc ... while those definitely slow down the resulting insert (Standard duplicate rules for Accounts added nearly 7 seconds to the processing time), the vast majority of the time was simply in creating that many objects. This should give you a good idea, per 10k records, how long it takes at baseline to process records: .0032 seconds per Account. Not too shabby.

linkMeasuring Eagerly-Evaluated Methods

We'll just update the AccountHandler object so that the beforeInsert method contains some vanilla processing methods as I had shown earlier:

classes/AccountHandler.cls
1linkpublic override void beforeInsert(List<SObject> newRecords) {

2link  List<Account> accounts = (List<Account>)newRecords;

3link

4link  this.trimAccountName(accounts);

5link  this.formatAccountPhone(accounts);

6link}

7link

8linkprivate void trimAccountName(List<Account> accounts) {

9link  for(Account acc : accounts) {

10link    acc.Name = acc.Name.normalizeSpace();

11link  }

12link}

13link

14linkprivate void formatAccountPhone(List<Account> accounts) {

15link  for(Account acc : accounts) {

16link    this.formatPhone(acc.Phone);

17link  }

18link}

19link

20linkprivate String formatPhoneNumber(String phone) {

21link  if(phone.length() == 10) {

22link    return '(' + phone.substring(0, 3) + ') '

23link        + phone.substring(3, 6) + '-'

24link        + phone.substring(6);

25link  } else if (phone.length() == 11 && phone.substring(0) == '1') {

26link      return this.formatPhoneNumber(

27link        phone.substring(

28link          1,

29link          phone.length() - 1)

30link      );

31link  }

32link

33link  return phone;

34link}

Two simple methods — let's see how much processing time that consumes:

1linkStarting execute after gathering sample records, time passed: 0.403 seconds

2linkEnding, time passed: 33.469 seconds

That's a 4.47% increase in processing time. With smaller number of records, of course, such increases are hardly noticeable — until they are. I think anybody who's converted a lead at the start of a greenfield project versus several years into the implementation can attest to the fact (which I have cited previously in the React.js versus Lightning Web Components post) that delays as small as 50ms can both be detected and negatively impact the end user experience.

linkDiving Into Lazy Evaluation

Somebody recently asked a question regarding the performance and organization of Apex triggers as they grow, and possible design patterns for cleaning up complicated handlers. Since it's something I've spent quite a bit of time thinking about as I pondered the LazyIterator framework, I directed them to read the Nebula Consulting post on Lazy Iterators. Their response?

That seems hard for the next guy to learn, to be honest

They're not wrong. Looking at the Nebula Bitbucket shows that a lot has changed since the article was written last year; many updates to the framework, and a lot of potential. But, like FFLib, the issues with larger frameworks lie in stimulating adoption. How can you get new users to grok the intent of your code, particularly with very abstract examples? Documentation helps, but it's not perfect, and as soon as you have documentation, you're in an arms-race with yourself to keep it up-to-date. Typically, frameworks achieve widespread adoption by providing some combination of three things:

• ease of learning
• enhancements on basic functionality
• performance improvements

These tenets hold true for domains outside of programming, of course, but across the tech stack it's easy to see different permutations of this concept:

• logging frameworks get adopted because, despite typically carrying a learning curve, they help to raise the visibility of errors
• immutability frameworks get adopted because they help to prevent hard-to-trace pointer bugs
• fluent frameworks get adopted because they make the code easier to read

In many ways, I'm the ideal consumer of the LazyIterator framework — I'm a consulting company looking to bring performance improvements to my clients, many of whom already deal with routine system slowdown due to growth. How can I wrap the functionality presented by the LazyIterator concept into something that's easier for others (including myself) to use and understand? The examples that follow are heavily-influenced by Aidan Harding's work. I re-implemented everything from scratch — not something that I think is necessary the vast majority of the time, but somethign that I think indeed helps when looking to further your understanding of new concepts.

linkRe-implementing A Lazy Iterator

This is a stirling use-case for inner classes. Much like how people got really excited when IIFE's became a big part of JavaScript development, inner classes allow you to hide the scary parts of your code from other consumers. And, much like IIFE's, they can abused/over-used. They're not always the answer! Another alternative, which is frequently talked about in the book Clean Code, is the Adapter pattern, where you isolate the code foreign to your codebase through the use of interfaces and boundary objects.

Since we're talking about trigger handler frameworks, I am going to tailor the code that follows towards exploring how to achieve lazy iteration in a trigger handler's context. It should be noted that the LazyIterator framework covers an enormous swath of material and use-cases; keeping it simple here will help to keep the overall size of what you're reading down to a manageable level.

I'll start "simple", with a wrapped iterator capable of detecting when SObjectFields have changed:

linkImplementing A Lazy Filter Function

classes/ObjectChangeProcessor.cls
1link//separate file for the interface

2link//because outside callers can

3link//(and need to) implement

4linkpublic interface BooleanFunction {

5link  Boolean isTrueFor(Object o);

6link}

7link

8linkpublic class ObjectChangeProcessor {

9link  private LazyIterator iterator;

10link

11link//assumes objects are in the same order

12link//as in Trigger.oldRecord, Trigger.new

13link  public ObjectChangeProcessor(List<SObject> oldObjects, List<SObject> newObjects) {

14link    this.iterator = new LazySObjectPairIterator(oldObjects, newObjects);

15link  }

16link

17link  public ObjectChangeProcessor filterByChangeInField(SObjectField field) {

18link    return this.filterByChangeInFields(new List<SObjectField>{ field });

19link  }

20link

21link  public ObjectChangeProcessor filterByChangeInFields(List<SObjectField> fields) {

22link    this.iterator = new LazyFilterIterator(this.iterator, new FieldChangedFilterProcessor(fields));

23link    return this;

24link  }

25link

26link  public ObjectChangeProcessor filter(BooleanFunction function) {

27link    this.iterator = new LazyFilterIterator(this.iterator, function);

28link    return this;

29link  }

30link

31link  public List<Object> toList(List<Object> toList) {

32link    return this.iterator.toList(toList);

33link  }

34link

35link//BASE LAZY ITERATOR

36link  virtual class LazyIterator implements Iterator<Object> {

37link    private final Iterator<Object> iterator;

38link

39link    public LazyIterator(Iterator<Object> iterator) {

40link      this.iterator = iterator;

41link    }

42link

43link    protected LazyIterator() {

44link//one of the more fun statements

45link//made possible by self-implementation ...

46link      this.iterator = this;

47link    }

48link

49link    public virtual Boolean hasNext() {

50link      return this.iterator.hasNext();

51link    }

52link

53link    public virtual Object next() {

54link      return this.iterator.next();

55link    }

56link

57link    public List<Object> toList(List<Object> toList) {

58link      while(this.hasNext()) {

59link        toList.add(this.next());

60link      }

61link      return toList;

62link    }

63link  }

64link

65link//Wrapped SObject Pair Iterator

66link  virtual class LazySObjectPairIterator extends LazyIterator {

67link    private final Iterator<SObject> oldIterator;

68link    private final Iterator<SObject> newIterator;

69link

70link    public LazySObjectPairIterator(List<SObject> oldObjects, List<SObject> newObjects) {

71link      super();

72link      this.newIterator = newObjects.iterator();

73link      this.oldIterator = oldObjects.iterator();

74link    }

75link

76link//wrapper POJO

77link  private class SObjectWrapper {

78link    public final SObject oldRecord, newRecord;

79link    public SObjectWrapper(SObject oldRecord, SObject newRecord) {

80link      this.oldRecord = oldRecord;

81link      this.newRecord = newRecord;

82link    }

83link  }

84link

85link//realistically, you could just do one of

86link//these, since it's required that

87link//both lists have the same # of elements

88link    public override Boolean hasNext() {

89link      return this.oldIterator.hasNext() &&

90link        this.newIterator.hasNext();

91link    }

92link

93link    public override Object next() {

94link      return new SObjectWrapper(

95link        this.oldIterator.next(),

96link        this.newIterator.next()

97link      );

98link    }

99link  }

100link

101link//Iterator that allows for filtering ...

102link  virtual class LazyFilterIterator extends LazyIterator {

103link    private Object next;

104link    private final BooleanFunction filter;

105link    public LazyFilterIterator(LazyIterator iterator, BooleanFunction filter) {

106link      super(iterator);

107link      this.filter = filter;

108link    }

109link

110link/* NB: the Nebula version uses

111linkanother method, "peek()", but for both

112linkterseness and expressiveness, I find this

113linkrecursive method more descriptive: hasNext()

114linkpeeks values ahead of the current object

115linkin the list for matches, advancing the

116linkinternal iterator's place till it

117linkfinds the next match or reaches the end.

118linkThis is tail-recursive and, as such,

119linkstack safe */

120link    public override Boolean hasNext() {

121link      if(super.hasNext()) {

122link        this.next = super.next();

123link        return this.filter.isTrueFor(this.next) ? true : this.hasNext();

124link      }

125link

126link      return false;

127link    }

128link

129link    public override Object next() {

130link      if(this.next != null && this.next instanceof SObjectWrapper) {

131link        return ((SObjectWrapper)this.next).newRecord;

132link      }

133link      return this.next;

134link    }

135link  }

136link

137link  class FieldChangedFilterProcessor implements BooleanFunction {

138link    private final List<SObjectField> fields;

139link    public FieldChangedFilterProcessor(SObjectField field) {

140link      this(new List<SObjectField>{ field });

141link    }

142link    public FieldChangedFilterProcessor(List<SObjectField> fields) {

143link      this.fields = fields;

144link    }

145link

146link    public Boolean isTrueFor(Object obj) {

147link      SObjectWrapper wrapper = (SObjectWrapper)obj;

148link      Boolean hasMatch = false;

149link      Integer counter = 0;

150link//since the matching variable is also

151link//what's being returned, I prefer this format

152link//to the usage of a "break" statement

153link      while(counter < this.fields.size() && !hasMatch) {

154link        hasMatch = wrapper.oldRecord == null ||

155link          wrapper.oldRecord.get(this.fields[counter]) !=

156link          wrapper.newRecord.get(this.fields[counter]);

157link        counter++;

158link      }

159link      return hasMatch;

160link    }

161link  }

162link}

As always, usage of the Decorator pattern means you're looking at a lot more code. However, I've minimized the usage of standalone custom classes in this version, using the ObjectChangeProcessor to wrap everything up with a bow. For more generic usages, you probably wouldn't wrap the LazyIterator itself. What does all of this code get us? Easy and lazily-implemented detection of records in a trigger that have changed based on field conditions:

classes/ObjectChangeProcessorTests.cls
1linkprivate class ObjectChangeProcessorTests {

2link  @isTest

3link  static void it_should_correctly_filter_records() {

4link    Account acc = new Account(

5link      Name = 'Test Account',

6link      NumberOfEmployees = 5

7link    );

8link

9link    Account newAcc = new Account(

10link      Name = acc.Name,

11link      NumberOfEmployees = acc.NumberOfEmployees + 2

12link    );

13link

14link    Account accTwo = new Account(

15link      Name = 'Test Two',

16link      NumberOfEmployees = 5

17link    );

18link

19link    Account accThree = new Account(

20link      Name = 'Test Three',

21link      NumberOfEmployees = 6

22link    );

23link

24link    Account accThreeNew = new Account(

25link      Name = accThree.Name,

26link      NumberOfEmployees = accThree.NumberOfEmployees + 1

27link    );

28link

29link    List<SObject> oldObjects = new List<SObject>{ acc, accTwo, accThree } ;

30link    List<SObject> newObjects = new List<SObject>{ newAcc, accTwo, accThreeNew };

31link

32link    ObjectChangeProcessor processor = new ObjectChangeProcessor(oldObjects, newObjects);

33link

34link    List<Account> accounts = (List<Account>)

35link      processor

36link        .filterByChangeInField(Account.NumberOfEmployees)

37link        .toList(new List<Account>());

38link

39link    System.assertEquals(2, accounts.size());

40link    System.assertEquals(7, accounts[0].NumberOfEmployees);

41link    System.assertEquals(7, accounts[1].NumberOfEmployees);

42link  }

43link}

Writing a test like this — documentation, in and of itself — is my preferred method for investigating a foreign object's API. Does it perform like I expect it to? Does it require complicated arguments to setup and maintain? The further you deviate from the SFDC included library for Apex, the harder it is going to be for somebody else to use.

When examining the original version of LazyFilterIterator's "hasNext" implementation, Aidan suggested that it might not be stack-safe. Apex allows for a maximum stack depth of 1000 units, and running up against that boundary condition wouldn't be covered by the upcoming QueueableTimer tests that you'll see below; because Apex Triggers artificially chunk operations into 200 record increments, it might lead to a false sense of confidence in the code's ability to process large amounts of objects. After tweaking the existing recursive function, I wrote the following test:

classes/ObjectChangeProcessorTests.cls
1link@isTest

2linkstatic void it_should_not_blow_the_stack_while_filtering() {

3link  //that oughtta' do it!

4link  Integer sentinelValue = 10^7;

5link  List<Account> accounts = new List<Account>();

6link  for(Integer index = 0; index < sentinelValue; index++) {

7link    accounts.add(new Account(Name = 'Test ' + index));

8link  }

9link

10link  ObjectChangeProcessor processor = new ObjectChangeProcessor(accounts);

11link  List<Object> sameAccounts = processor

12link    .filter(new AlwaysTrue())

13link    .toList(new List<Account>());

14link

15link  System.assertEquals(sentinelValue, sameAccounts.size());

16link  System.assert(true, 'Should make it here');

17link}

18link

19linkclass AlwaysTrue implements BooleanFunction {

20link  public Boolean isTrueFor(Object o) { return true; }

21link}

And the test passed. Joy. As an aside, I typically don't advocate for testing implementation details (the iterator should work the same regardless of the number of records!); that said, on SFDC, it's always advisable to have bulkified tests to verify that you don't exceed your SOQL/SOSL/DML allowances, and assuring that your custom iterator isn't going to blow up on a large data-set certainly falls into this bulkified testing mandate.

linkImplementing A Lazy Processor Function

Now I want to move on towards achieving feature parity through the LazyIterator with the code shown earlier for the AccountHandler object; namely, how can I load the iterator with functions that can act upon the SObjects passed into the trigger. This involves a sad case of boilerplate due to not being able to cast on a Iterator<SObject> to Iterator<Object>. Let's go back to the ObjectChangeProcessor:

classes/Function.cls
1linkpublic interface Function {

2link  void call(Object o);

3link}

classes/ObjectChangeProcessor.cls
1linkpublic class ObjectChangeProcessor {

2link  private LazyIterator iterator;

3link  private List<Function> functions;

4link

5link  public ObjectChangeProcessor(List<SObject> oldObjects, List<SObject> newObjects) {

6link    this(new LazySObjectPairIterator(oldObjects, newObjects));

7link  }

8link

9link/*alas, this constructor leads to the dreaded

10link"Operation cast is not allowed on type: System.ListIterator<SObject>" error

11linkpublic ObjectChangeProcessor(List<SObject> records) {

12link  this((Iterator<Object>)records.iterator());

13link}*/

14link

15link  public ObjectChangeProcessor(List<SObject> records) {

16link    //so we have to do this instead :-\

17link    this(new LazySObjectIterator(records.iterator()));

18link  }

19link

20link  private ObjectChangeProcessor(LazyIterator iterator) {

21link    this.iterator = iterator;

22link    this.functions = new List<Function>();

23link  }

24link

25link  public ObjectChangeProcessor addFunction(Function func) {

26link    this.functions.add(func);

27link    return this;

28link  }

29link

30link  public void process() {

31link    this.iterator.forEach(this.functions);

32link  }

33link}

And in the iterator inner class:

1linkpublic LazyIterator forEach(Function func) {

2link  return this.forEach(new List<Function>{ func });

3link}

4linkpublic LazyIterator forEach(List<Function> funcs) {

5link  while(this.hasNext()) {

6link//it's iterators all the way down!

7link    Iterator<Function> funcIterator = funcs.iterator();

8link    Object nextObject = this.next();

9link    while(funcIterator.hasNext()) {

10link      Function func = funcIterator.next();

11link      func.call(nextObject);

12link    }

13link  }

14link  return this;

15link}

Plus we need to add the LazySObjectIterator inner class since casting on the Iterator object is not allowed:

1linkvirtual class LazySObjectIterator extends LazyIterator {

2link  private final Iterator<SObject> iterator;

3link  public LazySObjectIterator(Iterator<SObject> iterator) {

4link    super();

5link    this.iterator = iterator;

6link  }

7link

8link  public override Boolean hasNext() {

9link    return this.iterator.hasNext();

10link  }

11link

12link  public override Object next() {

13link    return this.iterator.next();

14link  }

15link}

Going back to our AccountHandler example, it's time to encapsulate the phone/name update methods within classes:

classes/AccountHandler.cls
1linkpublic class AccountHandler extends TriggerHandler {

2link

3link  public override void beforeInsert(List<SObject> insertedRecords) {

4link    new ObjectChangeProcessor(insertedRecords)

5link      .addFunction(new NameNormalizer())

6link      .addFunction(new PhoneNormalizer())

7link      .process();

8link  }

9link

10link  class NameNormalizer implements Function {

11link    public void call(Object o) {

12link      Account acc = (Account)o;

13link      acc.Name = acc.Name.normalizeSpace();

14link    }

15link  }

16link

17link  class PhoneNormalizer implements Function {

18link    public void call(Object o) {

19link      Account acc = (Account)o;

20link      acc.Phone = this.formatPhoneNumber(

21link        //strip non-digits

22link        acc.Phone.replaceAll(

23link          '[^0-9]',

24link          ''

25link        )

26link      );

27link    }

28link

29link  private String formatPhoneNumber(String phone) {

30link    if(phone.length() == 10) {

31link          return '(' + phone.substring(0, 3) + ') '

32link              + phone.substring(3, 6) + '-'

33link              + phone.substring(6);

34link      } else if (phone.length() == 11

35link        && phone.substring(0) == '1') {

36link          return this.formatPhoneNumber(

37link            phone.substring(

38link              1, phone.length() - 1)

39link          );

40link      }

41link      return phone;

42link    }

43link  }

44link}

Note that testing the NameNormalizer and PhoneNormalizer inner classes is easily achievable, and they can also be broken out of the Handler into individual/wrapped classes as their responsibilities increase.

linkMeasuring Lazy Evaluation

Now that the AccountHandler code has been updated, it's finally time to re-run the QueueableTimer object to see how lazy iteration stands up, performance-wise. Note, again, that I take the average of many runs when reporting out on performance. In other news, "finally time" turned out to be ~4 hours of writing between the "QueueableTimer" runs — whoah!

1linkStarting execute after gathering sample records, time passed: 0.363 seconds

2linkEnding, time passed: 32.416 seconds

Plus, the results of the tests in ObjectChangeProcessorTests:

I'll take 5ms to iterate 10 million rows, yes please.

In general, I would hasten to say two things regarding the performance of the LazyIterator — both the vanilla for loop and Lazy Iterator approach were tested dozens of times and an average of their results were taken. That said, the standard deviation for both approaches is large enough that I would caution taking the results too seriously. While I don't find it hard to believe that relying heavily on the native iterators outperforms the additional cost of initializing objects, neither do I find the performance gain in real terms to be the deciding factor in adopting this framework.

linkWrapping Up

Reverse-engineering (an admittedly extremely small portion of) the LazyIterator proved to be good, clean fun. Like all good exercises, it left me with plenty of inspiration for how to apply the code to my own use-cases. While I had a few nits with the overall level of verbosity, in general I would say that the framework code is both well-annotated with Javadoc descriptions and remarkably expressive at a very high-level of abstraction — no easy feat. I left impressed, which is my highest praise.

I will definitely be making use of some of the code here and from the Nebula Consulting repo. I like the fluent nature of working with the wrapped iterator; I can also see, with some work, how I would expand upon the structure orchestrated here to accommodate my two other most frequent use-cases:

• iterating through a list with a Map<Id, Object> or Map<String, Object> that matches a value in that list, and performing processing
• iterating through a list with a Map<Id, List<Object>> or Map<String, List<Object>> that matches a value in that list, and performing processing

Additionally, there are some fun considerations for the LazyIterator — some of which are handled within the existing Nebula Consulting LazyIterator framework, some of which would be excellent additions:

• Allowing you to filter for multiple discrete (not necessarily mutually exclusive, but independent) criteria in a single iteration. This would really help with performing additional processing on subsets of data depending on different field changes / entry conditions without unnecessary iteration. You could definitely finangle this into an existing Function definition, but really you would be looking for a combination of the existing Function and BooleanFunction implementations, where all matches were tested for and, conditionally, processing was done if the result matched. Of course, depending on your business logic, there definitely exists the potential for independent updates to depend on one another in some happy temporal soup. Traditionally, showing that the order of functions being called matters makes use of explicit passing of variables to the further-down-the-line functions to make the coupling explicit. With a fluent iterator, another approach would be necessary; in looking again at the Nebula Repo, their ForkIterator does handle the first use-case (independent filtering), but massaging the API to better broadcast dependent forking is only a dream at the moment
• proper support for empty iterators (clasically, the use of a singleton "null" instance is used; a singleton because you only ever need one instance of it) — I've been promised a framework-wide approach using their EmptyIterator object is forthcoming!
• first class support for Maps, as discussed above. Ideally the LazyIterator would be able to both build a one-to-one (Map<Id, Object> or Map<String, Object>) or one-to-many (Map<Id, List<Object>> or Map<String, List<Object>>) collection as part of a Function and pass the results to future Functions for usage. Unfortunately, while casting is a "pain" with Lists, it's not even allowed with Maps in Apex, which would probably necessitate painful (and limiting) serialization/deserialization techniques (which has actual performance implications, as well)

As always, I hope that this post proved illuminating — if not on the seemingly endless iteration topic, then at least in having walked this road with me for some time. It's always appreciated.

Till next time!