Skip to content
Angular Start

Using the Expand Operator to Fetch Recursively

The last key feature we need to implement is to actually make use of the gifsRequired value (i.e. how many gifs we should display per request), but there is some hidden complexity here. Let’s consider how we would actually implement this.

At the moment, we fetch GIFs from this URL:

  `https://www.reddit.com/r/${subreddit}/hot/.json?limit=100` +
    (after ? `&after=${after}` : '')

Notice that we supply a limit=100 to tell the Reddit API how many posts we want to return. It then might seem sensible to implement a gifsRequired limit like this:

  `https://www.reddit.com/r/${subreddit}/hot/.json?limit=${gifsRequired}` +
    (after ? `&after=${after}` : '')

However, that won’t quite work for us. At the moment this is what we do:

  1. Fetch 100 posts from Reddit
  2. Filter those to only include valid GIFs
  3. Display those GIFs

The actual number of GIFs we end up displaying in our application won’t necessarily be 100 or whatever limit we supply, it could be anywhere from 0 to 100. This is where things get tricky.

We will continue to discuss this as we implement the solution, but the general idea that we want to implement is (assuming an initial gifsRequired value of 20):

  1. Fetch 100 posts from Reddit
  2. Filter those for valid GIFs
  3. If there are less than 20 valid GIFs, keep hitting the API until we do have 20 valid GIFs

The tricky part is that last step, and it is where the expand operator will become extremely useful.

Implement the expand operator

The expand operator might look intimidating, but really it is just one more operator that we are adding onto our stream. Let’s just add it, and talk about it after.

  this.fetchFromReddit(subreddit, lastKnownGif, 20).pipe(
    expand((response, index) => {
      const { gifs, gifsRequired, lastKnownGif } = response;
      const remainingGifsToFetch = gifsRequired - gifs.length;
      const maxAttempts = 15;


      const shouldKeepTrying =
        remainingGifsToFetch > 0 &&
        index < maxAttempts &&
        lastKnownGif !== null;


      return shouldKeepTrying
        ? this.fetchFromReddit(
          subreddit,
          lastKnownGif,
          remainingGifsToFetch
        )
        : EMPTY;
    })
  )

Although there is quite a bit of logic going on here, the basic idea can be broken down to this:

  • Do we need to fetch more gifs? Return the fetchFromReddit stream
  • Do we have enough gifs or do we want to give up? Return the EMPTY stream

If we return an observable stream in the expand operator, it will subscribe to that stream, and then the expand operator will run again. This allows us to keep recursively called the fetchFromReddit stream — with a different lastKnownGif and remainingGifsToFetch each time — until we are satisfied.

The logic we have set up will keep retrying until any of these are true:

  • We have enough gifs
  • We have tried 15 times
  • The lastKnownGif is null meaning that there are no gifs left to fetch

Once any of these conditions are met, we return the EMPTY stream which immediately completes and will cause the recursion to stop.

Now if we load up the application we should see that 20 gifs are displayed. If we scroll down to the bottom, 20 more should display. This generally is easy for the gifs subreddit because there are lots of gifs available in every request. To put our functionality to the test a bit more you might try subreddits with less gifs — pixelart and woodworking are two good examples that have at least some valid gifs, but will likely require some retries with the expand operator to get a full page of 20 (I checked the Network tab whilst trying to load the woodworking subreddit and saw that it took 4 requests to get enough gifs).

What we have built is an extremely complex stream — which is a combination of multiple different streams — and it is more advanced than the types of streams you will usually have to deal with. Again, I want to reiterate that you should view what we have done as more of a brain teaser to challenge your knowledge of observables and RxJS. A bunch of the concepts we have used here will come up from time to time in everyday usage of RxJS, just not usually all at once like this.

Although what we have built is complex, and the operators may be hard to understand in the beginning, RxJS does actually greatly simplify what we are building. The requirements we are trying to satisfy here are just plain hard.

With a few declarative streams and some operators, we have implemented a strategy to recursively retry hitting an API based on some condition, paginate, and react to form changes. This would likely be much more complex and error prone to do without using RxJS operators like we have.