We think about this a lot. From speaking with experienced coffee buyers and evaluators, the consensus is a roast that showcases a coffee in a true and fair way, reflecting and showing its potential for those that taste it. 

Clean, sweet, and transparent.

Said another way, a good sample roast should allow a cupper to assess and understand the quality and scope of the coffee. Whether or not this a standardised roast level or more tailored to a particular lab is an interesting conversation. A standardised roast across the industry may one day be possible, but for now it’s most important that the roasts are consistent in house. 

Each table of samples, and each sample, should be roasted the same so that each time you cup you’re basing your decisions in the same development space, so to speak. The roast should not be the variable you’re assessing. This is something our customers tell us they love about the IKAWA sample roaster – it’s so easy to get the same roast, sample-to-sample, and allows them to focus on evaluating the coffees. 

Whichever roast is chosen, the level of roast should be within the boundaries of a “good roast” that meets the above “clean, sweet, transparent”, but after that the way each coffee presents due to water, grinders, etc, is going to be slightly different. So having a consistent sample roast is essential.

In the context of Best of Panama, Cup of Excellence, or Taste of Harvest, competitions where the final round is judged by a group of international cuppers flying into a producing country to cup, maybe a sample roast should strive to showcase coffees in the most favourable way to the most number of people.

A good roast in this scenario has a different goal – at an international competition where the coffees will be auctioned off afterwards, the goal is to be appealing as well as transparent. We know that conditions lab-to-lab, roastery-to-roastery will differ slightly and that buyers and cuppers dial their evaluation down to very small nuances. 

The sample roasts at the competitions should be chosen to allow the coffee to score well, but also be considered to be ‘good enough’ (to showcase the coffee fairly) by the widest possible audience, rather than polarising (ie loved and hated), which can average a score down with the scoring format.

To simply test what effect the average score had, we did an ‘alternative evaluation process’ at the 2019 Nordic Roaster Forum in Oslo. After conducting a cupping where each cup was scored with Cropster Cup, we thought we’d test another approach. We asked the audience of around 25 to ‘put your hand up if you think this roast does not reflect the coffee well’. In our mini profile challenge, all the top 7 roasts had 3 or less hands up. To read more on our challenge at NRF, see part 2 of this series.

Potentially, this “yes or no” is a better marker for a good sample roast. Just like some coffees at these competitions, scoring a sample roast could average out to be in the middle of “love / hate”.

What is Average?
It is very difficult to roast a coffee so that it is perfect for everyone. To do so, we would all need to be calibrated to the same standards. We can dive into this a little further, and pull some scores from our workshop last year to highlight this more. 

Looking at the average scores from our workshop this year, if we remove the outliers (highest and lowest score for each cup) then we see a very tight grouping around 85 points for each cup.

This, at first glance, seems to suggest that the roast doesn’t matter, and that experienced professionals can taste the coffee if the roast is good enough. 

However, when we look at scores for each individual, the story opens up. We can look at  highest and lowest scores from the group and see the range: 

Here, we are getting a better look at how much a roast could affect someone’s perception of a coffee. How high or low does the score go when the only variable is the roast? 

When we took the average difference between each cuppers high and low score, it was surprisingly large: 4.5 points. 

This is likely exaggerated because the cuppers knew that the cupping was one coffee roasted many different ways and part of the goal was finding the best roast profile. 

But last year we held a similar exercise with a random assortment of coffees and asked cuppers to blindly score them. In one section, the same coffee was on the table more than once, and we saw similar results. 

For one coffee, a beautiful Parainema from Santa Barbara in Honduras, we had two sets of cups. Each had the same coffee yet two different (good) roasts, and there was a 1.5 point difference between sets.  

At another set of roasts, a lovely coffee from Guji Ethiopia, we had 3 good roasts for cuppers to score. Again, cuppers did not know that it was the same coffee. The average difference between the highest and lowest score was 4.1 points. 

That is a massive difference. 80 – 84, or 84 – 88, or 88-92. From a producers side, that is a considerable difference in income. From a roastery side, that is an excellent coffee or just a good one. 

Even 1.5 points is enough to affect the price of a coffee in a significant way, or have a coffee be passed up for another. Whether or not the results are exaggerated, this is really important, and displays what we already know from production roasting: small differences in roast make large differences in the cup.

So, how do we ensure that we are getting a roast that allows us to see the best in a coffee? 

The international jury abroad scenario is important, but not the normal setting for evaluating coffee for purchase. Usually, an evaluator is in their own environment, their own lab, either at an exporter, importer, roastery, or similar. 

Knowing your lab and figuring out which roasts works best with your water/grinder/etc is essential. Test your extraction on your roasts, see how different coffees develop in your conditions.

Equally as important, consistency in roast is key. Once you have found that sweet spot, make sure you can roast to it every time. Just like a production roast is affected by very small differences, so is a sample roast. A tool like the IKAWA Pro gives you assurance that you’re getting the right roast, every time. 

Finally, cupping is a developed skill. The professional green buyers had a tighter margin each time for our fun NRF workshops. Those that cup coffees more, and learn to cup in different environments, have more success when they are away from their calibrated labs.