In Lenskit 2.2.1 in evaluation when algorithms get to rank popular items the framework throws exception. The problem has been reported to the authors of Lenskit and it will be fixed soon. In this post I offer a workaround for this problem for those who use version 2.2.1.
When I do this:
evaluator.addMetric(new PrecisionRecallTopNMetric("", "", 5, ItemSelectors.mostPopular(5), ItemSelectors.trainingItems(), ItemSelectors.testRatingMatches(Matchers.greaterThanOrEqualTo(THRESHOLD))));
I get this exception:
Here is how the problem can be fixed. However, popular items are going to be chosen based on the whole dataset regardless of the training/test part. First, we need an ItemSelector class.
Finally, we need popular items:
I would like to thank Lenskit developers for quick replies. In my opinion, the framework is developed professionally. Hope this post is helpful.
When I do this:
evaluator.addMetric(new PrecisionRecallTopNMetric("", "", 5, ItemSelectors.mostPopular(5), ItemSelectors.trainingItems(), ItemSelectors.testRatingMatches(Matchers.greaterThanOrEqualTo(THRESHOLD))));
I get this exception:
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| org.grouplens.lenskit.eval.TaskExecutionException: error in evaluation job task | |
| at org.grouplens.lenskit.eval.traintest.TrainTestEvalTask.runEvaluations(TrainTestEvalTask.java:532) | |
| at org.grouplens.lenskit.eval.traintest.TrainTestEvalTask.perform(TrainTestEvalTask.java:448) | |
| at org.grouplens.lenskit.eval.traintest.SimpleEvaluator.call(SimpleEvaluator.java:344) | |
| at bionic.SimpleEvaluatorExample.main(SimpleEvaluatorExample.java:102) | |
| Caused by: com.google.common.util.concurrent.UncheckedExecutionException: com.google.common.util.concurrent.UncheckedExecutionException: java.lang.NullPointerException | |
| at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2203) | |
| at com.google.common.cache.LocalCache.get(LocalCache.java:3937) | |
| at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941) | |
| at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824) | |
| at org.grouplens.lenskit.eval.traintest.LenskitTestUser.getRecommendations(LenskitTestUser.java:97) | |
| at org.grouplens.lenskit.eval.metrics.topn.PrecisionRecallTopNMetric.doMeasureUser(PrecisionRecallTopNMetric.java:101) | |
| at org.grouplens.lenskit.eval.metrics.topn.PrecisionRecallTopNMetric.doMeasureUser(PrecisionRecallTopNMetric.java:47) | |
| at org.grouplens.lenskit.eval.metrics.AbstractMetric.measureUser(AbstractMetric.java:84) | |
| at org.grouplens.lenskit.eval.traintest.TrainTestJob$MetricWithAccumulator.measureUser(TrainTestJob.java:245) | |
| at org.grouplens.lenskit.eval.traintest.TrainTestJob.runEvaluation(TrainTestJob.java:142) | |
| at org.grouplens.lenskit.eval.traintest.TrainTestJob.call(TrainTestJob.java:86) | |
| at org.grouplens.lenskit.eval.traintest.JobGraph$JobNode.call(JobGraph.java:116) | |
| at org.grouplens.lenskit.eval.traintest.JobGraph$JobNode.call(JobGraph.java:102) | |
| at org.grouplens.lenskit.util.parallel.SequentialTaskGraphExecutor.execute(SequentialTaskGraphExecutor.java:37) | |
| at org.grouplens.lenskit.eval.traintest.TrainTestEvalTask.runEvaluations(TrainTestEvalTask.java:529) | |
| ... 3 more | |
| Caused by: com.google.common.util.concurrent.UncheckedExecutionException: java.lang.NullPointerException | |
| at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2203) | |
| at com.google.common.cache.LocalCache.get(LocalCache.java:3937) | |
| at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941) | |
| at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824) | |
| at com.google.common.cache.LocalCache$LocalLoadingCache.getUnchecked(LocalCache.java:4830) | |
| at org.grouplens.lenskit.eval.metrics.topn.ItemSelectors$PopularItemSelector.select(ItemSelectors.java:471) | |
| at org.grouplens.lenskit.eval.traintest.LenskitTestUser$RecommendLoader.load(LenskitTestUser.java:154) | |
| at org.grouplens.lenskit.eval.traintest.LenskitTestUser$RecommendLoader.load(LenskitTestUser.java:143) | |
| at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527) | |
| at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319) | |
| at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282) | |
| at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2197) | |
| ... 17 more | |
| Caused by: java.lang.NullPointerException | |
| at org.grouplens.lenskit.eval.metrics.topn.ItemSelectors$PopularItemSelector.apply(ItemSelectors.java:483) | |
| at org.grouplens.lenskit.eval.metrics.topn.ItemSelectors$PopularItemSelector.apply(ItemSelectors.java:458) | |
| at com.google.common.cache.CacheLoader$FunctionToCacheLoader.load(CacheLoader.java:151) | |
| Disconnected from the target VM, address: '127.0.0.1:65133', transport: 'socket' | |
| at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527) | |
| at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319) | |
| at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282) | |
| at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2197) | |
| ... 28 more |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| package bionic; | |
| import com.google.common.base.Function; | |
| import com.google.common.cache.CacheBuilder; | |
| import com.google.common.cache.CacheLoader; | |
| import com.google.common.cache.LoadingCache; | |
| import it.unimi.dsi.fastutil.longs.LongSet; | |
| import it.unimi.dsi.fastutil.longs.LongSets; | |
| import org.grouplens.lenskit.Recommender; | |
| import org.grouplens.lenskit.core.LenskitRecommender; | |
| import org.grouplens.lenskit.cursors.Cursor; | |
| import org.grouplens.lenskit.data.dao.ItemEventDAO; | |
| import org.grouplens.lenskit.data.event.Event; | |
| import org.grouplens.lenskit.data.history.ItemEventCollection; | |
| import org.grouplens.lenskit.eval.metrics.topn.ItemSelector; | |
| import org.grouplens.lenskit.eval.traintest.TestUser; | |
| import org.grouplens.lenskit.util.ScoredItemAccumulator; | |
| import org.grouplens.lenskit.util.TopNScoredItemAccumulator; | |
| import javax.annotation.Nullable; | |
| public class MyPopularItemSelector implements ItemSelector, Function<Recommender, LongSet> { | |
| private final LongSet set; | |
| private final LoadingCache<Recommender, LongSet> cache; | |
| public MyPopularItemSelector(LongSet set) { | |
| cache = CacheBuilder.newBuilder() | |
| .weakKeys() | |
| .build(CacheLoader.from(this)); | |
| this.set = set; | |
| } | |
| @Override | |
| public LongSet select(TestUser user) { | |
| return cache.getUnchecked(user.getRecommender()); | |
| } | |
| @Nullable | |
| @Override | |
| public LongSet apply(@Nullable Recommender input) { | |
| return set; | |
| } | |
| @Override | |
| public boolean equals(Object o) { | |
| if (this == o) return true; | |
| if (o == null || getClass() != o.getClass()) return false; | |
| MyPopularItemSelector that = (MyPopularItemSelector) o; | |
| if (set.equals(set)) return false; | |
| return true; | |
| } | |
| @Override | |
| public int hashCode() { | |
| return set.size(); | |
| } | |
| /** | |
| * Cache loader to extract the item universe from a recommender. | |
| */ | |
| private static class UniverseLoader extends CacheLoader<Recommender, LongSet> { | |
| private int count; | |
| private UniverseLoader(int count) { | |
| this.count = count; | |
| } | |
| public LongSet load(Recommender input) throws Exception { | |
| if (input == null) { | |
| return LongSets.EMPTY_SET; | |
| } | |
| LenskitRecommender rec = (LenskitRecommender) input; | |
| ItemEventDAO idao = rec.get(ItemEventDAO.class); | |
| ScoredItemAccumulator accum = new TopNScoredItemAccumulator(count); | |
| Cursor<ItemEventCollection<Event>> items = idao.streamEventsByItem(); | |
| try { | |
| for (ItemEventCollection<Event> item : items) { | |
| accum.put(item.getItemId(), item.size()); | |
| } | |
| } finally { | |
| items.close(); | |
| } | |
| return accum.finishSet(); | |
| } | |
| } | |
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| ... | |
| evaluator.addMetric(new PrecisionRecallTopNMetric("", "", 5, new MyPopularItemSelector(getPopItems(5)), ItemSelectors.trainingItems(), ItemSelectors.testRatingMatches(Matchers.greaterThanOrEqualTo(THRESHOLD)))); | |
| ... | |
| private static LongSet getPopItems(int popNum) { | |
| EventFormat eventFormat = new DelimitedColumnEventFormat(new RatingEventType()); | |
| DataSource dataSource = new GenericDataSource("split", new TextEventDAO(new File(DATASET_PATH), eventFormat), | |
| new PreferenceDomain(1, 5)); | |
| ItemEventDAO idao = dataSource.getItemEventDAO(); | |
| ScoredItemAccumulator accum = new TopNScoredItemAccumulator(popNum); | |
| Cursor<ItemEventCollection<Event>> items = idao.streamEventsByItem(); | |
| try { | |
| for (ItemEventCollection<Event> item : items) { | |
| accum.put(item.getItemId(), item.size()); | |
| } | |
| } finally { | |
| items.close(); | |
| } | |
| return accum.finishSet(); | |
| } |
I would like to thank Lenskit developers for quick replies. In my opinion, the framework is developed professionally. Hope this post is helpful.
No comments:
Post a Comment