About these Lessons

I’m an undergraduate from the National University of Singapore. I researched reinforcement learning for a full year, including full-time over winter and summer breaks, as part of an undergraduate research programme. I faced many uncertainties and difficulties while doing research over the year, and deliberated over choosing to spending time on research versus on industry-relevant skills.

This blog post condenses all the hard-earned lessons I’ve learnt after stumbling through mistakes and picking myself up again. After 1 year of research, I believe that my decision to spend 1 year’s worth of time on research was well worth it, purely from my takeaways, even though I ended up without a publication that I desired so much at the start of my journey. In writing this post, I hope that fellow researchers that face related challenges about research can learn without going through the hard way, and I believe that this post will be helpful to refer back to when stuck on various difficulties related to research.

This post is structured by lessons on:

  1. Research - Insights about research methodology
  2. Learning - Reflections about learning effectively
  3. Reinforcement Learning - Difficulties I faced with my research topic
  4. Workflow - Hard-earned lessons that incurred significant costs
  5. Motivation - Introspections about how to advance myself sustainably
  6. Support - Acknowledgements for the social structures that enabled me
  7. Mindset - Hard-earned lessons about life that I learnt from research

Each lesson consists of an advice, a brief explanation and the lesson through which I learnt it.

I would like to thank my research advisor Prof. Harold Soh from the Collaborative, Learning and Adaptive Robots (CLeAR) Lab at NUS for his mentorship, my seniors in the lab Bingcai and Kaiqi for their advice, my close friends Shiying, Jun Jie, Wai Ching, Ming Liang and Liying for their support, and many other people who have influenced and helped me in this short but memorable 1-year journey.

I am also looking for a research internship in 2022, ideally reinforcement learning; please let me know if you are recruiting!

Research

  1. The problem statement is the most important thing in research because without one, there’s no real need for the research idea.
  2. Develop a long-term vision of your research direction to avoid tunnel-visioning into a few specific lines of research.
  3. Reflect on the significance of problems that papers aim to address as there exists papers that explore ideas without addressing a real problem.
  4. When reading papers, reflect about the magnitude of improvement made towards addressing the targeted problem.
  5. Understand the significance of performance improvements with respect to the problem, such as the units or measure used.
  6. When reading papers, analyse closely how the authors overcame difficulties to make the idea work, because papers tend to highlight successes, unintentionally downplaying the difficulties needed to be overcome to make the proposed idea work.
  7. When evaluating ideas in papers, consider the compute and space required, and whether you can afford to try those ideas given your constraints.
  8. Simple, elegant ideas are preferable to complex, “Frankenstein” ideas that can seem more like a series of hacks.
  9. Try to work towards significant innovations instead of “delta improvements” that generates only little or even negligible insights.
  10. Research experiments should be minimally sufficient and target the hypothesis well so that time is not wasted due to redundant results.
  11. Research papers are about developing insights, not chasing performance improvements, since significant insights will lead to many subsequent papers.
  12. During idea generation, focus on how you’d solve problems and less on others’ existing solutions.
  13. Evaluate problem significance not just on small “toy problems”, as problems could turn out massive when scaled up or completely vanish in a real-world setting.
  14. Reflect on difficulties in your research field even if they may not be directly related to your targeted problem as those may eventually hinder your progress if left neglected.
  15. Don’t just read or you’ll never get started. At some point, start formulating potential solutions to problems.
  16. Consider the time requirements when scoping your problem, idea or experiment, as working on them may turn out infeasible with limited resources.
  17. Invest time in learning how to do research and not just on your research topic.
  18. Choose a research topic that your lab specialises in, not one that is indirectly related, or choose a lab according to your research interests.
  19. A research idea that is elegant is one that is minimally sufficient.
  20. Avoid being overambitious with a research idea, otherwise, chances are that the method won’t work or it works but you don’t understand why.
  21. Always situate your problem, motivation and experiment design around existing literature to enable comparison and context to justify design choices.
  22. Don’t take it personally if an idea that seems justified initially turns out ineffective, since not all ideas work well, yet some simple ideas can be unreasonably effective.

Learning

  1. Ensure a strong mastery of foundations.
  2. Don’t shy away from difficult topics — break them down into more manageable components, focusing on the years-long goal of mastery.
  3. Keep updated with research papers published across multiple different conferences.
  4. Frequently consolidate knowledge you learnt, through summaries and mindmaps.
  5. After reading many papers, write a survey to organise the structure of existing literature and synthesise current research problems, methods and directions.
  6. Especially important for complex topics, create a mindmap for the large diversity of existing problems, tasks and approaches in your field of research for an overview perspective.
  7. Summarise papers and their key contributions to enable efficient revision and future reference.
  8. Teach and present paper summaries to others to ensure you understand the nuances of the paper and get other perspectives that you might not have considered.
  9. Invest time in learning topics in adjacent or relevant fields, even if they may not be immediately related.
  10. Before attempting to generate potential paper ideas, ensure that you have a robust understanding of the current literature, otherwise time may be wasted when another paper might have already hinted at its ineffectiveness.
  11. Remember that online resources for explanations and code walkthroughs are available.
  12. Before formulating ideas, dive into the theory to thoroughly understand the nuances between the theory of different works.
  13. Invest time in software development skills.
  14. Explore and work on ideas from adjacent fields; it usually brings new insights into your research.
  15. Don’t shy away from theory papers that seem difficult initially; they are a treasure trove of learning opportunities.
  16. Be daring to express opinions and ask questions that you’re uncertain about, so as to receive feedback that directly updates your views.
  17. Keep updated with “key highlight” papers in adjacent fields, as such influential ideas may eventually reach into your research topic.

Reinforcement Learning

  1. Remember to consider variance of results and the usage of random seeds.
  2. Algorithms used for baseline comparisons should be evaluated with proper hyperparameter tuning.
  3. Consider the large number of possible hyperparameter configurations in experiment formulation.
  4. List all possible modes of failure after every experiment so that you can diagnose later problems easier.
  5. Bugs in the code are more likely to be the cause of unexpected inferior performance rather than the idea itself.
  6. Scope your research project according to available resources, such as focusing on problems that fundamentally require less compute, e.g. Data-Efficienct RL.
  7. Inspect papers’ design choices thoroughly, as other options are likely to be less ideal.
  8. Scope your project to focus on and analyse only a few selected components necessary to target the problem of interest, especially if it has many interacting components, e.g. in model-based reinforcement learning.
  9. Be aware of research challenges present in your research field, e.g. reproducibility and benchmarking.

Workflow

  1. Justify experiment steps with the hypothesis it aims to answer, so that you don’t waste time on experiments that don’t generate insights.
  2. Begin with simpler or smaller experiments to validate problems and ideas before moving to more complex and bigger ones.
  3. Remember to parallelise experiments, even using CPUs alongside GPUs can be very useful.
  4. Always consider all computational resources that are available to you and utilise them to the fullest, such as cloud compute credits.
  5. Use a dedicated reference manager while reading papers, e.g. Zotero or Mendeley.
  6. Invest time in setting up proper experimental tools and configurations, e.g. logging, Weights & Biases, code refactoring, as the time cost pays off much earlier than you expect.
  7. Categorise the papers you’ve read properly, such as through tags using a reference manager.
  8. Track dependencies and environment configurations of your project’s code.
  9. Invest time in learning to use Linux and set up servers.
  10. Organize your work environment, be it your physical desk and research papers, your digital notes and PDFs, or your saved Tweets and bookmarks.
  11. Don’t necessarily work and update your advisor week after week, since ideas sometimes require more than 1 week to develop before the justification seems sound.
  12. Make it a habit to work on research a few hours a day, even if you seem to make no progress.
  13. Employ a systematic approach to reading papers, such as annotating or writing summaries.
  14. Set deadlines on each individual task and keep to them strictly.
  15. Narrow down and formulate your research project exactly, so as to better structure your research methodology in the short term.
  16. Remember to clear space from trash folders.
  17. If you can, work on only a few commitments at a time, so that you have free time and space to digest, synthesise and be creative with your research.
  18. Invest time in improving your workflow across all levels of work, e.g. note-taking frameworks, work-break sprints, or simply clearing your desk at the end of each day.
  19. Make sure that you understand everything before extending an existing code base, as one modification can affect another component unexpected, or even worse, fail silently.
  20. Commit a fixed ratio of time to explore topics adjacent to your research field.

Support

  1. A community makes the journey of research easier.
  2. Be daring to reach out for help from seniors and more experienced people, even if it may seem trivial.
  3. Reach out to researchers to ask questions or request for help about the papers they author.
  4. Reach out often to fellow researchers to discuss ideas, especially those whose work you admire.
  5. Reach out to professors in your university; don’t wait for a dedicated research programme to get started with research.
  6. Research is not the only thing that matters in life; remember to take a break once in a while to spend time with family and friends.
  7. Try to bounce ideas off other people in your lab to get their perspectives.
  8. Get in touch with other researchers working in the same field as you, such as other professors in your university.
  9. Keep frequent contact and maintain a good relationship with your research advisor.

Motivation

  1. Remember to review and acknowledge the progress you’ve made and successes you’ve achieved.
  2. Celebrate small wins consistently.
  3. Use paper submission deadlines as a goal to work towards and be motivated about.
  4. Remember that the goal of publishing should only be used for motivation and not your main objective of research.
  5. Don’t fall prey to personal comparisons with other researchers, such as differences in competence, starting points, resources or available guidance, because everyone’s journey is unique.
  6. Remember to maintain your health and fitness.
  7. Know what motivates you and learn to manage your motivation.

Mindset

  1. Great things take time.
  2. It’s okay not to know many things; you only need to know a few things deeply.
  3. Don’t let yourself be too affected by the opportunity costs of doing research.
  4. Be persistent on hammering at the problem and don’t give up until you’ve exhausted all possible modes of failure, because research problems are difficult.
  5. It’s alright to not achieve concrete achievements such as publications, especially in your first few years of research.
  6. Stop thinking about the past and less-ideal starting points that you possibly had, and focus on what you can do now to improve yourself from here onwards.
  7. Be humbled by the difficulty and complexity of research; don’t be overconfident about your abilities.
  8. Know your strengths, embrace your weaknesses, and then leverage your strengths as much as possible.
  9. Research has a long-horizon and sparse-reward credit assignment problem, as results may not show until months or even years later.
  10. Don’t work on ideas simply because you think they might work. Instead, work on ideas that you are passionate about and it will show in your work.
  11. Sometimes, your best is not enough. And that’s alright.
  12. Focus on doing your best work, not a work that is better than others.
  13. Research is but one aspect of life.
  14. Your best idea is always your next idea, so if your current idea didn’t turn out well, look forward to your next idea and move on.
  15. Instead of focusing on failure, focus on the lessons you’d learnt in the process.
  16. You never really stop learning lessons.